Extended reality instrument interactive area for guided robotic surgery

Abstract

The present invention discloses a surgical system, which includes an XR headset, a tracking system, and an XR headset controller. The XR headset can be worn by a user during a surgical procedure and includes a perspective display screen, which is configured to display a world-registered XR image and allow at least a portion of a real-world scene to pass through it for viewing by the user. The tracking system can determine a real-world pose of the XR headset and a real-world pose of a real-world element. The real-world pose of the XR headset and the real-world pose of the real-world element are determined relative to a real-world coordinate system. The XR headset controller can generate the world-registered XR image based on the real-world pose of the XR headset and the real-world pose of the real-world element. The world-registered XR image includes a virtual element generated based on the characteristics of the real-world element.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation-in-part of U.S. patent application serial number 16 / 709,185 filed on December 10, 2019, which is incorporated herein in its entirety. Technical Field

[0003] The present disclosure relates to medical devices and systems, and more particularly, to computer-assisted navigation in surgeries performed using robotic surgical systems. Background Art

[0004] Computer-assisted navigation during surgery provides the surgeon with enhanced visualization of surgical instruments relative to radiographic images of the patient's anatomy. Navigational surgery typically includes components for tracking the position and orientation of surgical instruments using a disk or sphere array set up using a single near-infrared (NIR) stereo camera. In this scenario, three parameters compete for optimization: (1) accuracy, (2) robustness, and (3) ergonomics.

[0005] Navigating surgical procedures using existing navigation systems is susceptible to events that trigger intermittent pauses when people and / or objects obstruct the ability of the tracking components to track the posture of the patient, robot, and surgical instruments.There is a need to improve the tracking performance of navigation systems. Summary of the invention

[0006] Various embodiments disclosed herein relate to improvements in computer-assisted navigation during surgery. One or more extended reality ("XR") headsets may be equipped with tracking cameras that provide tracking information to a camera tracking system to be combined with tracking information from other tracking cameras, which may be part of another XR headset, an auxiliary tracking wand, or other device. By operating through various gesture chains disclosed herein, the camera tracking system is able to track tools and other objects with greater robustness and through a wide range of motion.

[0007] In one embodiment, a surgical system includes an extended reality ("XR") headset configured to be worn during a surgical procedure, a tracking system, and an XR headset controller. The XR headset includes a see-through display screen configured to display a world-registered XR image and allow at least a portion of a real-world scene to pass through it for viewing by a user. The tracking system is configured to determine a real-world pose of the XR headset and a real-world pose of a real-world element. The real-world pose of the XR headset and the real-world pose of the real-world element are determined relative to a real-world coordinate system. The XR headset controller is configured to generate a world-registered XR image based on the real-world pose of the XR headset and the real-world pose of the real-world element. The world-registered XR image includes a virtual element generated based on characteristics of the real-world element.

[0008] Disclosed are a related method of a surgical system and a related robotic surgical system.

[0009] Other surgical systems, methods, and computer program products according to embodiments will be apparent to those skilled in the art after reviewing the following drawings and detailed description.In addition, all embodiments disclosed herein are intended to be implemented alone or in combination in any manner and / or combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings illustrate certain non-limiting embodiments of the inventive concept, and are included to provide a further understanding of the present disclosure, and are incorporated in and constitute a part of this application. In the drawings:

[0011] Figure 1 An embodiment of a surgical system according to some embodiments of the present disclosure is depicted;

[0012] Figure 2 Some embodiments of the present disclosure are shown. Figure 1 Surgical robotic components of surgical systems;

[0013] Figure 3A Some embodiments of the present disclosure are shown. Figure 1 A camera tracking system component of a surgical system;

[0014] Figure 3B and Figure 3C The present invention is described in detail. Figure 1 a front view and an isometric view of another camera tracking system component used in conjunction with the surgical system;

[0015] Figure 4 An embodiment of an end effector connectable to a robotic arm and configured in accordance with some embodiments of the present disclosure is depicted;

[0016] Figure 5 depicts a medical procedure in which a surgical robot and camera system are positioned around a patient;

[0017] Figure 6 Depicted is a medical procedure Figure 5 A block diagram view of components of a surgical system;

[0018] Figure 7 Depicts the navigation features that can be used when using the surgical system. Figure 5 and Figure 6 Various display screens displayed on the monitor;

[0019] Figure 8 depicts a block diagram of some electrical components of a surgical robot according to some embodiments of the present disclosure;

[0020] Fig. 9 depicts a block diagram of components of a surgical system including an imaging device connected to a computer platform that may be operably connected to a camera tracking system and / or a surgical robot according to some embodiments of the present disclosure;

[0021] Fig.10 An embodiment of a C-arm imaging device that can be used in combination with a surgical robot according to some embodiments of the present disclosure is depicted;

[0022] Fig.11 depicts an embodiment of an O-arm imaging device that may be used in combination with a surgical robot according to some embodiments of the present disclosure;

[0023] Fig.12 depicts a block diagram view of components of a surgical system including a pair of XR headsets and an auxiliary tracking wand operating in accordance with some embodiments of the present disclosure;

[0024] Fig.13 depicts an XR headset configured in accordance with some embodiments of the present disclosure;

[0025] Fig.14 depicts electrical components of an XR headset that may be operably connected to a computer platform, an imaging device, and / or a surgical robot according to some embodiments of the present disclosure;

[0026] Fig.15 Depicted is a block diagram showing the arrangement of optical components of an XR headset, according to some embodiments of the present disclosure.

[0027] Fig.16depicts an exemplary view of a display screen through an XR headset for providing navigation assistance to manipulate surgical tools during a medical procedure in accordance with some embodiments of the present disclosure;

[0028] Fig.17 depicts an exemplary configuration of an auxiliary tracking stick with two pairs of stereo cameras configured in accordance with some embodiments of the present disclosure;

[0029] Figures 18 to 22 depicts an example of an error introduced during navigation assistance of a surgical tool used in a medical procedure according to some embodiments of the present disclosure;

[0030] Fig.23 An example of a medical procedure using navigation assistance according to some embodiments of the present disclosure is depicted;

[0031] Figures 24 to 32 An example of manipulating the navigation aid of FIG. 37 is depicted according to some embodiments of the present disclosure.

[0032] Fig.33 A block diagram of an example of a surgical system according to some embodiments of the present disclosure is depicted.

[0033] Fig.34 A block diagram is depicted of an example of a process performed by a surgical system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] The inventive concept will now be described more fully below with reference to the accompanying drawings, in which examples of embodiments of the inventive concept are shown. The inventive concept can be implemented in many different forms and should not be construed as being limited to the embodiments listed herein. Rather, these embodiments are provided to make the disclosure more complete and complete, and to fully convey the scope of each inventive concept to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. A component in one embodiment may be defaulted to being present in or used in another embodiment.

[0035] Various embodiments disclosed herein relate to improvements in computer-assisted navigation during surgery. An extended reality (XR) headset is operably connected to a surgical system and is configured to provide an interactive environment through which surgeons, assistants, and / or other personnel can view patient images and make selections therein, view computer-generated surgical navigation information and make selections therein, and / or control surgical equipment in an operating room. As described below, the XR headset can be configured to enhance a real-world scene with a computer-generated XR image. The XR headset can be configured to provide an augmented reality (AR) viewing environment by displaying a computer-generated XR image on a perspective display screen, which allows light from a real-world scene to pass through it for combined viewing by a user. Alternatively, the XR headset can be configured to provide a virtual reality (VR) viewing environment by preventing or substantially preventing light from a real-world scene from being directly viewed by a user when the user views a computer-generated AR image on a display screen. The XR headset can be configured to provide both AR and VR viewing environments. In one embodiment, both AR and VR viewing environments are provided by lateral bands of substantially different opacities arranged between the see-through display screen and the real-world scene, thereby providing a VR viewing environment for XR images aligned with the high opacity bands and providing an AR viewing environment for XR images aligned with the low opacity bands. In another embodiment, both AR and VR viewing environments are provided by control of a computer-adjustable opacity filter that variably constrains how much light from the real-world scene passes through the see-through display screen to be combined with the XR image viewed by the user. Therefore, the XR headset may also be referred to as an AR headset or a VR headset.

[0036] Figure 1 An embodiment of a surgical system 2 according to some embodiments of the present disclosure is depicted. Before performing an orthopedic or other surgical procedure, one may use, for example, Fig.10 C-arm imaging device 104 or Fig.11 A three-dimensional ("3D") image scan of the planned surgical area of ​​the patient is taken from an O-arm imaging device 106, or from another medical imaging device such as a computed tomography (CT) image or MRI. Such a scan may be performed preoperatively (e.g., most commonly a few weeks prior to the procedure) or intraoperatively. However, any known 3D or 2D image scan may be used in accordance with various embodiments of surgical system 2. The image scan is sent to a computer platform in communication with surgical system 2, such as computer platform 910 ( Fig. 9 ), the surgical system 900 may include a camera tracking system component 6, a surgical robot 4 (e.g., Figure 12), imaging devices (e.g., C-arm 104, O-arm 106, etc.), and an image database 950 for storing image scans of patients. View computer platform 910 ( Fig. 9 ) generates a surgical plan defining target poses for surgical tools used during a surgical procedure on a patient's anatomy. Exemplary surgical tools, also referred to as tools, may include, but are not limited to, drills, screwdrivers, retractors, and implants such as screws, spacers, intracorporeal fusion devices, plates, rods, etc. In some embodiments, the surgical plan defining the target plane is planned on the 3D image scan displayed on the display device.

[0037] As used herein, the term "pose" refers to the orientation and / or rotation angle of an object (e.g., a dynamic reference array, an end effector, a surgical tool, an anatomical structure, etc.) relative to another object and / or a defined coordinate system. Thus, a pose can be defined based solely on the multi-dimensional orientation of an object relative to another object and / or a defined coordinate system, solely on the multi-dimensional rotation angle of the object relative to another object and / or a defined coordinate system, or in combination with the multi-dimensional orientation and the multi-dimensional rotation angle. Thus, the term "pose" is used to refer to an orientation, a rotation angle, or a combination thereof.

[0038] Figure 1 The surgical system 2 can assist the surgeon during a medical procedure by, for example, providing for use of holding tools, alignment tools, application tools, guide tools, and / or positioning tools. In some embodiments, the surgical system 2 includes a surgical robot 4 and a camera tracking system assembly 6. The ability to mechanically couple the surgical robot 4 and the camera tracking system assembly 6 can allow the surgical system 2 to be manipulated and moved as a single unit, and allows the surgical system 2 to have a small footprint in an area, allowing for easier movement through narrow passages and around turns, and allowing for storage within a smaller area.

[0039] The surgical procedure can begin with the surgical system 2 being moved from a medical storage room to a medical procedure room. The surgical system 2 can be manipulated through doorways, halls, and elevators to reach the medical procedure room. Within the medical procedure room, the surgical system 2 can be physically divided into two separate and distinct systems (the surgical robot 4 and the camera tracking system assembly 6). The surgical robot 4 can be positioned adjacent to the patient in any suitable location to appropriately assist the medical staff. The camera tracking system assembly 6 can be positioned at the bottom of the patient, at the patient's shoulder, or at any other location suitable for tracking the current posture and posture movement of the surgical robot 4 and the tracked portion of the patient. The surgical robot 4 and the camera tracking system assembly 6 can be powered by an onboard power supply and / or plugged into an external wall socket.

[0040] The surgical robot 4 may be used to assist the surgeon by holding and / or using tools during a medical procedure. In order to properly utilize and hold the tools, the surgical robot 4 may rely on a number of motors, computers, and / or actuators to function properly. Figure 1 As depicted, the robot body 8 can serve as a structure in which multiple motors, computers, and / or actuators can be secured within the surgical robot 4. The robot body 8 can also provide support for the robotic telescopic support arm 16. The size of the robot body 8 can provide a stable platform to support attached components, and can house, conceal, and protect multiple motors, computers, and / or actuators that can operate the attached components.

[0041] The robot base 10 can act as a lower support for the surgical robot 4. In some embodiments, the robot base 10 can support the robot body 8 and can attach the robot body 8 to a plurality of drive wheels 12. This attachment to the wheels can allow the robot body 8 to move efficiently in space. The robot base 10 can run along the length and width of the robot body 8. The robot base 10 can be about two inches to about 10 inches high. The robot base 10 can cover, protect and support the drive wheels 12.

[0042] In some embodiments, Figure 1 As shown, at least one drive wheel 12 can be attached to the robot base 10. The drive wheel 12 can be attached to the robot base 10 at any position. Each individual drive wheel 12 can rotate around a vertical axis in any direction. A motor can be placed above, within, or adjacent to the drive wheel 12. Such a motor can allow the surgical system 2 to be manipulated to any position and stabilize and / or level the surgical system 2. A rod located within or adjacent to the drive wheel 12 can be pressed into a surface by the motor. The rod, not shown, can be made of any suitable metal to lift the surgical system 2. The rod can lift the drive wheel 10, which can lift the surgical system 2 to any height required to level or otherwise fix the surgical system 2 relative to the patient's orientation. The weight of the surgical system 2 is supported by a small contact area of ​​the rod on each wheel, preventing the surgical system 2 from moving during the medical procedure. This rigid positioning can prevent the object and / or person from accidentally moving the surgical system 2.

[0043] The robot track 14 can be used to facilitate moving the surgical system 2. The robot track 14 provides a person with the ability to move the surgical system 2 without grasping the robot body 8. Figure 1 As shown, the length of the robot track 14 can be the same length as the robot body 8, shorter than the robot body 8, and / or longer than the robot body 8. The robot track 14 can further provide protection to the robot body 8, thereby preventing the subject and / or medical personnel from contacting, hitting, or bumping into the robot body 8.

[0044] The robot body 8 may provide support for a selectively compliant articulated robotic arm, hereinafter referred to as a "SCARA". Due to the repeatability and compactness of the robotic arm, the use of the SCARA 24 within the surgical system 2 may be advantageous. The compactness of the SCARA may provide additional space within a medical procedure, which may allow a medical professional to perform a medical procedure without excessive clutter and restricted areas. The SCARA 24 may include a robotic telescopic support 16, a robotic support arm 18, and / or a robotic arm 20. The robotic telescopic support 16 may be positioned along the robot body 8. As shown in FIG. Figure 1 As shown, the robotic telescopic support 16 can provide support for the SCARA 24 and the display 34. In some embodiments, the robotic telescopic support 16 can extend and retract in a vertical direction. The body of the robotic telescopic support 16 can be any width and / or height configured to support the stress and weight placed thereon.

[0045] In some embodiments, the medical staff can move the SCARA 24 via commands submitted by the medical staff. As will be explained in further detail below, the commands can originate from a display 34, a tablet computer, and / or an XR headset (e.g., Fig. 9 The XR headset may eliminate the need for the medical staff to refer to any other display such as the display 34 or tablet, which enables the SCARA 24 to be configured without the display 34 and / or tablet. As will be explained in further detail below, the command may be generated by pressing a switch and / or pressing multiple switches, and / or may be generated based on gesture commands and / or voice commands sensed by the XR headset.

[0046] like Figure 5 As shown in , the activation assembly 60 may include a switch and / or multiple switches. The activation assembly 60 may be operable to transmit movement commands to the SCARA 24, thereby allowing the operator to manually manipulate the SCARA 24. When the switch or multiple switches are pressed, the medical staff may have the ability to move the SCARA 24 through the applied hand movement. Alternatively or additionally, as will be explained in further detail below, the operator may control the movement of the SCARA 24 through gesture commands and / or voice commands sensed by the XR headset. Additionally, when the SCARA 24 does not receive a command to move, the SCARA 24 may be locked in place to prevent accidental movement by medical staff and / or other objects. By locking in place, the SCARA 24 provides a stable platform through which the end effector 26 can guide surgical tools during medical procedures.

[0047] The robotic support arm 18 may be connected to the robotic telescopic support 16 by various mechanisms. Figure 1 and 2 , the robotic support arm 18 can rotate in any direction relative to the robotic telescopic support 16. The robotic support arm 18 can rotate three hundred and sixty degrees around the robotic telescopic support 16. The robotic arm 20 can be connected to the robotic support arm 18 in any suitable position and by various mechanisms that enable rotation in any direction relative to the robotic support arm 18. In one embodiment, the robotic arm 20 can rotate three hundred and sixty degrees relative to the robotic support arm 18. This free rotation allows the operator to position the robotic arm 20 according to the surgical plan.

[0048] Figure 4 and 5 The end effector 26 in the embodiment of the present invention is attached to the robotic arm 20 at any suitable position. The end effector 26 is configured to be attached to the end effector coupling 22 of the robotic arm 20 positioned by the surgical robot 4. The example end effector 26 includes a tubular guide that guides the movement of an inserted surgical tool relative to the anatomical structure on which a surgical procedure is to be performed.

[0049] In some embodiments, a dynamic reference array 52 is attached to the end effector 26. The dynamic reference array (also referred to herein as a "DRA") is a rigid body that can be placed on a patient's anatomy (e.g., bone), one or more XR headsets worn by personnel in the operating room, an end effector, a surgical robot, or a surgical tool in a guided surgical procedure. The computer platform 910 is configured to track the pose (e.g., position and rotational orientation) of the DRA in real time in combination with the camera tracking system assembly 6 or other 3D positioning system. The DRA may include a fiducial, such as the illustrated arrangement of a sphere. Such tracking of the 3D coordinates of the DRA may allow the surgical system 2 to determine the DRA's relative position to the patient's anatomy (e.g., bone). Figure 5 The posture of the target anatomical structure of the patient 50 in any multi-dimensional space.

[0050] like Figure 1 As shown, the light indicator 28 can be positioned on top of the SCARA 24. The light indicator 28 can be any type of light that illuminates to indicate the "condition" in which the surgical system 2 is currently operating. In some embodiments, the light can be generated by LED lights that form a ring around the light indicator 28. The light indicator 28 can include a completely permeable material that allows light to pass through the entire light indicator 28. The light indicator 28 can be attached to the lower display support 30. Figure 2As shown, the lower display support 30 can allow an operator to manipulate the display 34 to any suitable position. The lower display support 30 can be attached to the light indicator 28 by any suitable mechanism. In some embodiments, the lower display support 30 can rotate about the light indicator 28 or be rigidly attached to the light indicator. The upper display support 32 can be attached to the lower display support 30 by any suitable mechanism.

[0051] In some embodiments, the tablet computer can be used in conjunction with the display 34 and / or in the absence of the display 34. The tablet computer can be placed on the upper display support 32 in place of the display 34 and can be removed from the upper display support 32 during a medical procedure. In addition, the tablet computer can communicate with the display 34. The tablet computer can be connected to the surgical robot 4 via any suitable wireless and / or wired connection. In some embodiments, the tablet computer can program and / or control the surgical system 2 during a medical procedure. When the surgical system 2 is controlled with the tablet computer, all input and output commands can be replicated on the display 34. Using a tablet computer can allow the operator to manipulate the surgical robot 4 without having to move around the patient 50 and / or the surgical robot 4.

[0052] As will be explained below, in some embodiments, the surgeon and / or other personnel may wear an XR headset that may be used in conjunction with the display 34 and / or tablet, or the XR headset may eliminate the need to use the display 34 and / or tablet.

[0053] like Figure 3A and 5 As shown, the camera tracking system assembly 6 works in combination with the surgical robot 4 via a wired or wireless communication network. Figure 1 , 3A and 5, the camera tracking system assembly 6 may include some components similar to those of the surgical robot 4. For example, the camera body 36 may provide the functionality found in the robot body 8. The robot body 8 may provide an auxiliary tracking wand on which the camera 46 is mounted. The structure within the robot body 8 may also provide support for the electronic equipment, communication devices, and power supply used to operate the camera tracking system assembly 6. The camera body 36 may be made of the same material as the robot body 8. The camera tracking system assembly 6 may communicate directly with the XR headset, tablet computer, and / or display 34 via a wireless and / or wired network to enable the XR headset, tablet computer, and / or display 34 to control the functions of the camera tracking system assembly 6.

[0054] The camera body 36 is supported by a camera base 38. The camera base 38 may be used as the robot base 10. Figure 1In the embodiment of the present invention, the camera base 38 can be wider than the robot base 10. The width of the camera base 38 can allow the camera tracking system assembly 6 to be connected to the surgical robot 4. Figure 1 As depicted, the width of camera mount 38 may be large enough to fit outside of robotic base 10. The additional width of camera mount 38 may allow surgical system 2 to provide additional maneuverability and support for surgical system 2 when camera tracking system assembly 6 is coupled to surgical robot 4.

[0055] As with the robot base 10, a plurality of drive wheels 12 may be attached to the camera base 38. Similar to the operation of the robot base 10 and the drive wheels 12, the drive wheels 12 may allow the camera tracking system assembly 6 to be stabilized and leveled or set to a fixed orientation relative to the patient 50. Such stabilization may prevent the camera tracking system assembly 6 from moving during a medical procedure and may prevent the camera 46 on the auxiliary tracking wand from losing track of a DRA connected to the XR headset and / or the surgical robot 4, losing track of a DRA connected to the XR headset and / or the surgical robot 4, etc. Figure 3A and 5 The tracking of one or more DRAs 52 of the anatomical structure 54 and / or tool 58 within the designated area 56 is shown. This stability and maintenance of tracking enhances the ability of the surgical robot 4 to operate effectively with the camera tracking system assembly 6. Additionally, the wide camera base 38 can provide additional support for the camera tracking system assembly 6. Specifically, as Figure 3A and 5 As shown, the wide camera base 38 may prevent the camera tracking system assembly 6 from tipping over when the camera 46 is positioned over the patient.

[0056] The camera telescopic support 40 can support a camera 46 on an auxiliary tracking rod. In some embodiments, the telescopic support 40 can move the camera 46 higher or lower in a vertical direction. The camera handle 48 can be attached to the camera telescopic support 40 at any suitable location and is configured to allow an operator to move the camera tracking system assembly 6 to a planned orientation prior to a medical procedure. In some embodiments, the camera handle 48 can be used to lower and raise the camera telescopic support 40. The camera handle 48 can perform raising and lowering of the camera telescopic support 40 by pressing a button, a switch, a lever, and / or any combination thereof.

[0057] The lower camera support arm 42 may be attached to the camera telescoping support 40 at any suitable location, in embodiments such as Figure 1As shown, the lower camera support arm 42 can rotate three hundred and sixty degrees around the telescopic support 40. This free rotation can allow the operator to position the camera 46 in any suitable position. The lower camera support arm 42 can be connected to the telescopic support 40 by any suitable mechanism. The lower camera support arm 42 can be used to provide support for the camera 46. The camera 46 can be attached to the lower camera support arm 42 by any suitable mechanism. The camera 46 can pivot in any direction at the attachment area between the camera 46 and the lower camera support arm 42. In an embodiment, the curved track 44 can be disposed on the lower camera support arm 42.

[0058] The curved track 44 may be placed at any suitable location on the lower camera support arm 42. Figure 3A As shown, the curved track 44 can be attached to the lower camera support arm 42 by any suitable mechanism. The curved track 44 can be of any suitable shape, which can be a crescent, a circle, a flat, an ellipse, and / or any combination thereof. The camera 46 can be movably positioned along the curved track 44. The camera 46 can be attached to the curved track 44 by, for example, rollers, brackets, braces, motors, and / or any combination thereof. Motors and rollers, not shown, can be used to move the camera 46 along the curved track 44. As shown in FIG. Figure 3A As shown, during a medical procedure, if an object blocks the camera 46 from viewing one or more tracked DRAs, the motor can responsively move the camera 46 along the curved track 44. This motorized movement can allow the camera 46 to move to a new position that is no longer blocked by the object without moving the camera tracking system component 6. When the camera 46 is blocked from viewing one or more tracked DRAs, the camera tracking system component 6 can send a stop signal to the surgical robot 4, the XR headset, the display 34 and / or the tablet. The stop signal can prevent the SCARA 24 from moving until the camera 46 has reacquired the tracked DRA 52 and / or can warn the operator to wear the XR headset and / or view the display 34 and / or the tablet. The SCARA 24 can be configured to respond to the receipt of the stop signal by stopping further movement of the base and / or the end effector coupling 22 until the camera tracking system can resume tracking of the DRA.

[0059] Figure 3B and 3C Draws a picture that can be used with Figure 1 A front view and an isometric view of another camera camera tracking system assembly 6' that can be used together with a surgical system of a surgical robot or can be used independently of a surgical robot. For example, the camera tracking system assembly 6' can be used to provide guided surgery without using robotic guidance. Figure 3B and3C The camera tracking system component 6' and Figure 3A One of the differences between the camera tracking system components 6 is Figure 3B and 3C The camera tracking system assembly 6' includes a housing for transporting a computer platform 910. The computer platform 910 can be configured to: perform camera tracking operations to track the DRA; perform navigation surgery operations to provide surgical navigation information to a display device (e.g., an XR headset and / or other display device); and perform other computing operations disclosed herein. Thus, the computer platform 910 may include a navigation computer, such as Fig.14 One or more tour computers.

[0060] Figure 6 Depicted is a medical procedure Figure 5 A block diagram view of the components of a surgical system. Figure 6 , the navigation camera 46 on the auxiliary tracking wand has a navigation field of view 600 in which a reference array 602 attached to the patient, a reference array 604 attached to the surgical instrument, and the pose (e.g., position and orientation) of the robotic arm 20 are tracked. The navigation camera 46 may be a computer platform 910 configured to perform the operations described below. Figure 3B and 3C The reference array 602 is part of the camera tracking system assembly 6' of the surgical robot 4. The reference array achieves tracking by reflecting light in a known pattern, which is decoded by the tracking subsystem of the surgical robot 4 to determine the corresponding posture of the reference array. If the line of sight between the patient reference array 602 and the navigation camera 46 on the auxiliary tracking wand is blocked (e.g., by medical staff, instruments, etc.), further navigation of the surgical instruments may not be possible, and responsive notifications can temporarily stop further movement of the robotic arm 20 and the surgical robot 4, display a warning on the display 34, and / or provide an audible warning to the medical staff. The display 34 is available for use by the surgeon 610 and assistant 612, but viewing requires turning the head away from the patient and changing the eye focus to different distances and positions. The navigation software can be controlled by a technician 614 based on voice commands from the surgeon.

[0061] Figure 7 The figure shows how the surgical robot 4 can navigate in the navigation function of the surgical system 2. Figure 5 and 6The display screens may include, but are not limited to, a patient radiograph with an overlaid graphical representation of an instrument model positioned in the display screen relative to the anatomy based on the developed surgical plan and / or based on the pose of the tracked reference array, various user-selectable menus for controlling different stages of the surgical procedure, and dimensional parameters (e.g., length, width, and / or diameter) of a virtually projected implant.

[0062] For guided surgery, various processing components (e.g., computer platform 910) and associated software described below are provided that enable preoperative planning of a surgical procedure (e.g., implant placement) and electronic delivery of the plan to a computer platform 910 for providing navigation information to one or more users during the planned surgical procedure.

[0063] For robotic navigation, various processing components (e.g., computer platform 910) and associated software described below are provided that enable preoperative planning of a surgical procedure (e.g., implant placement) and electronic transfer of the plan to the surgical robot 4. The surgical robot 4 uses the plan to guide the robotic arm 20 and connected end effector 26 to provide target poses of the surgical tools relative to the patient's anatomy for the steps of the planned surgical procedure.

[0064] The following embodiments relate to the use of one or more XR headsets that can be worn by a surgeon 610, assistant 612 and / or other medical personnel to provide an improved user interface for receiving information from and / or providing control commands to a surgical robot, camera tracking system components 6 / 6' and / or other medical equipment in an operating room.

[0065] Figure 8 FIG. 4 is a block diagram showing some electrical components of the surgical robot 4 according to some embodiments of the present disclosure. Figure 8, a load cell (not shown) can be configured to track the force applied to the end effector coupling 22. In some embodiments, the load cell can communicate with multiple motors 850, 851, 852, 853 and / or 854. When the load cell senses a force, information about the amount of force applied can be distributed from the switch array and / or multiple switch arrays to the controller 846. The controller 846 can obtain the force information from the load cell and process it with a switching algorithm. The controller 846 uses the switching algorithm to control the motor driver 842. The motor driver 842 controls the operation of one or more motors 850, 851, 852, 853 and / or 854. The motor driver 842 can direct a specific motor to generate an equal amount of force, for example, measured by the load cell through the motor. In some embodiments, as indicated by the controller 846, the force generated can come from multiple motors, such as 850 to 854. Additionally, the motor driver 842 can receive input from the controller 846. The controller 846 can receive information about the direction of the force sensed by the force sensor from the force sensor. The controller 846 can use a motion controller algorithm to process this information. The algorithm can be used to provide information to a specific motor driver 842. In order to replicate the direction of the force, the controller 846 can activate and / or deactivate certain motor drivers 842. The controller 846 can control one or more motors, such as one or more of 850 to 854, to induce the movement of the end effector 26 in the direction of the force sensed by the force sensor. This force-controlled movement can allow the operator to move the SCARA 24 and the end effector 26 effortlessly and / or with very little resistance. The movement of the end effector 26 can be performed to position the end effector 26 in any suitable posture (i.e., relative to the position and angle orientation of the defined three-dimensional (3D) orthogonal reference axis) for use by medical personnel.

[0066] Figure 5 As best shown, the activation assembly 60 can be in the form of a bracelet that wraps around the end effector coupling 22. The activation assembly 60 can be located on any portion of the SCARA 24, any portion of the end effector coupling 22, can be worn by medical personnel (and communicate wirelessly), and / or any combination thereof. The activation assembly 60 can include a primary button and a secondary button.

[0067] Pressing the master button may allow the operator to move the SCARA 24 and the end effector coupling 22. According to one embodiment, once set in position, the SCARA 24 and the end effector coupling 22 may not move until the operator programs the surgical robot 4 to move the SCARA 24 and the end effector coupling 22, or uses the master button to move. In some instances, it may be necessary to press at least two non-adjacent master activation switches before the SCARA 24 and the end effector coupling 22 will respond to operator commands. Pressing at least two master activation switches may prevent accidental movement of the SCARA 24 and the end effector coupling 22 during a medical procedure.

[0068] Activated by the master button, the load cell can measure the magnitude and / or direction of the force applied by the operator (i.e., medical personnel) on the end effector coupling 22. This information can be transmitted to one or more motors within the SCARA 24, such as one or more of 850 to 854, which can be used to move the SCARA 24 and the end effector coupling 22. Information about the magnitude and direction of the force measured by the load cell can cause one or more motors (e.g., one or more of 850 to 854) to move the SCARA 24 and the end effector coupling 22 in the same direction as sensed by the load cell. This force-controlled movement can allow the operator to easily move the SCARA 24 and the end effector coupling 22, and because the motors move the SCARA 24 and the end effector coupling 22 while the operator moves the SCARA 24 and the end effector coupling 22, a lot of effort is not required.

[0069] In some examples, the operator can use the secondary button as a "selection" device. During the medical operation, the surgical robot 4 can notify the medical staff of certain situations through the XR headset 920, the display 34 and / or the light indicator 28. The XR headsets 920 are each configured to display images on a perspective display screen to form an extended reality image overlaid on a real-world object that can be viewed through the perspective display screen. The surgical robot 4 can prompt the medical staff to select functions, modes and / or evaluate the situation of the surgical system 2. Pressing the secondary button once can activate certain functions, modes and / or confirm information transmitted to the medical staff through the XR headset 920, the display 34 and / or the light indicator 28. In addition, pressing the secondary button multiple times in rapid succession can activate additional functions, modes and / or select information transmitted to the medical staff through the XR headset 920, the display 34 and / or the light indicator 28.

[0070] Further references Figure 8, the electrical components of the surgical robot 4 include a platform subsystem 802, a computer subsystem 820, a motion control subsystem 840, and a tracking subsystem 830. The platform subsystem 802 includes a battery 806, a power distribution module 804, a connector panel 808, and a charging station 810. The computer subsystem 820 includes a computer 822, a display 824, and a speaker 826. The motion control subsystem 840 includes a drive circuit 842, motors 850, 851, 852, 853, 854, stabilizers 855, 856, 857, 858, an end effector connector 844, and a controller 846. The tracking subsystem 830 includes an orientation sensor 832 and a camera converter 834. The surgical robot 4 may also include a removable foot pedal 880 and a removable tablet computer 890.

[0071] The input power is supplied to the surgical robot 4 through a power supply that can be provided to the power distribution module 804. The power distribution module 804 receives the input power and is configured to generate different power supply voltages provided to other modules, components and subsystems of the surgical robot 4. The power distribution module 804 can be configured to provide different voltage supplies to the connector panel 808, which can be provided to other components (such as a computer 822, a display 824, a speaker 826, a drive 842) to, for example, power the motors 850 to 854 and the end effector connector 844, and to the camera converter 834 and other components for the surgical robot 4. The power distribution module 804 can also be connected to a battery 806, which acts as a temporary power source when the power distribution module 804 does not receive power from the input power source. At other times, the power distribution module 804 can be used to charge the battery 806.

[0072] The connector panel 808 can be used to connect different devices and components to the surgical robot 4 and / or associated components and modules. The connector panel 808 may contain one or more ports for receiving lines or connectors from different components. For example, the connector panel 808 may have a grounding port for grounding the surgical robot 4 to other equipment, a port for connecting a foot pedal 880, and a port for connecting to a tracking subsystem 830, which may include an orientation sensor 832, a camera converter 834, and a DRA tracking camera 870. The connector panel 808 may also include other ports to allow USB, Ethernet, HDMI communication with other components (such as a computer 822). According to some embodiments, the connector panel 808 may include a wired and / or wireless interface for operably connecting the XR headset 920 to the tracking subsystem 830 and / or the computer subsystem 820.

[0073] The control panel 816 may provide various buttons or indicators for controlling the operation of the surgical robot 4 and / or providing information from the surgical robot 4 for the operator to observe. For example, the control panel 816 may include buttons for turning the surgical robot 4 on or off, raising or lowering the vertical column 16, and raising or lowering the stabilizers 855-858, which may be designed to engage the casters 12 to lock the surgical robot 4 without physically moving. Other buttons may stop the surgical robot 4 in the event of an emergency, which may remove all motor power and apply mechanical brakes to stop all movement from occurring. The control panel 816 may also have indicators (such as a line power indicator) that notify the operator of certain system conditions or the charge status of the battery 806. According to some embodiments, one or more XR headsets 920 may communicate, for example, via the connector panel 808 to control the operation of the surgical robot 4 and / or receive and display information generated by the surgical robot 4 for observation by a person wearing the XR headset 920.

[0074] The computer 822 of the computer subsystem 820 includes an operating system and software for operating the designated functions of the surgical robot 4. The computer 822 can receive and process information from other components (e.g., the tracking subsystem 830, the platform subsystem 802, and / or the motion control subsystem 840) to display information to the operator. Further, the computer subsystem 820 can provide output to the operator through the speaker 826. The speaker can be part of the surgical robot, part of the XR headset 920, or in another component of the surgical system 2. The display 824 can correspond to Figure 1 and 2 A display 34 is shown.

[0075] The tracking subsystem 830 may include an orientation sensor 832 and a camera converter 834. The tracking subsystem 830 may correspond to Figure 3A The camera tracking system assembly 6 of the DRA 52. The DRA tracking camera 870 operates in conjunction with the orientation sensor 832 to determine the pose of the DRA 52. Such tracking may be performed in a manner consistent with the present disclosure, including using infrared or visible light techniques, such as LEDs or reflective markers, to track the position of active or passive elements of the DRA 52, respectively.

[0076] The functional operations of the tracking subsystem 830 and the computer subsystem 820 may be included in Figure 3A and 3BThe computer platform 910 may be transported by the camera tracking system assembly 6'. The tracking subsystem 830 may be configured to determine a pose, such as the position and angular orientation of the tracked DRA. The computer platform 910 may also include a navigation controller configured to use the determined pose to provide navigation information to the user to guide them in moving the tracked tool relative to the position-registered patient image and / or the tracked anatomical structure during a planned surgical procedure. The computer platform 910 may be in a Figure 3B and 3C The computer platform 910 can be configured to communicate with the computer subsystem 820 and the computer subsystem 820 when used with a surgical robot. Figure 8 The computer platform 910 may communicate with other subsystems of the computer system to control movement of the end effector 26. For example, as will be explained below, the computer platform 910 may generate a graphical representation of a patient's anatomy, surgical tools, a user's hand, etc., having a displayed size, shape, color, and / or posture that is controlled based on a determined posture of one or more tracked DRAs, and the displayed graphical representation may be dynamically modified to track changes in the determined posture over time.

[0077] The motion control subsystem 840 can be configured to physically move the vertical column 16, the upper arm 18, the lower arm 20, or rotate the end effector coupling 22. The physical movement can be performed by using one or more motors 850 to 854. For example, the motor 850 can be configured to vertically raise or lower the vertical column 16. Figure 2 As shown, the motor 851 can be configured to move the upper arm 18 laterally about the point of engagement with the vertical column 16. Figure 2 As shown, motor 852 may be configured to move lower arm 20 laterally about a joint with upper arm 18. Motors 853 and 854 may be configured to move end effector coupling 22 to provide translational movement along and rotational movement about three-dimensional axes. Fig. 9 The illustrated computer platform 910 can provide control inputs to the controller 846 that direct the movement of the end effector coupling 22 to position the passive end effector connected thereto in a planned pose (i.e., position and angular orientation relative to defined 3D orthogonal reference axes) relative to the anatomical structure to be operated on during a planned surgical procedure. The motion control subsystem 840 can be configured to measure the orientation of the end effector coupling 22 and / or the end effector 26 using an integrated orientation sensor (e.g., an encoder).

[0078] Fig. 9A block diagram of components of a surgical system according to some embodiments of the present disclosure is depicted, the surgical system including a device connected to a camera tracking system component 6 ( Figure 3A ) or 6'( Figure 3B , Figure 3C ) and / or an imaging device of the computer platform 910 of the surgical robot 4 (e.g., C-arm 104, O-arm 106, etc.). Alternatively, at least some of the operations performed by the computer platform 910 disclosed herein may additionally or alternatively be performed by components of the surgical system.

[0079] refer to Fig. 9 , the computer platform 910 includes a display 912, at least one processor circuit 914 (also referred to as a processor for brevity), at least one memory circuit 916 (also referred to as a memory for brevity) containing computer readable program code 918, and at least one network interface 902 (also referred to as a network interface for brevity). The display 912 may be part of an XR headset 920 according to some embodiments of the present disclosure. The network interface 902 may be configured to connect to Fig.10 The C-arm imaging device 104, Fig.11 The O-arm imaging device 106, another medical imaging device, an image database 950 containing medical images of patients, components of the surgical robot 4 and / or other electronic devices.

[0080] When used with the surgical robot 4, the display 912 may correspond to Figure 2 The display 34 and / or Figure 8 The tablet computer 890 and / or the XR headset 920 operably connected to the surgical robot 4, the network interface 902 may correspond to Figure 8 The platform network interface 812, and the processor 914 may correspond to Figure 8 The network interface 902 of the XR headset 920 can be configured to communicate according to one or more wireless communication protocols (e.g., WLAN, 3GPP 4G and / or 5G (New Radio) cellular communication standards, etc.) through a wired network (e.g., thin-wire Ethernet) and / or through a wireless RF transceiver link.

[0081] The processor 914 may include one or more data processing circuits, such as general and / or special purpose processors, such as microprocessors and / or digital signal processors. The processor 914 is configured to execute computer readable program code 918 in the memory 916 to perform operations, which may include some or all of the operations described herein as being performed for surgical planning, guided surgery, and / or robotic surgery.

[0082] The computer platform 910 can be configured to provide surgical planning functionality. The processor 914 can be operated to display images of anatomical structures (e.g., vertebrae) received from one of the imaging devices 104 and 106 and / or from the image database 950 via the network interface 920 on the display device 912 and / or on the XR headset 920. The processor 914 receives an operator's definition of a location of an anatomical structure shown in one or more images where a surgical procedure (e.g., screw placement) is to be performed, such as by the operator touching and selecting a location on the display 912 for a planned surgical procedure or using a mouse-based cursor to define a location for a planned surgical procedure. As will be explained in further detail below, when an image is displayed in the XR headset 920, the XR headset can be configured to sense gesture-based commands formed by the wearer and / or sense voice-based commands spoken by the wearer, which commands can be used to control selection between menu items and / or control how objects are displayed on the XR headset 920.

[0083] The computer platform 910 can be configured to be able to perform anatomical measurements that may be particularly useful for knee surgery, similar to the measurement of various angles to determine the center of the hip, the center of the angle, natural landmarks (e.g., the transepicondylar line, the white side line, the posterior condylar line, etc.). Some measurements can be automatic, but some other measurements may involve human input or assistance. The computer platform 910 can be configured to allow the operator to input the selection of the correct implant for the patient, including the selection of size and alignment. The computer platform 910 can be configured to perform automatic or semi-automatic (involving human input) segmentation (image processing) of CT images or other medical images. The patient's surgical plan can be stored in a cloud-based server that can correspond to the database 950 for retrieval by the surgical robot 4.

[0084] For example, during orthopedic surgery, a surgeon can select a site to be cut (e.g., posterior femur, proximal tibia, etc.) using a computer screen (e.g., touch screen) or extended reality (XR) interaction (e.g., gesture-based commands and / or voice-based commands) through, for example, an XR headset 920. The computer platform 910 can generate navigation information that provides visual guidance to the surgeon to perform the surgical procedure. When used with the surgical robot 4, the computer platform 910 can provide guidance that allows the surgical robot 4 to automatically move the end effector 26 to a target pose so that the surgical tool is aligned with the target position to perform a surgical procedure on the anatomical structure.

[0085] In some embodiments, surgical system 900 can use two DRAs to track patient anatomical orientation, such as a DRA connected to the patient's tibia and a DRA connected to the patient's femur. System 900 can use standard navigation instruments for registration and inspection (e.g., a pointer similar to the pointer used in the Globus Excelsius GPS system for spinal surgery).

[0086] A particularly challenging task in guided surgery is how to plan the position of implants in the spine, knee, and other anatomical structures, where the surgeon strives to perform the task on a computer screen, which is a 2D representation of the 3D anatomical structure. The system 900 can solve this problem by using an XR headset 920 to display a three-dimensional (3D) computer-generated representation of the anatomical structure and the candidate implant device. Under the guidance of the computer platform 910, the computer-generated representations are scaled and posed relative to each other on the display screen, and can be manipulated by the surgeon while viewing through the XR headset 920. The surgeon can, for example, use gesture-based commands and / or voice-based commands sensed by the XR headset 920 to manipulate the displayed computer-generated representations of the anatomical structure, implants, surgical tools, etc.

[0087] For example, the surgeon can watch the displayed virtual handle on the virtual implant, and can manipulate (e.g., grasp and move) the virtual handle to move the virtual implant to the desired posture and adjust the planned implant placement relative to the graphical representation of the anatomical structure. Afterwards, during surgery, the computer platform 910 can display the navigation information that helps the surgeon to more accurately follow the surgical plan to insert the implant and / or perform another surgical procedure on the anatomical structure through the XR headset 920. When the surgical procedure involves bone removal, the progress (e.g., cutting depth) of the bone removal can be displayed in real time through the XR headset 920. Other features that can be displayed by the XR headset 920 can include but are not limited to gaps or ligament balance along the range of joint motion, contact lines along the range of joint motion on the implant, ligament tension and / or laxity rendered by color or other graphics, etc.

[0088] In some embodiments, the computer platform 910 can allow planning for the use of standard surgical tools and / or implants, such as posterior-stabilized implants and cruciate-retaining implants, cemented and cementless implants, and revision systems for surgeries associated with, for example, total or partial knee and / or hip replacements and / or trauma.

[0089] The automated imaging system may be used in conjunction with the computer platform 910 to acquire pre-operative, intra-operative, post-operative, and / or real-time image data of anatomical structures. Fig.10and Fig.11 An example automatic imaging system is depicted. In some embodiments, the automatic imaging system is a C-arm 104 ( Fig.10 ) imaging device or 106( Fig.11 ). (O Copyrighted by Medtronic Navigation, Inc., having its place of business in Louisville, Colorado, USA). It may be desirable to take x-rays of a patient from a number of different positions without requiring frequent manual repositioning of the patient as may be required in an x-ray system. C-arm 104 x-ray diagnostic devices can solve the problem of frequent manual repositioning and are well known in the medical field for surgery and other interventional procedures. Fig.10 As depicted, the C-arm includes an elongated C-shaped member terminating in a "C" shaped opposite distal end 112. The C-shaped member is attached to an X-ray source 114 and an image receptor 116. The space within the C-arm 104 of the arm provides space for a physician to care for a patient substantially uninterrupted by the X-ray support structure.

[0090] The C-arm is mounted so that the arm can be rotationally moved in two degrees of freedom (i.e., about two perpendicular axes in a spherical motion). The C-arm is slidably mounted to the X-ray support structure, which allows the C-arm to perform orbital rotational movement about its center of curvature, which can allow selective vertical and / or horizontal orientation of the X-ray source 114 and image receptor 116. The C-arm can also be rotatable laterally (i.e., in a perpendicular direction relative to the direction of orbital motion to enable selective adjustment of the positioning of the X-ray source 114 and image receptor 116 relative to the width and length of the patient). The spherical rotational aspect of the C-arm device allows the physician to X-ray the patient at an optimal angle determined relative to the specific anatomy being imaged.

[0091] Fig.11 O shown in 106 includes a gantry housing 124, which can enclose an image capture portion, not shown. The image capture portion includes an X-ray source portion and / or an X-ray emitting portion and an X-ray receiving portion and / or an image receiving portion, which can be positioned approximately one hundred and eighty degrees from each other and mounted on a rotor (not shown) relative to the track of the image capture portion. The image capture portion can be operatively rotated three hundred and sixty degrees during image acquisition. The image capture portion can rotate about a center point or axis, thereby allowing image data of the patient to be acquired from multiple directions or in multiple planes.

[0092] O with stand housing 124 The gantry housing 106 has a central opening for positioning around the object to be imaged, a radiation source that can be rotated around the interior of the gantry housing 124, and the radiation source can be adapted to project radiation from a plurality of different projection angles. The detector system is adapted to detect the radiation at each projection angle, thereby acquiring an image of the object from a plurality of projection planes in a quasi-simultaneous manner. The gantry can be attached to the support structure O in a cantilevered manner. A support structure, such as a wheeled mobile cart with wheels. The positioning unit preferably translates and / or tilts the gantry to a planned position and orientation under the control of a computerized motion control system. The gantry may include a source and a detector disposed relative to each other on the gantry. The source and the detector may be secured to a motorized rotor that can rotate the source and the detector in combination with each other around the interior of the gantry. The source may be pulsed in multiple positions and orientations in a partial and / or full three hundred and sixty degree rotation to perform multi-planar imaging of a target object positioned inside the gantry. The gantry may further include a track and bearing system for guiding the rotor as it rotates, the track and bearing system may carry the source and the detector. Both and / or one of the surgical system 106 and the C-arm 104 may be used as an automated imaging system to scan the patient and send the information to the surgical system 2 .

[0093] The images captured by the imaging system may be displayed on the XR headset 920 and / or another display device of the computer platform 910, the surgical robot 4, and / or another component of the surgical system 900. The XR headset 920 may be connected to one or more of the imaging devices 104 and / or 106 and / or the image database 950, for example, through the computer platform 910, to display images therefrom. A user may provide control input (e.g., gesture-based and / or voice-based commands) through the XR headset 920 to control the operation of one or more of the imaging devices 104 and / or 106 and / or the image database 950.

[0094] Fig.12 A block diagram of components of a surgical system including a pair of XR headsets 1200 and 1210 (head mounted displays HMD1 and HMD2) is depicted, wherein the XR headsets may correspond to Fig.13 The XR headset 920 shown in FIG. 1 and operates according to some embodiments of the present disclosure.

[0095] refer to Fig.12In the exemplary scenario of FIG. 6 , both assistant 612 and surgeon 610 wear XR headsets 1200 and 1210, respectively. It is optional for assistant 612 to wear XR headset 1210. As will be further described below, XR headsets 1200 and 1210 are configured to provide an interactive environment through which the wearer can view and interact with information related to the surgical procedure. This interactive XR-based environment can eliminate the need for the presence of technician 614 in the operating room and can eliminate the need for the use of Figure 6 Each XR headset 1200 and 1210 may include one or more cameras configured to provide an additional source of tracking DRA or other reference arrays attached to instruments, anatomical structures, end effector 26, and / or other devices. Fig.12 In the example, the XR headset 1200 has a field of view (FOV) 1202 for tracking the DRA and other objects, the XR headset 1210 has a FOV 1212 for tracking the DRA and other objects that partially overlaps with the FOV 1202, and the navigation camera 46 has another FOV 600 for tracking the DRA and other objects that partially overlaps with the FOVs 1202 and 1212.

[0096] If one or more cameras are obstructed from viewing a DRA attached to a tracked object (e.g., a surgical instrument), but the DRA is in view of one or more other cameras, the tracking subsystem 830 and / or navigation controller 828 can continue to seamlessly track the object without losing navigation. Additionally, if a DRA is partially obscured from the perspective of one camera, but the entire DRA is visible through multiple camera sources, the tracking inputs of the cameras can be merged to continue navigating the DRA. One of the XR headsets and / or the navigation camera 46 can view and track the DRA on another of the XR headsets to enable the computer platform 910 ( Fig. 9 and Fig.14 ), the tracking subsystem 830 and / or another computing component is capable of determining the pose of the DRA relative to one or more defined coordinate systems, such as the XR headset 1200 / 1210, the navigation camera 46 and / or another coordinate system defined for the patient, table and / or room.

[0097] XR headsets 1200 and 1210 may be operably connected to view video, pictures, and / or other information received from the operating room and / or provide commands to control various equipment in the operating room, including but not limited to neuromonitoring, microscopes, cameras, and anesthesia systems. Data from the various devices may be processed and displayed within the headset, such as displaying a patient's vital signs or a microscope feed.

[0098] Exemplary XR Headset Assemblies and Integration with Guided Surgery, Surgical Robots, and Other Devices

[0099] Fig.13 An XR headset 920 configured in accordance with some embodiments of the present disclosure is depicted. The XR headset includes a headband 1306 configured to secure the XR headset to a wearer's head, an electronics housing 1304 supported by the headband 1306, and a display screen 1302 extending laterally and downwardly from the electronics housing 1304. The display screen 1302 may be a see-through LCD display device or a semi-reflective lens that reflects an image projected by the display device toward the wearer's eyes. A set of DRA fiducials, such as points, are drawn or attached to one or both sides of the headset in a known manner spaced apart. The DRA on the headset enables a navigation camera on an auxiliary tracking stick to track the pose of the headset 920 and / or enables another XR headset to track the pose of the headset 920.

[0100] The display screen 1302 operates as a see-through display screen (also referred to as a combiner) that reflects light from the display panel of the display device toward the user's eyes. The display panel can be positioned between the electrical component housing and the user's head and angled to project virtual content toward the display screen 1302 to reflect toward the user's eyes. The display screen 1302 is semi-transparent and semi-reflective, allowing the user to see the reflected virtual content overlaid on the user's view of the real world scene. The display screen 1302 can have different opacity areas, such as the upper side band depicted having a higher opacity than the lower side band. The opacity of the display screen 1302 can be electronically controlled to adjust how much light from the real world scene passes through the user's eyes. The high opacity configuration of the display screen 1302 causes a high contrast virtual image to be overlaid on a dim view of the real world scene. The low opacity configuration of the display screen 1302 can cause a more blurred virtual image to be overlaid on a clearer view of the real world scene. The opacity can be controlled by applying an opaque material on the surface of the display screen 1302.

[0101] According to some embodiments, the surgical system includes an XR headset 920 and an XR headset controller, e.g. Fig.14 Controller 1430 or Fig.343410 in the controller. The XR headset 920 is configured to be worn by a user during a surgical procedure and has a see-through display screen 1302 that is configured to display an XR image and allow at least a portion of a real-world scene to pass through it for viewing by the user. The XR headset 920 also includes an opacity filter that is positioned between at least one of the user's eyes and the real-world scene when the see-through display screen 1302 is viewed by the user. The opacity filter is configured to provide opacity to light from the real-world scene. The XR headset controller is configured to communicate with a navigation controller (e.g., Fig.14 One or more controllers 828A, 828B and / or 828C) in the operative room communicate to receive navigation information from a navigation controller to provide guidance to the user during a surgical procedure performed on the anatomical structure, and is further configured to generate XR images based on the navigation information for display on the perspective display screen 1302.

[0102] The opacity of the display screen 1302 may be configured as a gradient with a more continuously varying opacity as the distance moves downward from the top of the display screen 1302. The darkest point of the gradient may be located at the top portion of the display screen 1302 and gradually become less transparent further down the display screen 1302 until the opacity is transparent or non-existent. In a further exemplary embodiment, the gradient may change from about 90% opacity to fully transparent at approximately mid-eye level of the display screen 1302. With the headset properly calibrated and placed, the mid-eye level may correspond to the point at which the user will be looking straight ahead, and the end of the gradient will be positioned at the "horizontal" line of the eye. The darker portion of the gradient will allow for clear and unambiguous vision of the virtual content and help block the intrusive brightness of the overhead operating room lights.

[0103] Using an opacity filter in this manner enables the XR headset 920 to provide virtual reality (VR) functionality along the upper portion of the display screen 1302 and AR functionality along the middle or lower portion of the display screen 1302 by substantially or completely blocking light from the real-world scene. This allows the user to have semi-transparent AR where it is needed and allows the use of clear optics for the patient's anatomy during procedures. Configuring the opacity filter 1302 as a gradient rather than a more constant opacity band can allow the wearer to experience a more natural transition between a more VR-type view and a more AR-type view without experiencing abrupt changes in the brightness of the real-world scene and depth of field that can otherwise fatigue the eyes, such as during more rapid shifts between upper and lower views.

[0104] The display panel and display screen 1302 can be configured to provide a wide field of view perspective XR display system. In one example configuration, they provide a user with an 80° diagonal field of view (FOV) and a 55° vertical coverage for viewing virtual content. Other diagonal FOV angles and vertical coverage angles can be provided by display panels of different sizes, lenses of different curvatures, and / or different distances and angular orientations between the display panel and the curved display screen 1302. In some embodiments, an XR headset with a wide field of view allows the XR headset to display a real-world registered XR image to the user, while the user's field of view is not centered on the real-world coordinates associated with the real-world registered XR image.

[0105] Fig.14 Electrical components of an XR headset 920 according to some embodiments of the present disclosure are depicted, and the XR headset can be operably connected to a computer platform 910, one or more of the imaging devices (such as a C-arm imaging device 104, an O-arm imaging device 106) and / or an image database 950 and / or a surgical robot 800.

[0106] The XR headset 920 provides an improved human-machine interface for performing guided surgical procedures. The XR headset 920 can be configured to provide functionality, for example, via the computer platform 910, including but not limited to any one or more of the following: recognition of gesture-based commands and / or voice-based commands, display of XR graphical objects on the display device 1450. The display device 1450 can be a video projector, a flat panel display, etc. that projects the displayed XR graphical objects onto the display screen 1302. The user can use the XR graphical objects as an overlay anchored to a specific real-world object viewed through the display screen 1302 ( Fig.13 ). The XR headset 920 may additionally or alternatively be configured to display video feeds from cameras mounted to one or more of the XR headsets 920 and other cameras on the display screen 1450.

[0107] The electrical components of the XR headset 920 may include a plurality of cameras 1440, a microphone 1442, a gesture sensor 1444, a posture sensor (e.g., an inertial measurement unit (IMU)) 1446, a display module 1448 including a display device 1450, and a wireless / wired communication interface 1452. As described below, the cameras 1440 of the XR headset may be visible light capture cameras, near infrared capture cameras, or a combination of both.

[0108] The camera 1440 may be configured to operate as a gesture sensor 1444 by capturing for recognition user gestures performed within the field of view of the camera 1440. Alternatively, the gesture sensor 1444 may be a proximity sensor and / or a touch sensor that senses gestures performed in proximity to the gesture sensor 1444 and / or senses physical contact (e.g., tapping the sensor or housing 1304). The posture sensor 1446 (e.g., an IMU) may include a multi-axis accelerometer, a tilt sensor, and / or another sensor that may sense rotation and / or acceleration of the XR headset 920 along one or more defined coordinate axes. Some or all of these electrical components may be contained in the component housing 1304 or may be contained in another housing that is configured to be worn elsewhere, such as on the hip or shoulder.

[0109] As described above, the surgical system 2 includes a camera tracking system component 6 / 6' and a tracking subsystem 830 that may be part of a computer platform 910. The surgical system may include an imaging device (e.g., a C-arm 104, an O-arm 106, and / or an image database 950) and / or a surgical robot 4. The tracking subsystem 830 is configured to determine the pose of a DRA attached to an anatomical structure, an end effector, a surgical tool, etc. The navigation controller 828 is configured to determine a target pose of the surgical tool relative to the anatomical structure based on the surgical plan, for example, based on the position of the surgical tool relative to the anatomical structure. Fig. 9 The surgical planning function performed by the computer platform 910 of the surgical robot defines the position of the surgical tool on the anatomical structure to perform the surgical procedure based on the pose of the anatomical structure determined by the tracking subsystem 830. The navigation controller 828 can be further configured to generate steering information based on the target pose of the surgical tool, the pose of the anatomical structure, and the pose of the surgical tool and / or the end effector, wherein the steering information indicates where the surgical tool and / or the end effector of the surgical robot should be moved to perform the surgical plan.

[0110] The electrical components of the XR headset 920 may be operably connected to the electrical components of the computer platform 910 via the wired / wireless interface 1452. The electrical components of the XR headset 920 may be operably connected, for example, via the computer platform 910 or directly connected via the wired / wireless interface 1452 to various imaging devices (e.g., C-arm imaging device 104, I / O-arm imaging device 106), image database 950, and / or other medical equipment.

[0111] The surgical system 2 also includes at least one XR headset controller 1430 (also referred to as an "XR headset controller" for brevity), which may reside in the XR headset 920, the computer platform 910, and / or another system component connected via a wired cable and / or a wireless communication link. Various functions are provided by software executed by the XR headset controller 1430. The XR headset controller 1430 is configured to receive navigation information from the navigation controller 828 to provide guidance to the user during a surgical procedure performed on an anatomical structure, and is configured to generate an XR image based on the navigation information displayed on the display device 1450 for projection on the perspective display screen 1302.

[0112] The configuration of the display device 1450 relative to the display screen (also referred to as a “see-through display screen”) 1302 is configured to display the XR images in such a way that the XR images appear to be in the real world when a user wearing the XR headset 920 looks through the display screen 1302. The display screen 1302 may be positioned in front of the user's eyes by a headband 1306.

[0113] The XR headset controller 1430 may be within a housing configured to be worn on the user's head or other part of the user's body when viewing the display screen 1302, or may be positioned away from the user viewing the display screen 1302 when communicatively connected to the display screen 1302. The XR headset controller 1430 may be configured to operably process signaling from the camera 1440, the microphone 142, and / or the gesture sensor 1446, and connect, for example, through the display module 1302 to display an XR image on the display screen 1450. Thus, the XR headset controller 1430, depicted as a circuit block within the XR headset 920, is understood to be operably connected to the other depicted components of the XR headset 920, but does not necessarily reside within a common housing (e.g., the electrical component housing 13 of Figure 1304) or otherwise be portable by the user. For example, the XR headset controller 1430 may reside within the computer platform 910, which in turn may reside within the Figure 3B and Figure 3C The computer tracking system 6' is shown within the housing.

[0114] Example user view through an XR headset

[0115] The XR headset operation can display 2D images and 3D models on the display screen 1302. The 2D image can be preferably displayed in the more opaque band of the display screen 1302 (upper band), and the 3D model can be more preferably displayed in the more transparent band of the display screen 1302 otherwise referred to as the environment area (bottom band). Below the lower band where the display screen 1302 ends, the wearer can have an unobstructed view of the operating room. Note that the XR content displayed on the display screen 1302 can be fluid. The 3D content may move to the opaque band depending on the orientation of the headset relative to the content, and the 2D content can be placed in the transparent band and stabilized relative to the real world. Additionally, the entire display screen 1302 can be dimmed under electrical control to convert the headset into a virtual reality for surgical planning or completely transparent during a medical procedure. As explained above, the XR headset 920 and associated operations not only support guided procedures, but can also be performed in conjunction with robotic-assisted procedures.

[0116] Fig.16 An exemplary view through a display screen 1302 of an XR headset 920 for providing navigation assistance to a user manipulating a surgical tool 1602 during a medical procedure is shown in accordance with some embodiments of the present disclosure. Fig.16 , when the surgical tool 1602 is brought near the tracked anatomical structure so that the reference arrays 1630 and 1632 connected to the surgical tool 1602 are in the camera 1440 ( Fig.15 ) and 46( Figure 6 ), a graphical representation of a tool 1600 may be displayed in 2D and / or 3D images relative to a graphical representation of an anatomical structure 1610. The user may use the viewed graphical representation to adjust a trajectory 1620 of a surgical tool 1602, which may be depicted extending from the graphical representation of the tool 2000 through the graphical representation of the anatomical structure 1610. The XR headset 920 may also display textual information and other objects 1640. A dashed line 1650 extending across the viewed display screen represents an exemplary division between upper and lower bands of different opacity levels.

[0117] Other types of XR images (virtual content) that may be displayed on display screen 1302 may include, but are not limited to, any one or more of the following:

[0118] I) 2D axial, sagittal and / or coronal views of the patient’s anatomy;

[0119] 2) an overlay of planned tool and surgical implant locations over currently tracked tool and surgical implant locations;

[0120] 3) Preoperative image library;

[0121] 4) Video feeds from microscopes and other similar systems or remote video conferencing;

[0122] 5) Options and configuration settings and buttons.

[0123] 6) A floating 3D model of the patient’s anatomy with surgical planning information;

[0124] 7) Real-time tracking of surgical instruments relative to the floating patient anatomy;

[0125] 8) Enhanced overlay of patient anatomy with instructions and guidance; and

[0126] 9) Enhanced overlay of surgical equipment.

[0127] Example configuration of cameras for tracking system components

[0128] Fig.17 An exemplary configuration of an auxiliary tracking stick 46 with two pairs of stereo navigation cameras configured according to some embodiments of the present disclosure is depicted; the auxiliary tracking stick 46 is Figure 3A , Figure 3B and Figure 3C The stereo navigation camera is a part of the camera tracking system assembly. According to one embodiment, the stereo navigation camera includes a pair of stereo spaced visible light capture cameras and another pair of stereo spaced near infrared capture cameras. Alternatively, only a pair of visible light capture cameras or only a pair of near infrared capture cameras can be used in the auxiliary tracking rod 46. Any multiple near infrared and / or visible light cameras can be used.

[0129] Posture measurement chain

[0130] As described above, guided surgery can include computer vision tracking and determining the pose (e.g., position and orientation in a six-degree-of-freedom coordinate system) of surgical instruments, such as by determining the pose of a DRA that includes spaced-apart fiducials, such as disks or spheres, arranged in a known manner. The computer vision uses spaced-apart navigation cameras, such as stereo cameras, configured to capture near-infrared and / or visible light. In this scenario, there are three parameters competing for optimization: (1) accuracy, (2) robustness, and (3) user ergonomics during the surgical procedure.

[0131] Some further aspects of the present disclosure relate to computer operations that combine (chain) measured gestures in a manner that can improve optimization of one or more of the above three parameters by incorporating additional navigation cameras mounted to one or more XR headsets. Fig.17As shown, according to some embodiments of the present disclosure, a pair of stereo visible light tracking cameras and another pair of stereo near infrared tracking navigation cameras can be attached to an auxiliary tracking wand of a camera tracking system assembly. An operating algorithm is disclosed that analyzes the posture of a DRA that is fully or partially observed (e.g., when not all fiducials of the DRA are observed by a pair of stereo cameras) and combines the observed postures or partial postures in a manner that can improve accuracy, robustness, and / or ergonomics during guided surgery.

[0132] As described above, the XR headset can be configured to enhance the real world scene with a computer-generated XR image. The XR headset can be configured to provide an XR viewing environment by displaying a computer-generated XR image on a perspective display screen, which allows light from the real world scene to pass through it for combined viewing by the user. Alternatively, the XR headset can be configured to provide a VR viewing environment by preventing or substantially preventing light from the real world scene from being directly viewed by the user along the viewing path of the displayed XR image. The XR headset can be configured to provide AR and VR viewing environments. In one embodiment, both AR and VR viewing environments are provided by lateral bands of substantially different opacities arranged between the perspective display screen and the real world scene, thereby providing a VR viewing environment for XR images aligned with the high opacity band, and providing an AR viewing environment for XR images aligned with the low opacity band. In another embodiment, both AR and VR viewing environments are provided by control of a computer-adjustable opacity filter that variably constrains how much light from the real-world scene passes through the see-through display screen to be combined with the XR image viewed by the user. Thus, the XR headset may also be referred to as an AR headset or a VR headset.

[0133] Also as described above, the XR headset may include a near infrared tracking camera and / or a visible light tracking camera configured to track a fiducial of a DRA connected to surgical instruments, patient anatomy, other XR headsets, and / or a robotic end effector. The use of near infrared tracking and / or visible light tracking on an XR headset provides additional tracking volume coverage beyond that provided by a camera on a single auxiliary tracking wand. Adding a near infrared tracking camera to an existing auxiliary tracking wand allows for more robust but less accurate tracking of the headset position than in visible light. Mechanically calibrating the visible and near infrared tracking coordinate systems enables the coordinate systems to be sufficiently aligned to perform a 3D DRA fiducial triangulation operation using stereo matching to jointly identify the pose of the DRA fiducial between the visible and near infrared tracking coordinate systems. The use of visible and near-infrared tracking coordinate systems can enable any one or more of the following: (a) identification of tools that cannot be identified using a single coordinate system; (b) improved pose tracking accuracy; (c) a wider range of motion without losing track of surgical instruments, patient anatomy, and / or robotic end effectors; and (d) natural tracking of the XR headset in the same coordinate system as the surgical instruments being navigated.

[0134] Generate XR images based on the pose of the XR headset in real-world coordinates

[0135] Reference now Figures 18 to 26 The exemplary embodiments of describe various operations that may be performed by surgical system 2 during a medical procedure to provide navigation assistance for surgical tools through an XR headset.

[0136] For guided surgery, some surgeons use a technique that involves taking multiple scans of the patient's anatomy and using the images represented on a 2D screen to navigate their tools to the correct position and orientation. The surgical tools can be tracked and displayed as an overlay on the scanned images, allowing the surgeon to see the position and orientation (also called posture) of the tools relative to the anatomy with the accuracy allowed by the tool tracking system without physically exposing the patient's anatomy (i.e., through minimally invasive surgery ("MIS")). This process can allow for minimally invasive surgery, however, these techniques can introduce additional difficulties.

[0137] In some instances, freehand and robotically guided surgery may require interaction with a 2D display (e.g., a computer monitor). The display can be used to view scanned patient images, plan implant placement, and control the robot in the case of robotically guided surgery. A sterile drape may be placed over a touch screen monitor or computer mouse interface, preventing precise and smooth physical control. In addition, if the monitor is outside the surgeon's field of view while looking at the patient, the surgeon may be forced to repeatedly look away from the surgical site in order to view the image.

[0138] In additional or alternative examples, the navigation tool can use 2D stitched images from the scans to display, such as axial and sagittal views. However, navigating several 2D images simultaneously may be unintuitive and difficult for new surgeons to learn. Although the process can use CT and X-ray scans to create reconstructions of the scanned anatomical structures and generate computerized 3D models, it may be impractical to use a 2D display to navigate, view, and manipulate the 3D model.

[0139] The use of XR displays in surgical settings can allow patient information to be displayed in 3D on the real world. However, displaying XR content in the context of surgical navigation presents additional challenges. For example, motion sickness is a common problem when using large field of view XR displays, but this problem can be minimized by careful planning of the user interface. Large-scale head-registered content is known to be a common cause of motion sickness and can be avoided by using world-registered or object-registered content as described in various embodiments herein.

[0140] In addition, if Figures 18 to 22 As shown, using patient registration to replace real (AR) content through an XR headset is inherently prone to posture errors. In addition to the errors caused by tool tracking (one of the most important sources of error in current surgical displays), errors can also be introduced by tracking the XR headset, the calibration of the headset optics, and the posture of the headset relative to the surgeon's line of sight. If visual occlusions are not tracked, the imaging may not properly occlude objects behind the display and may interfere with the surgeon's vision of the surgical site. In addition, if the patient anatomy is not exposed or is occluded by the XR content, errors in the imaging may not be immediately obvious, which may cause the surgeon to over-rely on incorrect information.

[0141] Fig.18 Depicted are examples of 2D screens that a surgeon may reference to help position tools and / or implants during a surgical procedure. Fig.19 Draws the Fig.18By tracking the position of the physical tool relative to the patient, the surgeon's tool position can be displayed relative to the scanned imaging. The display can depict the displayed position of the tool 1810 and the ideal position of the implant at position 1820. However, the tracking may not be perfect, and errors introduced by such imperfections may cause the displayed position of the tool to shift. Figure 18 to Figure 19 In the example of FIG. 18 , the tool may actually be at position 1850 relative to the anatomy, and the resulting implant may be at position 1860 relative to the anatomy. If the pose given by tool tracking is incorrect by an error offset 1890, the tool in display 1810 will be offset by an error offset 1890 relative to the actual tool position 1850. Because the surgeon uses the offset display to guide their instrument to the implant location, the implant will be slightly misplaced by an error offset 1890 from where it should ideally appear in the surgeon's plan.

[0142] Figure 20 to Figure 22 It shows that Figures 18 to 19 and another view of the corresponding surgical procedure. Navigating the surgery using an XR display still requires tool tracking, which produces an error offset 1890. However, other sources of error, such as errors in tracking the XR headset and calibrating the XR display, cumulatively produce another sizable offset represented by the XR display offset error 2092. When viewing patient-aligned or anatomy-aligned XR content, the XR display offset error 2092 causes the XR content 2070 to be visually displaced from the actual position 2030 of the anatomy. The XR content obstructs the surgeon's view of the patient, so they place their instruments at their true positions 2110 relative to the XR content. Additionally, guidance is provided based on the tracked tool positions, so the total difference between the ideal implant position and the actual resulting implant position can be as much as the sum of the error offset 1890 and the XR display offset error 2092.

[0143] If world-registered XR content is used, as disclosed herein, the XR display offset error 2092 still exists. However, because the XR content is separated from the patient and does not obstruct the surgeon's view, the XR display offset error 2092 does not affect the surgeon's tool placement or the guidance display relative to the scanned anatomy. Instead, the difference between the ideal implant position 1820 and the resulting implant position 1860 is limited to the error offset 1890, the same as current surgical guide display methods.

[0144] The various embodiments described herein use Fig.233D reconstructed anatomical model view of an XR display as shown. Rather than providing navigation and interaction in a head-registered or patient-registered manner, information is shown relative to a world-registered 3D model of the patient's anatomy 2320 that "floats" above the patient. Combining the XR display 2310 with the ability to track hand movements adds another level of interactivity. In addition to displaying virtual tools 2333 and virtual implant locations relative to the virtual patient anatomy 2320, the XR 3D anatomy view can be interactive and used to control a surgical robot.

[0145] exist Fig.23 , a surgeon 22390 navigates during spinal surgery using a 3D anatomical view, depicted from the perspective of another person across the operating table using an XR headset. A patient 2380 has been scanned and the images have been used to construct a 3D model of the spine 2320, as shown in the XR area 2310. The surgical plan specifies the ideal tool positions, which are displayed in the floating model. A 3D model representation of the surgeon's tools 2335 is also displayed as a virtual tool 2333 relative to the spine model 2320, which is based on the tracked physical position of the real-world tool 2335 at the real-world implant location 2322. The real-world XR image can include a virtual guide 2330 that indicates the desired position of the virtual tool 2333 relative to the virtual anatomical structure 2320.

[0146] After acquiring medical imaging from the patient, a 3D model reconstruction can be created. This model is presented in the user interface of the XR display. As shown in XR, the model is world-registered (meaning that when the wearer of the display moves their head, the model remains in place) and is positioned roughly half a meter above the patient by default. Fusion of hand tracking with the XR system allows the user to interact with the 3D model using their hands and perform gestures. This level of interaction allows the model to be used as a control interface for planning, navigation, and cross-platform hardware control. The following are some examples of possible applications for interacting with anatomical views.

[0147] In some embodiments, the surgeon may interact with the anatomical view to determine stabilization options. Stabilization options may include whether the view is patient-stabilized (the content moves when the patient moves) or world-stabilized (the content remains stationary if the patient moves).

[0148] In additional or alternative embodiments, the surgeon may interact with the anatomical view to determine view adjustments. Figure 24 to Figure 25 It is depicted that the surgeon can use their hands to rotate the model to get a better view from other angles or to unfold the model to see more detail without tilting it.

[0149] In additional or alternative embodiments, the surgeon can interact with the anatomical view to determine the placement of the implant. In addition to the existing 2D slice views and the manipulation provided by GPS and similar software, surgical implants can be planned or adjusted in 3D.

[0150] In additional or alternative embodiments, the surgeon can interact with the anatomical view to control tool navigation. A 3D representation of the tracked surgical tool is displayed in the display in scale and pose relative to the patient's anatomy, and an indicator can guide the surgeon to correct the position as specified in the plan.

[0151] In additional or alternative embodiments, the surgeon may interact with the anatomical view to control the surgical robot. Fig.26 It is depicted that in a robotically guided surgery, the surgeon 2390 can use hand gestures to select individual implants 2610, 2620, 2626, 2630 on the XR 3D anatomical view 2310 to guide the robot 2690 to move to the correct position to place the implants 2612, 2622, 2632. In this example, the virtual implant 2610 corresponds to the completed real-world implant 2612; the virtual implant 2620 corresponds to the real-world implant 2622 currently being positioned, the virtual implant 2630 corresponds to the next real-world implant 2632; and the virtual implant 2626 that has been planned as part of the surgical procedure.

[0152] In additional or alternative embodiments, the surgeon can interact with the anatomical view to record or view an overview of the surgical procedure. Fig.26 As shown, the surgeon can use gestures to select implants on the anatomical view and mark them as completed, thereby providing the surgeon with a visual representation of their overall progress.

[0153] In additional or alternative embodiments, the surgeon can interact with the anatomical view to select a 2D scan. For example, the surgeon can view a gallery of 2D images organized by the scan position along the 3D model and select them using gestures.

[0154] Figure 24 to Figure 25 It is shown that using hand tracking 3D anatomical views can become interactive. Surgeons can use their hands to control and manipulate the view of the patient's anatomy to view the model in more detail. Fig.24 By selecting a portion of the view, the surgeon can expand this area. Depending on the context, this expanded view can be used to plan implant placement or navigate the surgery in more detail. Fig.25 In the 3D image, surgeons can rotate and expand the view with intuitive gestures.

[0155] Fig.26The surgeon is depicted using their hands to select the next planned implant location in the XR 3D anatomical view, which signals the surgical robot to move to the next location. The real-world XR image also includes a progress overlay that indicates which implants have been completed, which implant sites are currently being operated on, and which have not yet been started. For example, completed virtual implants may be displayed in black, virtual implants currently being operated on may be displayed in dark gray, and virtual implants that have not yet been started may be displayed in white.

[0156] Using XR 3D anatomy views for navigation, planning, and as interactive menus provides several advantages. In some instances, using a 3D model derived from a scan of the patient instead of a generic model provides an accurate view of the patient that can be used for surgical planning and navigation. This also means that any 3D model derived from any kind of medical imaging can be used, so the view can ultimately be used for any type of surgery. The view can be generated before or during surgery.

[0157] In additional or alternative embodiments, keeping the content world-registered prevents errors in the world-registered or object-registered surrogate real content from affecting the accuracy of XR-guided surgery. This effectively allows the system to rely on accuracy from tool tracking just as it would for a 2D monitor display.

[0158] In additional or alternative embodiments, the immersion provided by XR and the ability to have a 3D view will allow surgeons to navigate in a more intuitive manner compared to current methods that use multiple 2D views, which is particularly important for training new surgeons to perform navigated surgeries.

[0159] In additional or alternative embodiments, using world-registered content with world stabilization or patient stabilization tracking rather than head-registered stabilization allows the surgeon to use their head to lean forward to get a better view of the model without having to manually manipulate it. This also allows anyone else watching the surgery with their own XR display to watch the surgeon's interaction with their own XR display, and reduces the chance that a user will experience motion sickness associated with large head-stabilized / registered displays.

[0160] In additional or alternative embodiments, because the content is world-registered rather than patient-registered, the surgeon can manipulate and move the model to a personalized position that makes them comfortable and select desired stabilization options. The surgeon can also use intuitive gestures to rotate or expand the view itself as needed and easily return to a default or personalized position.

[0161] In additional or alternative embodiments, providing a 3D view above the patient, within the field of view of the surgical site allows the surgeon to view imaging and navigation in a more comfortable manner and prevents them from needing to look away from the patient. Fig.13 An XR headset with a curved display 1302) allows the surgeon to maintain a 3D view in the peripheral field of vision.

[0162] In additional or alternative embodiments, because the imaging is located above the surgical site, from a "multi-person" perspective, the surgeon can freely point to and explain the patient anatomy to the observer wearing the XR display with a lower chance of accidentally hitting or contacting an unsterile surface or patient.

[0163] In additional or alternative embodiments, the surgeon may plan, navigate, and control the surgical robot through a single intuitive display, significantly improving workflow.

[0164] In additional or alternative embodiments, interaction with the XR 3D anatomical view in XR does not require physically touching the display, reducing sterility concerns and eliminating difficulties caused by interacting with a touch display or a mouse / keyboard with a drape over it.

[0165] Extended reality instrument interaction area

[0166] Reference now FIG. 27A to FIG. 32 The exemplary embodiments of describe various operations that may be performed by surgical system 2 during a medical procedure to provide navigation assistance for surgical tools through an XR headset.

[0167] In some embodiments, by combining motion tracking of surgical tools with head and hand tracking in extended reality, "interaction zones" can be used to hide, show, disable, and interact with elements of the augmented reality UI. These interaction zones can be portions of the 3D tracking volume that can be predetermined or defined during surgery using tracked instrument and hand tracking controls. Using these interaction zones, the surgeon can, for example, define areas during surgery where specific instruments should not be allowed, disable or hide the UI to prevent unintentional actions, interact with UI elements using surgical instruments, and ignore tracking data for tools and hands from elsewhere in the OR.

[0168] In some embodiments, the interaction zone can be defined in real time. The surgeon or medical professional can move the monitoring marker relative to the patient and view its position in the XR headset. The position of this monitoring marker can be tracked and updated. A default box can be defined. Hand tracking controls can be used to modify the position. A fine-tuning control dial can also be used to adjust the lock position. The headset wearer can repeat this procedure until the headset wearer is satisfied and edit it again later. The headset wearer will also have alternating hand controls to reorder the lock position into different boxes. One lock point can be part of more than one bounding box to define a joint interaction zone. Lock points can be ordered into multiple boxes to define multiple interaction zones. Controls can be used to clear this area. Controls can be used to define which instruments should interact with the interaction zone.

[0169] In additional or alternative embodiments, the interactive zones may be preconfigured by software using relative measurements. For example, a user interface ("UI") may only appear or allow interaction when a hand or tool is above a certain height relative to the head. The default interactive zones may be adjusted in the UI settings. Once zones are defined, they may be hidden, obscured, or viewed contextually, depending on type and user settings.

[0170] Various embodiments herein describe the use of interactive zones during surgical procedures. Although interactive zones are often described individually, an XR headset can display multiple interactive zones that are independent and / or overlapping. The following are just a few examples that surgeons can use.

[0171] In some embodiments, the interactive zone may include a navigation zone. The extended reality display may allow for minimally invasive, freehand, and robotic guided surgery. By creating a navigation zone near the patient, the surgeon may be guided to the appropriate position and orientation using the model in the extended reality according to the surgical plan, or a new plan may be created based on the current tool trajectory and position. The navigation zone may only allow tools within the zone to be navigated and used as a trigger to initiate a navigation display in the extended reality. This also means that tools detected outside the navigation zone do not need to be accurately or completely tracked, which may improve the performance of computer vision-based surgical tool tracking.

[0172] FIG. 27A to FIG. 27B An example of a navigation area used with a 3D XR anatomy 2750 associated with a real-world surgical procedure 2752 is depicted. Real-world tools 2754 are present within the real-world navigation area 2790, such as Fig.27B As shown, it does not exist in the real world guide area 2790. Therefore, Fig.27AThe XR display in FIG. 27 is a planning mode XR image 2710 and depicts a virtual planning tool 2712 at a planned location. In response to the tracking system detecting that the real-world tool 2754 is located within the navigation area 2790, the planning mode XR image 2710 is replaced by the navigation mode XR image 2730, as shown in FIG. Fig.27B In the navigation mode XR image 2730 , a virtual tool 2760 is displayed, which corresponds to the real-world tool 2754 and the virtual planning tool 2712 to help move the real-world tool 2754 and the defined navigation area 2710 .

[0173] In some embodiments, the interactive zone may include a transition zone. Extended reality displays may cause problems with displaying too much information, which is particularly problematic in settings with large field of view displays and where immediate action is required in sensitive working areas such as operating tables. It is important for surgeons to be able to work without being hindered by UI elements. Specific areas can be used to automatically destroy all UI elements when tools or hands are brought into it, like a failsafe. The UI can be restored by removing tracked items from this area or by using another area. Similarly, transition zones can be used to display or hide data based on context, such as the aforementioned summoning of the navigation view and hiding the plan view when tools are brought into the surgical field. (The navigation zone can be considered a specialized transition zone).

[0174] In some embodiments, the interactive zone includes an exclusion zone. It may be critical that certain areas of the operating environment are always clearly visible and never have XR coverage. The exclusion zone prevents UI elements from obstructing the view of the zone when viewed from any angle.

[0175] In some embodiments, the interaction zone includes a warning zone. If certain areas in the surgical space are off-limits to the tracked surgical robot or surgical tools, a warning zone can be defined. If one of these instruments crosses one of the bounding boxes containing the interaction zone, one or more configurable warning events will occur. When used with a surgical robot, the warning zone created by its motion projection path can be displayed in 3D space in extended reality. When a tool, hand, or tracked headset obstructs the robot's projected path, the robot can also be forced to stop its movement, adding another degree of safety. If the interaction zone is crossed, the type of alarm or warning that will be issued is also configurable. It can include a beep, XR occlusion, or text warning in the headset and monitor.

[0176] FIG. 28A to FIG. 28B A warning zone 2880 defined by the surgeon is shown. Fig.28B When the warning area 2880 is displayed, a warning 2892 is displayed in the XR UI 2890 that the surgeon can dismiss.

[0177] FIG. 29A to FIG. 29B The real-world robotic arm 2930 and the future position 2940 of the robotic arm 2930 are communicated, as well as a suggested path 2932 along which the real-world robotic arm 2930 may move to reach the future position. Fig.29B In the example, the tool 2920 is detected to move into the suggested path 2930, and a warning message 2992 is added to the XR UI 2990. In addition, when the tool enters the warning zone while the robot is in the middle of its motion, the robot stops or determines a new path to the future location.

[0178] In some embodiments, the interaction zone may include a tool selection zone. A designated area of ​​the surgical tracking volume may be used to select which tools should be focused by the UI. The surgeon may select the tool they are currently using by placing a tool in this zone and having the surgeon confirm the tool (either through the UI or even by simply pointing their head towards the tool for a moment). The extended reality display may then filter out tool information that is not relevant to the currently selected tool and perform robotic calculations based on the geometry of the selected tool. This simple tool selection zone ensures that the surgeon only receives relevant data and keeps the extended reality area clean.

[0179] exist FIG. 30A to FIG. 30C An example of the tool selection process is shown in Fig. 30A , the surgeon passes their tool 3024 to their assistant. However, the UI 3090 still displays the data for this tool 3024 and does not display the surgeon's new tool 3022. Fig. 30B In , the surgeon brings the new tool 3022 into the selection area 3020 and waits for the circle around the check mark to complete, or uses a gesture to confirm the selection. Fig. 30C , the new tool 3022 has been registered. The UI data 3090 now shows the surgeon's tool 3022 and filters out the data from the other tool 3024.

[0180] In some embodiments, the interactive area includes a UI area. When tool tracking is used with an extended reality display, the UI itself can be changed by using the tool. The surgical tool can be used as a pen to write notes in 3D space, activate UI elements, place 3D models of screws and other surgical implants, rotate or move existing 3D models, or change shaders, among other uses.

[0181] FIG. 31A to FIG. 31B The 3D model 3110 generated using a scan of the patient is shown with its own UI area 3190. As the surgeon passes through this area with tools 3120, they can "peel off" the external shader and instead make the 2D scan into cross-sections 3150. They can do this from any angle as a way to quickly classify different cross-sections of the anatomy.

[0182] exist Fig.32 In another example of a UI area shown, a surgeon creates a UI area 3290 for sterile note taking during surgery. A surgical tool 3210 can be used as a virtual pen when brought into the defined UI area 3290. Tracking trigger points on the tool are used to indicate when to apply "ink" or "lift off" between words. Additionally, the tool can be used to select an image icon 3250 to trigger a camera on the XR headset to record an image that can be tied to notes. These assistants can help surgeons construct post-operative notes.

[0183] An interaction zone is a portion of a tracking volume that can be used to trigger an augmented reality UI change or issue a robotic command. Some embodiments describe a method for defining a 3D interaction zone and attribute its use to the method.

[0184] In some embodiments, creating an interaction zone from a tracking volume allows for contextual interactions / changes in an XR interface or robotic control scheme. Using hand and tool tracking allows interaction zones to be defined and modified in a simple manner in a sterile environment. Interaction zones can be defined by the user using tracked surgical instruments or hands, so no additional tools are required in the operating room.

[0185] In additional or alternative embodiments, the interactive zones may be defined by the XR software in context with respect to the tracked object (e.g., placing a navigation zone near a tracked patient reference point or a warning zone near a robot end effector). The interactive zones may be modified at any time, such as by displaying the zones in XR and using tone points on the edges.

[0186] In additional or alternative embodiments, the interactive zone is practical as a safety precaution in a surgical environment, allows for intuitive UI use, and can keep the UI area clean and the surgeon's view of the patient clear.

[0187] In some embodiments, the interactive zone can have various applications, and the type of interactive zone determines its behavior. For example, the navigation zone allows the UI to provide navigation guidance to the user in context. It also assists the object tracking algorithm by optimizing accuracy only for tools in the navigation zone. When the tracked object is inside the patient, the transition zone enables the user to clearly see the patient while providing rich tools and functions by exposing, hiding, or changing the UI. They also limit the possibility of the surgeon inadvertently activating the UI while working. The prohibition zone prevents the XR user interface from obstructing the surgeon's view of critical areas. The warning zone provides the surgeon with an additional level of perception, avoids unintentional contact with the patient's body, and ensures that the movement path of the surgical robot is unobstructed. For example, in freehand surgery, when an instrument is placed in a dangerous position, the surgeon has little way to get feedback. The warning zone allows immediate and easy reading of feedback. The tool selection zone ensures that the UI remains clean when tracking multiple tools. The UI zone allows for simple and intuitive use of the UI, allowing extended reality to fully realize its potential for surgical guidance.

[0188] Fig.33 3300. As shown, the surgical system 3300 includes an XR headset 3320; a computer platform 910; a camera tracking system 6, 6'; an imaging device; and a surgical robot 800. The computer platform 910 includes a tracking subsystem 830, a navigation controller 828, an XR headset controller 3330, and an interactive zone controller 3390. The tracking subsystem 830 can track the real-world posture of the XR headset 3320, the surgical robot 800, and other real-world elements relative to the real-world coordinate system based on data from the camera tracking system 6, 6'. The navigation controller 828 can determine the difference between the real-world posture of a real-world element such as a surgical tool and the planned position of the real-world element. The XR headset controller 3330 can generate a real-world XR image for display by the XR headset based on the real-world posture of the XR headset, so that the real-world XR image is bound to the real-world coordinate system. The interactive zone controller 3390 can determine the pose and type of the interactive zone relative to the real-world coordinate system and provide information about the interactive zone to the XR headset controller 3330 for determining when the surgeon is interacting with the interactive zone. In some embodiments, the surgical system may include only a portion of these features. For example, the surgical system may include an XR headset 3320, a tracking subsystem 830, and an XR headset controller 3330.

[0189] Fig.3434 is a flow chart of a process performed by a surgical system (e.g., surgical system 3300). At block 3410, the tracking subsystem 830 determines a real-world pose of a real-world element relative to a real-world coordinate system. In some embodiments, the tracking subsystem determines the real-world pose of the XR headset and the real-world pose of the real-world element.

[0190] At block 3420, the XR headset controller 3330 defines a portion of the real-world coordinate system as an interactive zone. In some embodiments, the headset controller defines the interactive zone as one of a navigation zone, a transition zone, a warning zone, a prohibition zone, a tool selection zone, and a user interface ("UI") zone.

[0191] In some embodiments, the surgical system further includes an interactive zone controller configured to define a size, a posture, and / or a type of the interactive zone based on a type of real-world element and / or a real-world posture.

[0192] In block 3430, the XR headset controller determines whether a real-world element is within the interaction zone.

[0193] In box 3440, the XR headset controller 3330 determines user input based on the tracking data of the real-world gesture of the real-world element.

[0194] At block 3450, the XR headset controller 3330 generates a world-registered XR image based on the real-world pose of the XR headset. In some embodiments, the XR headset controller generates a world-registered XR image based on the real-world pose of the XR headset and the real-world pose of the real-world element. The world-registered XR image includes virtual elements generated based on characteristics of the real-world element. The virtual elements are posed based on the real-world pose of the XR headset and are displayed on a see-through display screen within the user's field of view. In some embodiments, the XR headset includes a display with a wide field of view (e.g., Fig.13 ) which allows the world-registered XR image to continue to be displayed while the surgeon is looking in a slightly different direction.

[0195] In additional or alternative embodiments, the XR headset controller is configured to generate a world-registered XR image based on the real-world pose of the XR headset, wherein virtual elements of the world-registered XR image are displayed on a perspective display screen and the virtual elements are maintained at a constant pose relative to a real-world coordinate system regardless of the real-world pose of the XR headset.

[0196] In additional or alternative embodiments, the XR headset controller is configured to generate a world-registered XR image based on a real-world pose of the XR headset, wherein virtual elements of the world-registered XR image are displayed on a perspective display screen and the virtual elements have a pose that maintains a constant offset from the real-world pose of the real-world element regardless of the real-world pose of the XR headset.

[0197] In additional or alternative embodiments, the XR headset controller is further configured to determine user input based on the tracking data of the real-world pose of the real-world element. The XR headset controller is further configured to generate a world-registered XR image further based on the user input.

[0198] In additional or alternative embodiments, the XR headset controller is configured to respond to determining that a real-world pose of the real-world element is within the interaction zone by generating a virtual element within the world-registered XR image.

[0199] At block 3460, the XR headset controller 3330 displays the world-registered XR image. In some embodiments, the world-registered XR image is displayed on a see-through display screen of an XR headset configured to be worn by a user during a surgical procedure. The see-through display screen may be configured to display the world-registered XR image and allow at least a portion of a real-world scene to pass through it for viewing by the user.

[0200] In some embodiments, the XR headset controller 3330 is further configured to communicate with a navigation controller to receive navigation information from the navigation controller, the navigation information providing guidance to the user during a surgical procedure on an anatomical structure. The XR headset controller is configured to generate a world-registered XR image based on the navigation information to further include user interface ("UI") data for display on the see-through display screen.

[0201] In an additional or alternative embodiment, the interaction zone is a navigation zone. The tracking system is further configured to determine the real-world pose of the real-world element only when the real-world pose of the real-world element is within the navigation zone. The XR headset controller is configured to generate a world-registered XR image to contain UI data associated with the virtual element corresponding to the real-world element in response to determining that the real-world element is within the navigation zone.

[0202] In additional or alternative embodiments, the interaction zone is a transition zone. The XR headset controller is configured to generate the world-registered XR image to further exclude a portion of the UI data in response to determining that the real-world element is within the transition zone. The XR headset controller is configured to generate the world-registered XR image to further include a portion of the UI data in response to determining that the real-world element is outside of the transition zone.

[0203] In an additional or alternative embodiment, the interaction zone is a warning zone. The navigation controller is further configured to determine that the warning zone includes a projection path of motion of the robotic arm in a real-world coordinate system. The navigation controller is further configured to determine whether the real-world element is within the warning zone. In response to determining that the real-world element is within the warning zone, the robotic arm is prevented from moving along the projection path. The XR headset controller is configured to generate a world-registered XR image such that a virtual path associated with the projection path of motion of the robotic arm is displayed. The XR headset is further configured to output a warning to the user in response to the real-world element being within the warning zone.

[0204] In additional or alternative embodiments, the interaction zone is an exclusion zone, and the XR headset controller is configured to generate a world-registered XR image such that a user's view of the exclusion zone is not obstructed by the XR image.

[0205] In an additional or alternative embodiment, the interaction zone is a tool selection zone. The tracking system is further configured to determine whether to select and / or deselect the first real-world element based on determining that the first real-world element is within the tool selection zone. The tracking system is configured to determine a real-world pose of only the selected real-world element outside of the tool selection zone.

[0206] In additional or alternative embodiments, the interaction zone is a user interface (“UI”) zone. The XR headset controller is further configured to modify the world-registered XR image based on the real-world gesture of manipulating the real-world element in response to determining that the real-world gesture of the real-world element is within the UI zone.

[0207] In an additional or alternative embodiment, the XR headset further comprises a gesture sensor configured to output an indication of a gesture formed by the user to the XR headset controller. The XR headset controller is further configured to define and / or modify a size, posture, and / or type of the interaction zone based on the indication of the gesture formed by the user.

[0208] In an additional or alternative embodiment, the surgical system further comprises a surgical robot and a navigation controller. The surgical robot comprises a robot base, a robot arm connected to the robot base, and at least one motor. The robot arm is configured to position an end effector, the end effector being configured to guide the movement of a surgical tool. The at least one motor is operatively connected to move the robot arm relative to the robot base. The navigation controller is configured to determine a target pose of the surgical tool based on a surgical plan and a pose based on an anatomical structure, wherein the surgical plan defines the position on the anatomical structure where the surgical procedure will be performed using the surgical tool. The navigation controller is also configured to generate steering information based on the target pose of the surgical tool, the pose of the anatomical structure, and the pose of the surgical tool. The steering information indicates the position to which the surgical tool needs to move under the control of the at least one motor.

[0209] In an additional or alternative embodiment, the XR headset is a first XR headset configured to be worn by a first user during a surgical procedure and includes a see-through display screen configured to display a first world-registered XR image and allow at least a first portion of a real-world scene to pass through it for viewing by the first user. The first world-registered XR image includes a first virtual element corresponding to a real-world element. The surgical system may also include a second XR headset configured to be worn by a second user during a surgical procedure and includes a see-through display screen configured to display a second world-registered XR image and allow at least a second portion of a real-world scene to pass through it for viewing by the second user. The second world-registered XR image includes a second virtual element corresponding to a real-world element. The tracking system is also configured to determine a real-world pose of the first XR headset and a real-world pose of the second XR headset. The XR headset controller includes at least one XR headset controller and is further configured to generate a second world-registered XR image based on a real-world pose of a second XR headset. The first world-registered XR image and the second world-registered XR image are different.

[0210] With respect to some embodiments of surgical systems and related methods, Fig.34 Various operations of the flowcharts may be optional.

[0211] Additional Definitions and Examples:

[0212] In the above description of each embodiment of the inventive concept, it should be understood that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the inventive concept. Unless otherwise defined, all terms used herein (including technical terms and scientific and technological terms) have the same meanings commonly understood by those skilled in the art to which the inventive concept belongs. It should be further understood that the terms as defined in the commonly used dictionary should be interpreted as having the meaning consistent with its meaning in the context of this specification and in the relevant field, and are clearly defined in this article, and will not be interpreted in an idealized or overly formal sense.

[0213] When an element is referred to as being "connected to" or "coupled to" "in response to" another element or a variant thereof, it may be directly connected to, coupled to or responsive to another element, or there may be an intermediate element. In contrast, when an element is referred to as being "directly connected to", "directly coupled to", "directly responsive to" another element or a variant thereof, there is no intermediate element. Similar elements are referred to with similar numbers throughout the text. In addition, "connection", "connection", "response" or its variants as used herein may include wireless connection, connection or response. Unless the context clearly indicates otherwise, as used herein, the singular forms "a, an" and "the" are intended to include plural forms. For the sake of brevity and / or clarity, well-known functions or configurations may not be described in detail. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0214] It should be understood that, although the terms first, second, third, etc. can be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish an element / operation from another element / operation. Therefore, without departing from the teaching of the present invention concept, the first element / operation in some embodiments can be referred to as the second element / operation in other embodiments. Throughout this specification, the same reference numerals or the same figure designators represent the same or similar elements.

[0215] As used herein, the terms "comprise / comprising / comprises", "include / including / includes", "have / has / having" or variations thereof are open ended and include one or more stated features, integers, elements, steps, components, or functions, but do not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or combinations thereof. In addition, as used herein, the general abbreviation "such as", derived from the Latin phrase "exempli gratia", may be used to introduce or specify one or more general examples of previously mentioned items and is not intended to limit such items. The general abbreviation "that is", derived from the Latin phrase "id est", may be used to specify a specific item from a more general statement.

[0216] Example embodiments are described herein with reference to block diagrams and / or flow diagrams of computer-implemented methods, devices (systems and / or apparatuses), and / or computer program products. It should be understood that the blocks of the block diagrams and / or flow diagrams and the combination of blocks in the block diagrams and / or flow diagrams can be implemented by computer program instructions executed by one or more computer circuits. These computer program instructions can be provided to a processor circuit of a general-purpose computer circuit, a special-purpose computer circuit, and / or other programmable data processing circuit to produce a machine so that instructions executed by a processor of a computer and / or other programmable data processing device convert and control transistors, values ​​stored in memory locations, and other hardware components within such circuit systems to implement the functions / actions specified in the block diagrams and / or one or more flow diagram blocks, and thereby create a device (function) and / or structure for implementing the functions / actions specified in the block diagrams and / or one or more flow diagram blocks.

[0217] These tangible computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing device to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture including instructions for implementing the functions / actions specified in the block diagram or one or more flowchart blocks. Thus, embodiments of the inventive concept may be embodied in hardware and / or software (including firmware, resident software, microcode, etc.) that runs on a processor such as a digital signal processor, which may be collectively referred to as a "circuitry system," "module," or variations thereof.

[0218] It should also be noted that, in some alternative implementations, the function / action marked in the frame may not occur in the order marked in the flow chart. For example, according to the function / action involved, the two frames shown continuously can actually be performed substantially simultaneously, or the frame can sometimes be performed in reverse order. In addition, the function of a given frame of a flow chart and / or block diagram can be divided into multiple frames, and / or the function of two or more frames of a flow chart and / or block diagram can be integrated at least in part. Finally, without departing from the scope of the present invention concept, other frames can be added / inserted between the illustrated frames, and / or frame / operation can be omitted. In addition, although some figures in the figure include arrows on the communication path to illustrate the main direction of communication, it should be understood that communication can occur in the direction opposite to the arrows depicted.

[0219] Without substantially departing from the principle of the inventive concept, many changes and modifications may be made to the embodiments. All such variants and changes are intended to be included in the scope of the inventive concept herein. Therefore, the subject matter disclosed above should be considered illustrative rather than restrictive, and the examples of the attached embodiments are intended to cover all such modifications, enhancements and other embodiments that fall within the spirit and scope of the inventive concept. Therefore, in order to be allowed to the greatest extent by law, the scope of the inventive concept will be determined by the most widely allowed interpretation of the present disclosure, which includes the examples of the following embodiments and their equivalents, and should not be limited to or confined to the aforementioned specific detailed description.

Claims

1. A surgical system, comprising: an extended reality headset configured to be worn by a user during a surgical procedure and comprising a see-through display screen configured to display a world-registered extended reality image and to allow at least a portion of a real-world scene to pass through for viewing by the user; as well as a tracking system configured to determine a real-world pose of the extended reality headset and a real-world pose of a real-world element, the real-world pose of the extended reality headset and the real-world pose of the real-world element being determined relative to a real-world coordinate system; as well as an extended reality headset controller configured to generate the world-registered extended reality image based on the real-world pose of the extended reality headset and the real-world pose of the real-world element, wherein the world-registered augmented reality image has virtual elements displayed on the see-through display screen, the virtual elements being maintained at a constant pose relative to the real-world coordinate system regardless of the real-world pose of the augmented reality headset to allow the user to simultaneously see a real patient and a world-registered 3D image floating near the patient, wherein the augmented reality headset controller is further configured to communicate with a navigation controller to receive navigation information from the navigation controller, the navigation information providing guidance to the user during the surgical procedure on the anatomical structure, and Wherein the extended reality headset controller is configured to generate the world-registered extended reality image based on the navigation information to further include user interface data for display on the see-through display screen.

2. The surgical system of claim 1 , wherein the virtual element is generated based on characteristics of the real-world element, wherein the virtual element is posed based on the real-world pose of the augmented reality headset to be displayed on the see-through display screen within the user's field of view.

3. The surgical system of claim 2, wherein the virtual element has a pose that maintains a constant offset from the real-world pose of the real-world element regardless of the real-world pose of the augmented reality headset.

4. The surgical system of claim 3, wherein the extended reality headset controller is further configured to determine user input based on tracking data of the real-world gesture of the real-world element, Wherein the extended reality headset controller is configured to generate the world-registered extended reality image further based on the user input.

5. The surgical system of claim 3, wherein a portion of the real world coordinate system is interpreted by the augmented reality headset controller to define an interaction zone, wherein the extended reality headset controller is further configured to determine whether the real-world gesture of the real-world element is within the interaction zone, and Wherein the extended reality headset controller is configured to respond to determining that the real-world pose of the real-world element is within the interaction zone by generating the virtual element within the world-registered extended reality image.

6. The surgical system of claim 1 , wherein a portion of the real world coordinate system is interpreted by the augmented reality headset controller to define an interaction zone, wherein the interaction zone is a navigation zone, wherein the tracking system is further configured to determine the real-world pose of the real-world element only when the real-world pose of the real-world element is within the navigation zone, Wherein the extended reality headset controller is configured to generate the world-registered extended reality image to contain the user interface data associated with the virtual element corresponding to the real-world element in response to determining that the real-world element is within the navigation zone.

7. The surgical system of claim 1 , wherein a portion of the real world coordinate system is interpreted by the augmented reality headset controller to define an interaction zone, wherein the interaction zone is a transition zone, wherein the augmented reality headset controller is configured to generate the world-registered augmented reality image to further exclude a portion of the user interface data in response to determining that the real-world element is within the transition zone, and Wherein the extended reality headset controller is configured to generate the world-registered extended reality image to further include the portion of the user interface data in response to determining that the real-world element is outside of the transition zone.

8. The surgical system of claim 1 , wherein a portion of the real world coordinate system is interpreted by the augmented reality headset controller to define an interaction zone, wherein the interaction zone is a warning zone, The navigation controller is further configured to: determining that the warning zone includes a projected path of motion of the robot arm in the real-world coordinate system; determining whether the real-world element is within the warning zone; and responsive to determining that the real-world element is within the warning zone, preventing the robotic arm from moving along the projected path, wherein the extended reality headset controller is configured to generate the world-registered extended reality image such that a virtual path associated with the projected path of motion of the robotic arm is displayed, and Wherein the augmented reality headset is further configured to output a warning to the user in response to the real-world element being within the warning zone.

Images