Surgical systems

Abstract

The present invention relates to a surgical system. A virtual model of a planned instrument attachment can be provided to ensure the correct selection of a physical instrument attachment. An XR head-mounted device controller can generate the shape and posture of the virtual model of the planned instrument attachment based on predetermined information associated with the planned instrument attachment and based on the posture of the instrument relative to the XR head-mounted device. The XR head-mounted device can display the virtual model on a perspective display screen of the XR head-mounted device, and the perspective display screen is configured to allow at least a portion of a real-world scene to pass through the perspective display screen.

Description

Technical Field

[0001] The present disclosure relates to medical devices and systems, and more particularly, to computer-assisted navigation in surgical procedures performed using robotic surgical systems. Background Art

[0002] Computer-assisted navigation in surgery provides the surgeon with enhanced visualization of surgical instruments relative to radiographic images of the patient's anatomy. Navigated surgery typically involves tracking the position and orientation of surgical instruments using a single near-infrared ("NIR") stereo camera setup via a disk or sphere array. In this scenario, there are three parameters that compete for optimization: (1) accuracy, (2) robustness, and (3) ergonomics.

[0003] Navigated surgical procedures using existing navigation systems are prone to events that trigger intermittent pauses while people and / or objects obstruct the ability of the tracking assembly to track the pose of the patient, robot, and surgical instruments.There is a need to improve the tracking performance of navigation systems. Summary of the invention

[0004] Various embodiments disclosed herein relate to improvements in computer-assisted navigation during surgical procedures. One or more extended reality ("XR") headsets may be equipped with tracking cameras that provide tracking information to a camera tracking system to combine with tracking information from other tracking cameras (which may be part of another XR headset), auxiliary tracking poles, or other devices. Through the various gesture linking operations disclosed herein, the camera tracking system may be able to track tools and other objects with greater robustness and through a wide range of motions.

[0005] In some embodiments, a surgical system is provided. The surgical system includes an extended reality ("XR") head-mounted device and an XR head-mounted device controller. The XR head-mounted device is configured to be worn by a user during a surgical procedure and includes a perspective display screen, which is configured to display a virtual model of a planned instrument attachment and allow at least a portion of a real-world scene to pass through the perspective display screen for viewing by the user. The real-world scene includes an instrument. The XR head-mounted device controller is configured to generate a shape and pose of the virtual model of the planned instrument attachment based on predetermined information associated with the planned instrument attachment and based on a pose of the instrument relative to the XR head-mounted device.

[0006] In other embodiments, a surgical system is provided. The surgical system includes an extended reality ("XR") head-mounted device and an XR head-mounted device controller. The XR head-mounted device is configured to be worn by a user during a surgical procedure and includes a perspective display screen, which is configured to display a virtual model of a planned instrument attachment and allow at least a portion of a real-world scene to pass through the perspective display screen for viewing by the user. The real-world scene includes a physical instrument attachment. The XR head-mounted device controller is configured to generate a shape and a pose of the virtual model of the planned instrument attachment based on predetermined information of the planned instrument attachment and based on a pose of the physical instrument attachment relative to the XR head-mounted device.

[0007] A related method performed by a surgical system and a related robotic surgical system is disclosed.

[0008] Other surgical systems, methods, and computer program products according to the embodiments will be apparent to those skilled in the art upon review of the following drawings and detailed description. All such surgical systems, methods, and computer program products are intended to be included in this specification, within the scope of the present disclosure, and protected by the appended claims. In addition, all embodiments disclosed herein are intended to be implemented alone or in combination in any manner and / or combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings illustrate certain non-limiting embodiments of the inventive concept, and are included to provide a further understanding of the present disclosure, and are incorporated in and constitute a part of this application. In the drawings:

[0010] Figure 1 An embodiment of a surgical system according to some embodiments of the present disclosure is presented;

[0011] Figure 2 Some embodiments of the present disclosure are shown Figure 1 A surgical robotic component of a surgical system;

[0012] Figure 3A Some embodiments of the present disclosure are shown Figure 1 A camera tracking system component of a surgical system;

[0013] Figure 3B and 3C It shows that some embodiments of the present disclosure can be used with Figure 1 a front view and an isometric view of another camera tracking system component for use with a surgical system;

[0014] Figure 4An embodiment of an end effector attachable to a robotic arm and configured in accordance with some embodiments of the present disclosure is presented;

[0015] Figure 5 Medical operations were demonstrated in which surgical robots and camera systems were positioned around the patient;

[0016] Figure 6 Demonstrated for medical operations Figure 5 A block diagram view of components of a surgical system;

[0017] Figure 7 Demonstrated that when using the navigation capabilities of the surgical system Figure 5 and 6 Various display screens displayed on the monitor;

[0018] Figure 8 A block diagram showing some electrical components of a surgical robot according to some embodiments of the present disclosure;

[0019] Fig. 9 A block diagram showing components of a surgical system including an imaging device connected to a computer platform that may be operatively connected to a camera tracking system and / or a surgical robot according to some embodiments of the present disclosure is presented;

[0020] Fig.10 An embodiment of a C-arm imaging device that can be used in combination with a surgical robot according to some embodiments of the present disclosure is presented;

[0021] Fig.11 An embodiment of an O-arm imaging device that may be used in combination with a surgical robot according to some embodiments of the present disclosure is presented;

[0022] Fig.12 shows a block diagram view of components of a surgical system including a pair of XR headsets and auxiliary tracking poles operating according to some embodiments of the present disclosure;

[0023] Fig.13 An XR head mounted device configured according to some embodiments of the present disclosure is shown;

[0024] Fig.14 An electrical assembly of an XR head mounted device that may be operably connected to a computer platform, one or more imaging devices, and / or a surgical robot according to some embodiments of the present disclosure is shown;

[0025] Fig.15 A block diagram illustrating an arrangement of optical components of an XR head mounted device according to some embodiments of the present disclosure is shown;

[0026] Fig.16 shows an example view of a display screen of an XR head mounted device for providing navigation assistance to manipulate a surgical tool during a medical procedure in accordance with some embodiments of the present disclosure;

[0027] Fig.17 An example configuration of an auxiliary tracking pole having two pairs of stereo cameras configured in accordance with some embodiments of the present disclosure is shown;

[0028] Fig.18A -C shows an example of a view through a display screen of an XR head mounted device according to some embodiments of the present disclosure, wherein a virtual model or information associated with an instrument accessory is displayed;

[0029] Fig.19A -C shows an example of a view through a display screen of an XR head mounted device according to some embodiments of the present disclosure, wherein a virtual model of an instrument accessory in a posture based on the instrument is displayed;

[0030] Fig. 20A -C shows an example of a view through a display screen of an XR head mounted device according to some embodiments of the present disclosure, wherein a virtual model of an instrument accessory in an instrument-based pose matches a real-world instrument accessory;

[0031] Fig.21A -C shows an example of a view through a display screen of an XR head mounted device according to some embodiments of the present disclosure, wherein a virtual model of an instrument accessory in an instrument-based pose does not match a real-world instrument accessory;

[0032] Figure 22-23 A flow chart is presented of an example of a process performed by a surgical system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0033] The inventive concept will now be described more fully below with reference to the accompanying drawings, in which examples of embodiments of the inventive concept are shown. The inventive concept can be implemented in many different forms and should not be construed as being limited to the embodiments listed herein. Rather, these embodiments are provided to make the disclosure more complete and complete, and to fully convey the scope of each inventive concept to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. A component in one embodiment may be defaulted to being present in or used in another embodiment.

[0034] Various embodiments disclosed herein relate to improvements in computer-assisted navigation during surgical procedures. An extended reality (XR) head-mounted device is operatively connected to a surgical system and is configured to provide an interactive environment through which surgeons, assistants, and / or other personnel can view patient images and make selections therein, view computer-generated surgical navigation information and make selections therein, and / or control surgical equipment in an operating room. As will be explained below, the XR head-mounted device can be configured to enhance a real-world scene with a computer-generated XR image. The XR head-mounted device can be configured to provide an augmented reality (AR) viewing environment by displaying a computer-generated XR image on a perspective display screen that allows light from a real-world scene to pass through for combined viewing by a user. Alternatively, the XR head-mounted device can be configured to provide a virtual reality ("VR") viewing environment by preventing or substantially preventing light from a real-world scene from being directly viewed by a user when the user is viewing a computer-generated AR image on a display screen. The XR head-mounted device can be configured to provide both an AR viewing environment and a VR viewing environment. In one embodiment, both the AR viewing environment and the VR viewing environment are provided by lateral bands of significantly different opacities arranged between the see-through display screen and the real-world scene, so that the VR viewing environment is provided for the XR images aligned with the high opacity bands and the AR viewing environment is provided for the XR images aligned with the low opacity bands. In another embodiment, both the AR viewing environment and the VR viewing environment are provided by computer-adjustable control of an opacity filter that variably limits the amount of light from the real-world scene that passes through the see-through display screen to be combined with the XR image viewed by the user. Therefore, the XR head-mounted device may also be referred to as an AR head-mounted device or a VR head-mounted device.

[0035] Figure 1 An embodiment of a surgical system 2 according to some embodiments of the present disclosure is shown. Before performing an orthopedic or other surgical procedure, one may use, for example, Fig.10 C-arm imaging device 104 or Fig.11 A three-dimensional ("3D") image scan of the planned surgical area of ​​the patient is taken from an O-arm imaging device 106, or from another medical imaging device such as a computed tomography ("CT") image or an MRI. Such a scan may be performed preoperatively (e.g., most commonly a few weeks prior to the procedure) or intraoperatively. However, any known 3D or 2D image scan may be used in accordance with various embodiments of surgical system 2. The image scan is sent to a computer platform in communication with surgical system 2, such as computer platform 910 ( Fig. 9 ), the surgical system may include a camera tracking system component 6, a surgical robot 4 (e.g., Figure 12), imaging devices (e.g., C-arm 104, O-arm 106, etc.), and an image database 950 for storing image scans of patients. View computer platform 910 ( Fig. 9 ) generates a surgical plan defining a target pose for surgical tools to be used during a surgical procedure performed on the patient's anatomy. Example surgical tools (also referred to as tools) may include, but are not limited to, drills, screw drivers, retractors, and implants such as screws, spacers, interbody fusion devices, plates, rods, etc. In some embodiments, the surgical plan defining the target plane is planned on the 3D image scan displayed on the display device.

[0036] As used herein, the term "pose" refers to the position and / or rotation angle of an object (e.g., a dynamic reference array, an end effector, a surgical tool, an anatomical structure, etc.) relative to another object and / or a defined coordinate system. Therefore, the pose can be defined based only on the multi-dimensional position of an object relative to another object and / or a defined coordinate system, based only on the multi-dimensional rotation angle of the object relative to another object and / or a defined coordinate system, or a combination of multi-dimensional position and multi-dimensional rotation angle. Therefore, the term "pose" is used to refer to the position, the rotation angle, or a combination thereof.

[0037] Figure 1 The surgical system 2 can assist the surgeon during a medical procedure by, for example, providing for use of holding tools, alignment tools, application tools, guide tools, and / or positioning tools. In some embodiments, the surgical system 2 includes a surgical robot 4 and a camera tracking system assembly 6. The ability to mechanically couple the surgical robot 4 and the camera tracking system assembly 6 can allow the surgical system 2 to be manipulated and moved as a single unit, and allow the surgical system 2 to have a small footprint in an area, allowing for easier movement through narrow passages and around turns, and allowing for storage within a smaller area.

[0038] The surgical procedure can begin with the surgical system 2 being moved from the medical storage room to the medical procedure room. The surgical system 2 can be manipulated through doorways, halls, and elevators to reach the medical procedure room. Within the medical procedure room, the surgical system 2 can be physically divided into two separate and distinct systems (the surgical robot 4 and the camera tracking system assembly 6). The surgical robot 4 can be positioned adjacent to the patient in any suitable location to appropriately assist the medical staff. The camera tracking system assembly 6 can be positioned at the bottom of the patient, at the patient's shoulder, or at any other location suitable for tracking the current posture and posture movement of the surgical robot 4 and the patient's track portion. The surgical robot 4 and the camera tracking system assembly 6 can be powered by an onboard power supply and / or plugged into an external wall socket.

[0039] The surgical robot 4 may be used to assist the surgeon by holding and / or using tools during a medical procedure. In order to properly utilize and hold the tools, the surgical robot 4 may rely on a plurality of motors, computers, and / or actuators to function properly. Figure 1 As shown, the robot body 8 can serve as a structure in which multiple motors, computers, and / or actuators can be secured within the surgical robot 4. The robot body 8 can also provide support for the robotic telescopic support arm 16. The size of the robot body 8 can provide a stable platform to support attached components and can house, conceal, and protect multiple motors, computers, and / or actuators that can operate the attached components.

[0040] The robot base 10 can act as a lower support for the surgical robot 4. In some embodiments, the robot base 10 can support the robot body 8 and can attach the robot body 8 to a plurality of powered wheels 12. This attachment to the wheels can allow the robot body 8 to move efficiently in space. The robot base 10 can run along the length and width of the robot body 8. The robot base 10 can be about two inches to about 10 inches high. The robot base 10 can cover, protect and support the powered wheels 12.

[0041] In some embodiments, Figure 1 As shown, at least one powered wheel 12 can be attached to the robot base 10. The powered wheel 12 can be attached to the robot base 10 at any position. Each individual powered wheel 12 can rotate around a vertical axis in any direction. The motor can be placed above, inside or adjacent to the powered wheel 12. Such a motor can allow the surgical system 2 to be manipulated to any position and stabilize and / or level the surgical system 2. A rod located inside or adjacent to the powered wheel 12 can be pressed into a surface by a motor. The rod not shown can be made of any suitable metal to lift the surgical system 2. The rod can lift the powered wheel 10 (which can lift the surgical system 2) to any height required to level or otherwise fix the surgical system 2 relative to the patient's orientation. The weight of the surgical system 2 is supported by a small contact area of ​​the rod on each wheel, preventing the surgical system 2 from moving during the medical procedure. This rigid positioning can prevent objects and / or people from accidentally moving the surgical system 2.

[0042] The robot track 14 can be used to facilitate moving the surgical system 2. The robot track 14 provides a person with the ability to move the surgical system 2 without grasping the robot body 8. Figure 1As shown, the length of the robot track 14 can be the same length as the robot body 8, shorter than the robot body 8, and / or longer than the robot body 8. The robot track 14 can further provide protection to the robot body 8, thereby preventing the subject and / or medical personnel from contacting, hitting, or bumping into the robot body 8.

[0043] The robot body 8 may provide support for a selectively compliant articulated robotic arm, hereinafter referred to as a "SCARA". Due to the repeatability and compactness of the robotic arm, the use of the SCARA 24 within the surgical system 2 may be advantageous. The compactness of the SCARA may provide additional space within a medical procedure, which may allow a medical professional to perform a medical procedure without excessive clutter and restricted areas. The SCARA 24 may include a robotic telescopic support 16, a robotic support arm 18, and / or a robotic arm 20. The robotic telescopic support 16 may be disposed along the robot body 8. As shown in FIG. Figure 1 As shown, the robotic telescopic support 16 can provide support for the SCARA 24 and the display 34. In some embodiments, the robotic telescopic support 16 can extend and retract in a vertical direction. The body of the robotic telescopic support 16 can be any width and / or height configured to support the stress and weight placed thereon.

[0044] In some embodiments, the medical staff can move the SCARA 24 via commands submitted by the medical staff. As will be explained in further detail below, the commands can originate from a display 34, a tablet computer, and / or an XR headset (e.g., Fig. 9 The XR headset may eliminate the need for medical personnel to refer to any other display, such as the display 34 or a tablet computer, which enables the SCARA 24 to be configured without the display 34 and / or the tablet computer. As will be explained in further detail below, the command may be generated by pressing a switch and / or pressing multiple switches, and / or may be generated based on gesture commands and / or voice commands sensed by the XR headset.

[0045] like Figure 5As shown in , the activation assembly 60 may include a switch and / or multiple switches. The activation assembly 60 may be operable to transmit movement commands to the SCARA 24, thereby allowing the operator to manually manipulate the SCARA 24. When the switch or multiple switches are pressed, the medical staff may have the ability to move the SCARA 24 through the applied hand movement. Alternatively or additionally, as will be explained in further detail below, the operator may control the movement of the SCARA 24 through gesture commands and / or voice commands sensed by the XR head-mounted device. Additionally, when the SCARA 24 does not receive a command to move, the SCARA 24 may be locked in place to prevent accidental movement by medical staff and / or other objects. By locking in place, the SCARA 24 provides a stable platform through which the end effector 26 can guide surgical tools during a medical procedure.

[0046] The robotic support arm 18 may be connected to the robotic telescopic support 16 by various mechanisms. Figure 1 and 2 , the robotic support arm 18 can rotate in any direction relative to the robotic telescopic support 16. The robotic support arm 18 can rotate three hundred and sixty degrees around the robotic telescopic support 16. The robotic arm 20 can be connected to the robotic support arm 18 in any suitable position and by various mechanisms that enable rotation in any direction relative to the robotic support arm 18. In one embodiment, the robotic arm 20 can rotate three hundred and sixty degrees relative to the robotic support arm 18. This free rotation allows the operator to position the robotic arm 20 according to the surgical plan.

[0047] Figure 4 and 5 The end effector 26 in the example is attached to the robotic arm 20 at any suitable position. The end effector 26 is configured to be attached to the end effector coupling 22 of the robotic arm 20 positioned by the surgical robot 4. The example end effector 26 includes a tubular guide that guides the movement of an inserted surgical tool relative to the anatomical structure on which the surgical procedure is to be performed.

[0048] In some embodiments, a dynamic reference array 52 is attached to the end effector 26. The dynamic reference array (also referred to herein as a "DRA") is a rigid body that can be placed on a patient's anatomy (e.g., bone), one or more XR headsets worn by personnel in an operating room, an end effector, a surgical robot, or a surgical tool in a navigated surgical procedure. The computer platform 910 is configured in combination with the camera tracking system assembly 6 or other 3D positioning system to track the pose (e.g., position and rotational orientation) of the DRA in real time. The DRA may include reference points, such as the illustrated arrangement of a sphere. Such tracking of the 3D coordinates of the DRA may allow the surgical system 2 to determine the position of the DRA relative to the patient's anatomy (e.g., bone). Figure 5 The pose of the target anatomical structure of the patient 50 in any multi-dimensional space.

[0049] like Figure 1 As shown, the light indicator 28 can be positioned on top of the SCARA 24. The light indicator 28 can be any type of light that illuminates to indicate the "condition" in which the surgical system 2 is currently operating. In some embodiments, the light can be generated by LED lights that form a ring around the light indicator 28. The light indicator 28 can include a completely permeable material that allows light to pass through the entire light indicator 28. The light indicator 28 can be attached to the lower display support 30. As shown in FIG. Figure 2 As shown, the lower display support 30 can allow an operator to manipulate the display 34 to any suitable position. The lower display support 30 can be attached to the light indicator 28 by any suitable mechanism. In some embodiments, the lower display support 30 can rotate about the light indicator 28 or be rigidly attached to the light indicator. The upper display support 32 can be attached to the lower display support 30 by any suitable mechanism.

[0050] In some embodiments, a tablet computer can be used in conjunction with and / or without the display 34. The tablet computer can be placed on the upper display support 32 in place of the display 34 and can be removable from the upper display support 32 during a medical procedure. In addition, the tablet computer can communicate with the display 34. The tablet computer can be connected to the surgical robot 4 via any suitable wireless and / or wired connection. In some embodiments, the tablet computer can program and / or control the surgical system 2 during a medical procedure. When the surgical system 2 is controlled with the tablet computer, all input and output commands can be replicated on the display 34. Using a tablet computer can allow the operator to manipulate the surgical robot 4 without having to move around the patient 50 and / or the surgical robot 4.

[0051] As will be explained below, in some embodiments, the surgeon and / or other personnel may wear an XR headset that can be used in conjunction with the display 34 and / or a tablet computer, or one or more XR headsets may eliminate the need to use the display 34 and / or a tablet computer.

[0052] like Figure 3A and 5 As shown, the camera tracking system assembly 6 works in conjunction with the surgical robot 4 via a wired or wireless communication network. Figure 1 , 3 and 5, the camera tracking system assembly 6 can include some components similar to the surgical robot 4. For example, the camera body 36 can provide the functions found in the robot body 8. The robot body 8 can provide an auxiliary tracking rod on which the camera 46 is mounted. The structure within the robot body 8 can also provide support for electronic equipment, communication devices and power supplies for operating the camera tracking system assembly 6. The camera body 36 can be made of the same material as the robot body 8. The camera tracking system assembly 6 can communicate directly with the XR head-mounted device, tablet computer and / or display 34 via a wireless and / or wired network to enable the XR head-mounted device, tablet computer and / or display 34 to control the functions of the camera tracking system assembly 6.

[0053] The camera body 36 is supported by a camera base 38. The camera base 38 may be used as the robot base 10. Figure 1 In the embodiment of the present invention, the camera base 38 can be wider than the robot base 10. The width of the camera base 38 can allow the camera tracking system assembly 6 to be connected to the surgical robot 4. Figure 1 As shown, the width of the camera mount 38 can be large enough to fit outside of the robot base 10. The additional width of the camera mount 38 can allow the surgical system 2 to provide additional maneuverability and support for the surgical system 2 when the camera tracking system assembly 6 is connected to the surgical robot 4.

[0054] As with the robot base 10, a plurality of powered wheels 12 may be attached to the camera base 38. Similar to the operation of the robot base 10 and powered wheels 12, the powered wheels 12 may allow the camera tracking system assembly 6 to be stabilized and leveled or set to a fixed orientation relative to the patient 50. Such stabilization may prevent the camera tracking system assembly 6 from moving during a medical procedure and may prevent the camera 46 on the auxiliary tracking pole from losing track of a DRA connected to the XR headset and / or the surgical robot 4 and / or losing track of a DRA connected to the XR headset and / or the surgical robot 4. Figure 3A and 5The tracking of one or more DRAs 52 of the anatomical structure 54 and / or tool 58 within the designated area 56 is shown. This stability and maintenance of tracking enhances the ability of the surgical robot 4 to operate effectively with the camera tracking system assembly 6. Additionally, the wide camera base 38 can provide additional support for the camera tracking system assembly 6. Specifically, as Figure 3A and 5 As shown, the wide camera base 38 can prevent the camera tracking system assembly 6 from tipping over when the camera 46 is positioned over the patient.

[0055] The camera telescoping support 40 can support a camera 46 on an auxiliary tracking pole. In some embodiments, the telescoping support 40 can move the camera 46 higher or lower in a vertical direction. The camera handle 48 can be attached to the camera telescoping support 40 at any suitable location and is configured to allow an operator to move the camera tracking system assembly 6 to a planned position prior to a medical procedure. In some embodiments, the camera handle 48 can be used to lower and raise the camera telescoping support 40. The camera handle 48 can perform raising and lowering of the camera telescoping support 40 by pressing a button, a switch, a lever, and / or any combination thereof.

[0056] The lower camera support arm 42 may be attached to the camera telescoping support 40 at any suitable location, in embodiments such as Figure 1 As shown, the lower camera support arm 42 can rotate three hundred and sixty degrees around the telescopic support 40. This free rotation can allow the operator to position the camera 46 in any suitable position. The lower camera support arm 42 can be connected to the telescopic support 40 by any suitable mechanism. The lower camera support arm 42 can be used to provide support for the camera 46. The camera 46 can be attached to the lower camera support arm 42 by any suitable mechanism. The camera 46 can pivot in any direction at the attachment area between the camera 46 and the lower camera support arm 42. In an embodiment, the curved track 44 can be disposed on the lower camera support arm 42.

[0057] The curved track 44 may be placed at any suitable location on the lower camera support arm 42. Figure 3A As shown, the curved track 44 can be attached to the lower camera support arm 42 by any suitable mechanism. The curved track 44 can be of any suitable shape, which can be a crescent, a circle, a flat, an ellipse, and / or any combination thereof. The camera 46 can be movably positioned along the curved track 44. The camera 46 can be attached to the curved track 44 by, for example, rollers, brackets, braces, motors, and / or any combination thereof. Motors and rollers not shown can be used to move the camera 46 along the curved track 44. As shown in FIG. Figure 3AAs shown, during a medical procedure, if an object blocks the camera 46 from observing one or more tracked DRAs, the motor can responsively move the camera 46 along the curved track 44. This motorized movement can allow the camera 46 to move to a new position that is no longer blocked by the object without moving the camera tracking system component 6. When the camera 46 is blocked from observing one or more DRAs, the camera tracking system component 6 can send a stop signal to the surgical robot 4, the XR head-mounted device, the display 34 and / or the tablet computer. The stop signal can prevent the SCARA 24 from moving until the camera 46 has reacquired the tracked DRA 52 and / or can warn the operator to wear the XR head-mounted device and / or watch the display 34 and / or the tablet computer. This SCARA 24 can be configured to respond to the receipt of the stop signal by stopping further movement of the base and / or the end effector coupling 22 until the camera tracking system can resume tracking of the DRA.

[0058] Figure 3B and 3C Demonstrated that it can be Figure 1 1 and 1 ' are front and isometric views of another camera tracking system assembly 6' that can be used with a surgical system of a surgical robot or can be used independently of a surgical robot. For example, the camera tracking system assembly 6' can be used to provide navigated surgery without the use of robotic guidance. Figure 3B and 3C The camera tracking system component 6′ and Figure 3A One of the differences between the camera tracking system components 6 is Figure 3B and 3C The camera tracking system assembly 6′ includes a housing for transporting a computer platform 910. The computer platform 910 can be configured to: perform camera tracking operations to track the DRA; perform guided surgical operations that provide surgical navigation information to a display device (e.g., an XR headset and / or other display device); and perform other computing operations disclosed herein. Thus, the computer platform 910 can include a navigation computer, such as Fig.14 One or more navigation computers in the navigation computer.

[0059] Figure 6 Demonstrated for medical operations Figure 5 A block diagram view of the components of a surgical system. Figure 6 , the navigation camera 46 on the auxiliary tracking pole has a navigation field of view 600 in which a reference array 602 attached to the patient, a reference array 604 attached to the surgical instrument, and the pose (e.g., position and orientation) of the robotic arm 20 are tracked. The navigation camera 46 may be a computer platform 910 that includes a computer platform configured to perform the operations described below. Figure 3B and3C The reference array 602 is part of the camera tracking system assembly 6′ of the surgical robot 4. The reference array achieves tracking by reflecting light in a known pattern, which is decoded by the tracking subsystem of the surgical robot 4 to determine the corresponding pose of the reference array. If the line of sight between the patient reference array 602 and the navigation camera 46 in the auxiliary tracking pole is blocked (e.g., by medical personnel, instruments, etc.), further navigation of the surgical instruments may not be possible, and a responsive notification can temporarily stop further movement of the robotic arm 20 and the surgical robot 4, display a warning on the display 34, and / or provide an audible warning to the medical staff. The display 34 is available to the surgeon 610 and assistant 612, but viewing requires turning the head away from the patient and changing the eye focus to different distances and positions. The navigation software can be controlled by a technician 614 based on voice commands from the surgeon.

[0060] Figure 7 It is shown that when using the navigation function of the surgical system 2, the surgical robot 4 can be used to perform a surgical operation. Figure 5 and 6 The display screens may include, but are not limited to, a patient radiograph with an overlaid graphical representation of an instrument model positioned in the display screen relative to the anatomy based on the developed surgical plan and / or based on the pose of the tracked reference array, various user-selectable menus for controlling different stages of the surgical procedure, and virtually projected dimensional parameters of the implant (e.g., length, width, and / or diameter).

[0061] For navigated surgery, various processing components (e.g., computer platform 910) and associated software described below are provided that enable preoperative planning of a surgical procedure (e.g., implant placement) and electronic delivery of the plan to a computer platform 910 for providing navigation information to one or more users during the planned surgical procedure.

[0062] For robotic navigation, various processing components (e.g., computer platform 910) and associated software described below are provided that enable preoperative planning of a surgical procedure (e.g., implant placement) and electronic transfer of the plan to the surgical robot 4. The surgical robot 4 uses the plan to guide the robotic arm 20 and connected end effector 26 to provide target poses of the surgical tools relative to the patient's anatomy for the steps of the planned surgical procedure.

[0063] The following embodiments relate to the use of one or more XR head-mounted devices that can be worn by a surgeon 610, assistant 612 and / or other medical personnel to provide an improved user interface for receiving information from and / or providing control commands to a surgical robot, camera tracking system components 6 / 6′ and / or other medical equipment in an operating room.

[0064] Figure 8 FIG. 1 is a block diagram showing some electrical components of a surgical robot 4 according to some embodiments of the present disclosure. Figure 8 , a load cell (not shown) can be configured to track the force applied to the end effector coupling 22. In some embodiments, the load cell can communicate with multiple motors 850, 851, 852, 853 and / or 854. When the load cell senses a force, information about the amount of force applied can be distributed from the switch array and / or multiple switch arrays to the controller 846. The controller 846 can obtain the force information from the load cell and process it with a switching algorithm. The controller 846 uses the switching algorithm to control the motor driver 842. The motor driver 842 controls the operation of one or more motors 850, 851, 852, 853 and / or 854. The motor driver 842 can direct a specific motor to generate an equal amount of force, such as measured by the load cell through the motor. In some embodiments, as indicated by the controller 846, the force generated can come from multiple motors, such as 850-854. Additionally, the motor driver 842 can receive input from the controller 846. The controller 846 can receive information about the direction of the force sensed by the force sensor from the force sensor. The controller 846 can use a motion controller algorithm to process this information. The algorithm can be used to provide information to a specific motor driver 842. In order to replicate the direction of the force, the controller 846 can activate and / or deactivate certain motor drivers 842. The controller 846 can control one or more motors, such as one or more of 850-854, to induce the movement of the end effector 26 in the direction of the force sensed by the force sensor. This force-controlled movement can allow the operator to move the SCARA 24 and the end effector 26 effortlessly and / or with very little resistance. The movement of the end effector 26 can be performed to position the end effector 26 in any suitable posture (i.e., relative to the position and angle orientation of the defined three-dimensional (3D) orthogonal reference axis) for use by medical personnel.

[0065] Figure 5The activation assembly 60 best shown can form a bracelet that wraps around the end effector coupling 22. The activation assembly 60 can be located on any portion of the SCARA 24, any portion of the end effector coupling 22, can be worn by medical personnel (and communicate wirelessly), and / or any combination thereof. The activation assembly 60 can include a primary button and a secondary button.

[0066] Pressing the master button may allow the operator to move the SCARA 24 and the end effector coupling 22. According to one embodiment, once set in position, the SCARA 24 and the end effector coupling 22 may not move until the operator programs the surgical robot 4 to move the SCARA 24 and the end effector coupling 22, or uses the master button to move. In some instances, it may be necessary to press at least two non-adjacent master activation switches before the SCARA 24 and the end effector coupling 22 will respond to operator commands. Pressing at least two master activation switches may prevent accidental movement of the SCARA 24 and the end effector coupling 22 during a medical procedure.

[0067] Activated by the main button, the load cell can measure the magnitude and / or direction of the force applied by the operator (i.e., medical personnel) on the end effector coupling 22. This information can be transmitted to one or more motors within the SCARA 24, such as one or more of 850-854, which can be used to move the SCARA 24 and the end effector coupling 22. Information about the magnitude and direction of the force measured by the load cell can cause one or more motors (e.g., one or more of 850-854) to move the SCARA 24 and the end effector coupling 22 in the same direction as sensed by the load cell. This force-controlled movement can allow the operator to easily move the SCARA 24 and the end effector coupling 22, and because the motors move the SCARA 24 and the end effector coupling 22 while the operator moves the SCARA 24 and the end effector coupling 22, a lot of effort is not required.

[0068] In some instances, the operator can use the secondary button as a "select" device. During a medical operation, the surgical robot 4 can notify medical personnel of certain situations through one or more XR head-mounted devices 920, displays 34, and / or light indicators 28. One or more XR head-mounted devices 920 are each configured to display an image on a perspective display screen to form an extended reality image overlaid on a real-world object that can be viewed through the perspective display screen. The surgical robot 4 can prompt medical personnel to select functions, modes, and / or evaluate the situation of the surgical system 2. Pressing the secondary button once can activate certain functions, modes, and / or confirm information transmitted to medical personnel through one or more XR head-mounted devices 920, displays 34, and / or light indicators 28. In addition, pressing the secondary button multiple times in rapid succession can activate additional functions, modes, and / or select information transmitted to medical personnel through one or more XR head-mounted devices 920, displays 34, and / or light indicators 28.

[0069] Further references Figure 8 , the electrical components of the surgical robot 4 include a platform subsystem 802, a computer subsystem 820, a motion control subsystem 840, and a tracking subsystem 830. The platform subsystem 802 includes a battery 806, a power distribution module 804, a connector board 808, and a charging station 810. The computer subsystem 820 includes a computer 822, a display 824, and a speaker 826. The motion control subsystem 840 includes a drive circuit 842, motors 850, 851, 852, 853, 854, stabilizers 855, 856, 857, 858, an end effector connector 844, and a controller 846. The tracking subsystem 830 includes a positioning sensor 832 and a camera converter 834. The surgical robot 4 may also include a removable foot pedal 880 and a removable tablet computer 890.

[0070] The input power is supplied to the surgical robot 4 through a power supply that can be provided to the power distribution module 804. The power distribution module 804 receives the input power and is configured to generate different power supply voltages provided to other modules, components and subsystems of the surgical robot 4. The power distribution module 804 can be configured to provide different voltage supplies to the connector board 808, which can be provided to other components (such as a computer 822, a display 824, a speaker 826, a drive 842) to, for example, power the motors 850-854 and the end effector connector 844, and to the camera converter 834 and other components for the surgical robot 4. The power distribution module 804 can also be connected to a battery 806, which acts as a temporary power source when the power distribution module 804 does not receive power from the input power source. At other times, the power distribution module 804 can be used to charge the battery 806.

[0071] The connector board 808 can be used to connect different devices and components to the surgical robot 4 and / or associated components and modules. The connector board 808 may contain one or more ports for receiving lines or connectors from different components. For example, the connector board 808 may have a grounding port for grounding the surgical robot 4 to other equipment, a port for connecting a foot pedal 880, and a port for connecting to a tracking subsystem 830, which may include a positioning sensor 832, a camera converter 834, and a DRA tracking camera 870. The connector board 808 may also include other ports to allow USB, Ethernet, HDMI communication with other components (such as a computer 822). According to some embodiments, the connector board 808 may include a wired and / or wireless interface for operably connecting one or more XR head-mounted devices 920 to the tracking subsystem 830 and / or the computer subsystem 820.

[0072] The control panel 816 may provide various buttons or indicators for controlling the operation of the surgical robot 4 and / or providing information from the surgical robot 4 for the operator to observe. For example, the control panel 816 may include buttons for turning the surgical robot 4 on or off, raising or lowering the vertical column 16, and raising or lowering the stabilizers 855-858, which may be designed to engage the casters 12 to lock the surgical robot 4 without physically moving. Other buttons may stop the surgical robot 4 in the event of an emergency, which may remove all motor power and apply mechanical brakes to stop all movement from occurring. The control panel 816 may also have indicators (such as a line power indicator) that notify the operator of certain system conditions or the charge status of the battery 806. According to some embodiments, one or more XR head-mounted devices 920 may communicate, for example, via the connector plate 808 to control the operation of the surgical robot 4 and / or receive and display information generated by the surgical robot 4 for observation by a person wearing the XR head-mounted device 920.

[0073] The computer 822 of the computer subsystem 820 contains an operating system and software for operating the designated functions of the surgical robot 4. The computer 822 can receive and process information from other components (e.g., the tracking subsystem 830, the platform subsystem 802, and / or the motion control subsystem 840) to display information to the operator. Further, the computer subsystem 820 can provide output to the operator through the speaker 826. The speaker can be part of the surgical robot, part of the XR headset 920, or in another component of the surgical system 2. The display 824 can correspond to Figure 1 and 2 A display 34 is shown.

[0074] Tracking subsystem 830 may include positioning sensors 832 and camera transducers 834. Tracking subsystem 830 may correspond to camera tracking system component 6 of FIG. 3. DRA tracking camera 870 operates with positioning sensors 832 to determine the pose of DRA 52. Such tracking may be performed in a manner consistent with the present disclosure, including the use of infrared or visible light techniques, such as LEDs or reflective markers, that track the positions of active or passive elements of DRA 52, respectively.

[0075] The functional operations of the tracking subsystem 830 and the computer subsystem 820 may be included in the Figure 3A and 3B The computer platform 910 may be transported by the camera tracking system assembly 6′. The tracking subsystem 830 may be configured to determine a pose, such as the position and angular orientation of the tracked DRA. The computer platform 910 may also include a navigation controller configured to use the determined pose to provide navigation information to a user to guide them in moving the tracked tool relative to the registered patient image and / or the tracked anatomical structure during a planned surgical procedure. The computer platform 910 may be in a Figure 3B and 3C The computer platform 910 can be configured to communicate with the computer subsystem 820 and the computer subsystem 820 when used with a surgical robot. Figure 8 The computer platform 910 may communicate with other subsystems of the computer system to control movement of the end effector 26. For example, as will be explained below, the computer platform 910 may generate a graphical representation of a patient's anatomy, surgical tools, a user's hand, etc., having a displayed size, shape, color, and / or posture that is controlled based on the determined posture of one or more tracked DRAs, and the displayed graphical representation may be dynamically modified to track changes in the determined posture over time.

[0076] The motion control subsystem 840 can be configured to physically move the vertical column 16, the upper arm 18, the lower arm 20, or rotate the end effector coupling 22. The physical movement can be performed by using one or more motors 850-854. For example, the motor 850 can be configured to vertically raise or lower the vertical column 16. Figure 2 As shown, the motor 851 can be configured to move the upper arm 18 laterally about the point of engagement with the vertical column 16. Figure 2 As shown, motor 852 may be configured to move lower arm 20 laterally about a joint with upper arm 18. Motors 853 and 854 may be configured to move end effector coupling 22 to provide translational movement along and rotational movement about three-dimensional axes. Fig. 9The illustrated computer platform 910 can provide control inputs to the controller 846 that direct the movement of the end effector coupling 22 to position the passive end effector connected thereto in a planned pose (i.e., position and angular orientation relative to defined 3D orthogonal reference axes) relative to the anatomical structure to be operated on during the planned surgical procedure. The motion control subsystem 840 can be configured to measure the positioning of the end effector coupling 22 and / or the end effector 26 using an integrated positioning sensor (e.g., an encoder).

[0077] Fig. 9 A block diagram of components of a surgical system according to some embodiments of the present disclosure is shown, the surgical system including a device connected to a camera tracking system component 6 ( Figure 3A ) or 6′( Figure 3B , 3C ) and / or an imaging device (e.g., C-arm 104, O-arm 106, etc.) of a computer platform 910 of the surgical robot 4. Alternatively, at least some of the operations performed by the computer platform 910 disclosed herein may additionally or alternatively be performed by components of the surgical system.

[0078] refer to Fig. 9 , the computer platform 910 includes a display 912, at least one processor circuit 914 (also referred to as a processor for brevity), at least one memory circuit 916 (also referred to as a memory for brevity) containing computer readable program code 918, and at least one network interface 902 (also referred to as a network interface for brevity). The display 912 may be part of an XR head mounted device 920 according to some embodiments of the present disclosure. The network interface 902 may be configured to connect to Fig.10 The C-arm imaging device 104, Fig.11 The O-arm imaging device 106, another medical imaging device, an image database 950 containing medical images of patients, components of the surgical robot 4 and / or other electronic equipment.

[0079] When used with the surgical robot 4, the display 912 may correspond to Figure 2 The display 34 and / or Figure 8 The tablet computer 890 and / or the XR head mounted device 920 operably connected to the surgical robot 4, the network interface 902 may correspond to Figure 8 The platform network interface 812, and the processor 914 may correspond to Figure 8Computer 822. The network interface 902 of the XR head-mounted device 920 can be configured to communicate according to one or more wireless communication protocols (e.g., WLAN, 3GPP4G and / or 5G (New Radio) cellular communication standards, etc.) through a wired network (e.g., thin-wire Ethernet) and / or through a wireless RF transceiver link.

[0080] The processor 914 may include one or more data processing circuits, such as general and / or special purpose processors, such as microprocessors and / or digital signal processors. The processor 914 is configured to execute computer readable program code 918 in the memory 916 to perform operations, which may include some or all of the operations described herein as being performed for surgical planning, navigated surgery, and / or robotic surgery.

[0081] The computer platform 910 can be configured to provide surgical planning functionality. The processor 914 can be operated to display images of anatomical structures (e.g., vertebrae) received from one of the imaging devices 104 and 106 and / or from the image database 950 via the network interface 920 on the display device 912 and / or the XR head-mounted device 920. The processor 914 receives an operator's definition of the location of the anatomical structure shown in one or more images where a surgical procedure (e.g., screw placement) is to be performed, such as by the operator touching the display 912 to select the location for the planned surgical procedure or using a mouse-based cursor to define the location for the planned surgical procedure. As will be explained in further detail below, when the image is displayed in the XR head-mounted device 920, the XR head-mounted device can be configured to sense gesture-based commands formed by the wearer and / or sense voice-based commands spoken by the wearer, which can be used to control selection between menu items and / or control how objects are displayed on the XR head-mounted device 920.

[0082] The computer platform 910 can be configured to be able to perform anatomical measurements that may be particularly useful for knee surgery, similar to the measurement of various angles to determine the center of the hip, the center of the angle, natural landmarks (e.g., the transepicondylar line, the white side line, the posterior condylar line, etc.). Some measurements can be automatic, while some other measurements can involve human input or assistance. The computer platform 910 can be configured to allow the operator to input the selection of the correct implant for the patient, including the selection of size and alignment. The computer platform 910 can be configured to perform automatic or semi-automatic (involving human input) segmentation (image processing) of CT images or other medical images. The patient's surgical plan can be stored in a cloud-based server that can correspond to the database 950 for retrieval by the surgical robot 4.

[0083] For example, during an orthopedic surgery, a surgeon may select a site to be cut (e.g., posterior femur, proximal tibia, etc.) using a computer screen (e.g., a touch screen) or extended reality ("XR") interaction (e.g., gesture-based commands and / or voice-based commands) through, for example, an XR headset 920. The computer platform 910 may generate navigation information that provides visual guidance to the surgeon to perform the surgical procedure. When used with the surgical robot 4, the computer platform 910 may provide guidance that allows the surgical robot 4 to automatically move the end effector 26 to a target pose so that the surgical tool is aligned with the target position to perform a surgical procedure on the anatomical structure.

[0084] In some embodiments, surgical system 900 can use two DRAs to track patient anatomical locations, such as a DRA connected to the patient's tibia and a DRA connected to the patient's femur. System 900 can use standard navigated instruments for registration and inspection (e.g., a pointer similar to the pointer used in the Globus Excelsius GPS system for spinal surgery).

[0085] A particularly challenging task in navigated surgery is how to plan the positioning of implants in the spine, knee, and other anatomical structures, where the surgeon strives to perform the task on a computer screen, which is a 2D representation of the 3D anatomical structure. The system 900 can solve this problem by using an XR head-mounted device 920 to display three-dimensional (3D) computer-generated representations of anatomical structures and candidate implants. Under the guidance of the computer platform 910, the computer-generated representations are scaled and posed relative to each other on the display screen and can be manipulated by the surgeon while viewing through the XR head-mounted device 920. The surgeon can, for example, use gesture-based commands and / or voice-based commands sensed by the XR head-mounted device 920 to manipulate the displayed computer-generated representations of anatomical structures, implants, surgical tools, etc.

[0086] For example, the surgeon can watch the displayed virtual handle on the virtual implant, and can manipulate (e.g., grasp and move) the virtual handle to move the virtual implant to the desired posture and adjust the planned implant placement relative to the graphical representation of the anatomical structure. Afterwards, during surgery, the computer platform 910 can display navigation information that helps the surgeon to more accurately follow the surgical plan to insert the implant and / or perform another surgical procedure on the anatomical structure through the XR head-mounted device 920. When the surgical procedure involves bone removal, the progress (e.g., cutting depth) of bone removal can be displayed in real time by the XR head-mounted device 920. Other features that can be displayed by the XR head-mounted device 920 can include but are not limited to gaps or ligament balance along the range of joint motion, contact lines along the range of joint motion on the implant, ligament tension and / or laxity rendered by color or other graphics, etc.

[0087] In some embodiments, the computer platform 910 can allow planning for the use of standard surgical tools and / or implants, such as posterior-stabilized implants and cruciate-retaining implants, cemented and cementless implants, and revision systems for surgical procedures associated with, for example, total or partial knee and / or hip replacements and / or trauma.

[0088] The automated imaging system may be used in conjunction with the computer platform 910 to acquire pre-operative, intra-operative, post-operative, and / or real-time image data of anatomical structures. Fig.10 and 11 An example automatic imaging system is shown. In some embodiments, the automatic imaging system is a C-arm 104 ( Fig.10 ) Imaging device or O-arm 106( Fig.11(Medtronic Navigation, Inc., with a location in Louisville, Colorado, USA, owns the O-arm ). It may be desirable to x-ray a patient from many different positions without the need for frequent manual repositioning of the patient, as may be required in an x-ray system. C-arm 104 x-ray diagnostic equipment can solve the problem of frequent manual repositioning and is well known in the medical field for surgical and other interventional procedures. Fig.10 As shown, the C-arm includes an elongated C-shaped member terminating in a "C" shaped opposite distal end 112. The C-shaped member is attached to an x-ray source 114 and an image receptor 116. The space within the C-arm 104 of the arm provides space for a physician to care for a patient substantially uninterrupted by the x-ray support structure.

[0089] The C-arm is mounted so that the arm can be rotationally moved in two degrees of freedom (i.e., about two perpendicular axes in a spherical motion). The C-arm is slidably mounted to the x-ray support structure, which allows the C-arm to perform orbital rotational movement about its center of curvature, which can allow for selective vertical and / or horizontal orientation of the x-ray source 114 and image receptor 116. The C-arm can also be rotatable laterally (i.e., in a perpendicular direction relative to the direction of orbital motion to enable selective adjustment of the positioning of the x-ray source 114 and image receptor 116 relative to the width and length of the patient). The spherical rotational aspect of the C-arm device allows the physician to x-ray the patient at an optimal angle determined relative to the specific anatomy being imaged.

[0090] Fig.11 O-arm shown in 106 includes a gantry housing 124, which can enclose an image capture portion that is not shown. The image capture portion includes an x-ray source portion and / or an x-ray emitting portion and an x-ray receiving portion and / or an image receiving portion, which can be positioned approximately one hundred and eighty degrees from each other and mounted on a rotor (not shown) relative to the track of the image capture portion. The image capture portion can be operatively rotated three hundred and sixty degrees during image acquisition. The image capture portion can rotate about a center point or axis, thereby allowing image data of the patient to be acquired from multiple directions or in multiple planes.

[0091] O-arm with stand housing 124 The gantry housing 106 has a central opening for positioning around the object to be imaged, a radiation source that can be rotated around the interior of the gantry housing 124, and the radiation source can be adapted to project radiation from a plurality of different projection angles. The detector system is adapted to detect the radiation at each projection angle, thereby acquiring an image of the object from a plurality of projection planes in a quasi-simultaneous manner. The gantry can be attached to the support structure O-arm in a cantilever manner. A support structure, such as a wheeled mobile cart with wheels. The positioning unit preferably translates and / or tilts the gantry to a planned position and orientation under the control of a computerized motion control system. The gantry may include a source and a detector positioned relative to each other on the gantry. The source and the detector may be secured to a motorized rotor that can rotate the source and the detector in combination with each other around the interior of the gantry. The source may be pulsed in multiple positions and orientations in a partial and / or full three hundred and sixty degree rotation to perform multi-planar imaging of a target object positioned inside the gantry. The gantry may further include a track and bearing system for guiding the rotor as it rotates, the track and bearing system may carry the source and the detector. O-arm Both and / or one of the surgical system 106 and the C-arm 104 may be used as an automated imaging system to scan the patient and send the information to the surgical system 2 .

[0092] The images captured by the imaging system may be displayed on the XR head mounted device 920 and / or another display device of the computer platform 910, the surgical robot 4, and / or another component of the surgical system 900. The XR head mounted device 920 may be connected to one or more of the imaging devices 104 and / or 106 and / or the image database 950, for example, through the computer platform 910, to display images therefrom. The user may provide control input (e.g., gesture-based and / or voice-based commands) through the XR head mounted device 920 to control the operation of one or more of the imaging devices 104 and / or 106 and / or the image database 950.

[0093] Fig.12 A block diagram of components of a surgical system including a pair of XR head mounted devices 1200 and 1210 (head mounted displays HMD1 and HMD2) is shown, and the XR head mounted devices may correspond to Fig.13 The XR head mounted device 920 shown in FIG. 1 and operates according to some embodiments of the present disclosure.

[0094] refer to Fig.12 In the example scenario of FIG. 6 , both assistant 612 and surgeon 610 wear XR headsets 1200 and 1210, respectively. It is optional for assistant 612 to wear XR headset 1210. As will be further described below, XR headsets 1200 and 1210 are configured to provide an interactive environment through which the wearer can view and interact with information related to the surgical procedure. This interactive XR-based environment can eliminate the need for technician 614 to be present in the operating room and can eliminate the need for the use of Figure 6Each XR headset 1200 and 1210 may include one or more cameras configured to provide an additional source for tracking a DRA or other reference array attached to an instrument, anatomy, end effector 26, and / or other device. Fig.12 In the example, the XR head-mounted device 1200 has a field of view (FOV) 1202 for tracking the DRA and other objects, the XR head-mounted device 1210 has a FOV 1212 for tracking the DRA and other objects that partially overlaps with the FOV 1202, and the navigation camera 46 has another FOV 600 for tracking the DRA and other objects that partially overlaps with the FOVs 1202 and 1212.

[0095] If one or more cameras are obstructed from viewing a DRA attached to a tracked object (e.g., a surgical instrument), but the DRA is in view of one or more other cameras, the tracking subsystem 830 and / or navigation controller 828 can continue to seamlessly track the object without losing navigation. Additionally, if the DRA is partially obscured from the perspective of one camera, but the entire DRA is visible through multiple camera sources, the tracking inputs of the cameras can be merged to continue navigating the DRA. One of the XR headsets and / or navigation cameras 46 can view and track the DRA on another XR headset to allow the computer platform 910 ( Fig. 9 and 14 ), the tracking subsystem 830 and / or another computing component can determine the pose of the DRA relative to one or more defined coordinate systems, such as the XR head-mounted device 1200 / 1210, the navigation camera 46, and / or another coordinate system defined for the patient, table and / or room.

[0096] XR headsets 1200 and 1210 may be operatively connected to view video, pictures, and / or other information received from a surgical operating room and / or provide commands to control various equipment in the operating room, including but not limited to neuromonitoring, microscopes, cameras, and anesthesia systems. Data from the various equipment may be processed and displayed within the headset, such as displaying a patient's vital signs or a microscope feed.

[0097] Example XR headset components and integration with navigated surgery, surgical robots, and other devices

[0098] Fig.13An XR head-mounted device 920 configured in accordance with some embodiments of the present disclosure is shown. The XR head-mounted device includes a headband 1306 configured to secure the XR head-mounted device to the head of a wearer, an electronics housing 1304 supported by the headband 1306, and a display screen 1302 extending laterally across and downwardly from the electronics housing 1304. The display screen 1302 may be a see-through LCD display device or a semi-reflective lens that reflects an image projected by the display device toward the wearer's eyes. A set of DRA reference points (e.g., points) are drawn or attached in a known spaced manner to be arranged on one or both sides of the head-mounted device. The DRA on the head-mounted device enables a navigation camera on an auxiliary tracking pole to track the pose of the head-mounted device 920 and / or enables another XR head-mounted device to track the pose of the head-mounted device 920.

[0099] Display screen 1302 operates as a perspective display screen (also referred to as a combiner) that reflects light from a display panel of a display device toward the user's eyes. The display panel can be positioned between the electronics housing and the user's head, and angled to project virtual content toward display screen 1302 to reflect toward the user's eyes. Display screen 1302 is semi-transparent and semi-reflective, allowing the user to see the reflected virtual content overlaid on the user's view of the real world scene. Display screen 1302 can have different opacity areas, such as the upper lateral bands whose opacity is higher than the lower lateral bands as shown. The opacity of display screen 1302 can be electrically controlled to adjust the amount of light from the real world scene passing through to reach the user's eyes. The high opacity configuration of display screen 1302 enables a high contrast virtual image to be overlaid on a dim view of the real world scene. The low opacity configuration of display screen 1302 can enable a more blurred virtual image to be overlaid on a clearer view of the real world scene. Opacity can be controlled by applying an opaque material on the surface of display screen 1302.

[0100] According to some embodiments, the surgical system includes an XR headset 920 and an XR headset controller, such as Fig.14 Controller 1430 or Fig.15 1430 in the controller. The XR head mounted device 920 is configured to be worn by a user during a surgical procedure and has a see-through display screen 1302 that is configured to display an XR image and allow at least a portion of a real-world scene to pass through the see-through display screen for viewing by the user. The XR head mounted device 920 also includes an opacity filter that is positioned between at least one of the user's eyes and the real-world scene when the see-through display screen 1302 is viewed by the user. The opacity filter is configured to provide opacity to light from the real-world scene. The XR head mounted device controller is configured to communicate with a navigation controller (e.g., Fig.14 one or more controllers 828A, 828B and / or 828C) in the navigation controller to receive navigation information from the navigation controller to provide guidance to the user during a surgical procedure performed on the anatomical structure, and is further configured to generate XR images based on the navigation information for display on the perspective display screen 1302.

[0101] The opacity of the display screen 1302 may be configured to have a gradient of opacity that changes more continuously with distance downward from the top portion of the display screen 1302. The darkest point of the gradient may be located at the top portion of the display screen 1302 and gradually become less transparent further down the display screen 1302 until the opacity becomes transparent or non-existent. In another example embodiment, the gradient may change from about 90% opacity to fully transparent at approximately mid-eye level of the display screen 1302. With the head mounted device properly calibrated and placed, the mid-eye level may correspond to the point at which the user will be looking straight ahead, and the end of the gradient will be located at the "horizontal" line of the eye. The darker portion of the gradient will allow for clear and unambiguous vision of the virtual content and help block the intrusive brightness of the overhead operating room lights.

[0102] Using an opacity filter in this manner enables the XR headset 920 to provide virtual reality ("VR") functionality along an upper portion of the display screen 1302 and AR functionality along a middle or lower portion of the display screen 1302 by substantially or completely blocking light from the real-world scene. This allows the user to have semi-transparent AR where it is needed and allows for the use of clear optics for the patient's anatomy during procedures. Configuring the display screen 1302 as a gradient rather than a more constant band of opacity can allow the wearer to experience a more natural transition between a more VR-type view and a more AR-type view without experiencing abrupt changes in the brightness of the real-world scene and depth of field that can otherwise fatigue the eyes, such as during more rapid shifts between upper and lower views.

[0103] The display panel and display screen 1302 can be configured to provide a wide field of view perspective XR display system. In one example configuration, they provide a user with an 80° diagonal field of view (FOV) and a 55° vertical coverage for the user to view virtual content. Other diagonal FOV angles and vertical coverage angles can be provided by display panels of different sizes, lenses of different curvatures, and / or different distances and angular orientations between the display panel and the curved display screen 1302. In some embodiments, an XR head-mounted device with a wide field of view allows the XR head-mounted device to display real-world registered XR images to the user, while the user's field of view is not centered on the real-world coordinates associated with the real-world registered XR images.

[0104] Fig.14The electrical components of an XR head-mounted device 920 according to various embodiments of the present disclosure are shown, and the XR head-mounted device can be operably connected to a computer platform 910, one or more imaging devices among the imaging devices (such as a C-arm imaging device 104, an O-arm imaging device 106) and / or an image database 950 and / or a surgical robot 800.

[0105] The XR headset 920 provides an improved human-machine interface for performing a navigated surgical procedure. The XR headset 920 may be configured, for example, through the computer platform 910 to provide functionality including, but not limited to, any one or more of: recognition of gesture-based commands and / or voice-based commands, display of XR graphics objects on the display device 1450. The display device 1450 may be a video projector, a flat panel display, etc. that projects the displayed XR graphics objects on the display screen 1302. The user may anchor the XR graphics objects to the display screen 1302 ( Fig.13 ) to be viewed over an overlay of a real-world object being viewed. Additionally or alternatively, the XR head-mounted device 920 may be configured to display video feeds from cameras and other cameras mounted to one or more XR head-mounted devices 920 on the display screen 1450.

[0106] The electrical components of the XR head mounted device 920 may include a plurality of cameras 1440, a microphone 1442, a gesture sensor 1444, a posture sensor (e.g., an inertial measurement unit (“IMU”)) 1446, a display module 1448 including a display device 1450, and a wireless / wired communication interface 1452. As will be explained below, the cameras 1440 of the XR head mounted device may be visible light capture cameras, near infrared capture cameras, or a combination of both.

[0107] The camera 1440 may be configured to operate as a gesture sensor 1444 by capturing user gestures performed within the field of view of one or more cameras 1440 for recognition. Alternatively, the gesture sensor 1444 may be a proximity sensor and / or a touch sensor that senses a gesture performed in proximity to the gesture sensor 1444 and / or senses physical contact (e.g., tapping the sensor or housing 1304). The posture sensor 1446 (e.g., an IMU) may include a multi-axis accelerometer, a tilt sensor, and / or another sensor that can sense rotation and / or acceleration of the XR head mounted device 920 along one or more defined coordinate axes. Some or all of these electrical components may be contained in the component housing 1304 or may be contained in another housing that is configured to be worn elsewhere, such as on the hip or shoulder.

[0108] As explained above, the surgical system 2 includes a camera tracking system component 6 / 6′ and a tracking subsystem 830, which can be part of the computer platform 910. The surgical system can include an imaging device (e.g., a C-arm 104, an O-arm 106, and / or an image database 950) and / or a surgical robot 4. The tracking subsystem 830 is configured to determine the pose of a DRA attached to an anatomical structure, an end effector, a surgical tool, etc. The navigation controller 828 is configured to determine the pose of a DRA attached to an anatomical structure, an end effector, a surgical tool, etc. based on a surgical plan (e.g., based on a surgical plan defined by a computer system 910, wherein the surgical tool is to be used to perform a surgical procedure on the anatomical structure). Fig. 9 The navigation controller 828 may be configured to determine a target pose of the surgical tool relative to the anatomical structure based on the pose of the anatomical structure determined by the tracking subsystem 830. The navigation controller 828 may be further configured to generate manipulation information based on the target pose of the surgical tool, the pose of the anatomical structure, and the pose of the surgical tool and / or the end effector, wherein the manipulation information indicates a position to which the surgical tool and / or the end effector of the surgical robot should be moved to execute the surgical plan.

[0109] The electrical components of the XR head-mounted device 920 can be operatively connected to the electrical components of the computer platform 910 through the wired / wireless interface 1452. The electrical components of the XR head-mounted device 920 can be operatively connected to various imaging devices (e.g., C-arm imaging device 104, I / O-arm imaging device 106, image database 950) and / or other medical equipment, for example, through the computer platform 910 or directly connected through the wired / wireless interface 1452.

[0110] The surgical system 2 further includes at least one XR headset controller 1430 (also referred to as an "XR headset controller" for brevity), which may reside in the XR headset 920, the computer platform 910, and / or in another system component connected by a wired cable and / or a wireless communication link. The software executed by the XR headset controller 1430 provides various functions. The XR headset controller 1430 is configured to receive navigation information from the navigation controller 828 to provide guidance to the user during a surgical procedure performed on an anatomical structure, and is configured to generate an XR image based on the navigation information for display on the display device 1450 for projection on the perspective display screen 1302.

[0111] The configuration of the display device 1450 relative to the display screen (also referred to as a “see-through display screen”) 1302 is configured to display the XR images in a manner such that the XR images appear to be in the real world when viewed through the display screen 1302 by a user wearing the XR head mounted device 920. The display screen 1302 may be positioned in front of the user's eyes by a head strap 1306.

[0112] The XR headset controller 1430 may be within a housing configured to be worn on the user's head or other part of the user's body while viewing the display screen 1302, or may be positioned away from the user viewing the display screen 1302 while communicatively connected to the display screen 1302. The XR headset controller 1430 may be configured to operatively process signaling from the camera 1440, microphone 142, and / or gesture sensor 1446, and connect to display an XR image on the display device 1450 for viewing by the user on the display screen 1302. Thus, the XR headset controller 1430, shown as a circuit box within the XR headset 920, should be understood to be operatively connected to the other shown components of the XR headset 920, but not necessarily reside in a common housing (e.g., Fig.13 For example, the XR head mounted device controller 1430 may reside in the computer platform 910, which in turn may reside in the computer platform 910. Figure 3B and 3C The computer is shown inside the housing of the tracking system assembly 6'.

[0113] Example user viewing through an XR headset

[0114] The XR head mounted device operation can display both 2D images and 3D models on the display screen 1302. The 2D image can be preferably displayed in the less transparent band (upper band) of the display screen 1302, and the 3D model can be more preferably displayed in the more transparent band (otherwise referred to as the environment area) (bottom band) of the display screen 1302. Below the lower band where the display screen 1302 ends, the wearer can have an unobstructed view of the surgical operating room. Note that the location where the XR content is displayed on the display screen 1302 can be fluid. The location where the 3D content is displayed may move to the opaque band depending on the positioning of the head mounted device relative to the content, and the location where the 2D content is displayed can be placed in the transparent band and stabilized relative to the real world. Additionally, the entire display screen 1302 can be darkened under electrical control to convert the head mounted device into a virtual reality for surgical planning or completely transparent during a medical procedure. As explained above, the XR head mounted device 920 and associated operations not only support navigated procedures, but can also be performed in conjunction with robot-assisted procedures.

[0115] Fig.16 An example view of a display screen 1302 of an XR headset 920 for providing navigation assistance to a user manipulating a surgical tool 1602 during a medical procedure is shown in accordance with some embodiments of the present disclosure. Fig.16 , when the surgical tool 1602 is brought into proximity with the tracked anatomical structure so that the dynamic reference arrays 1630 and 1632 connected to the surgical tool 1602 become in the camera 1440 ( Fig.15 ) and / 46( Figure 6 ), a graphical representation of a tool 1600 may be displayed in 2D and / or 3D images relative to a graphical representation of an anatomical structure 1610. The user may use the viewed graphical representation to adjust a trajectory 1620 of a surgical tool 1602, which may be shown extending from the graphical representation of the tool 2000 through the graphical representation of the anatomical structure 1610. The XR headset 920 may also display textual information and other objects 1640. A dashed line 1650 extending across the viewed display screen represents an example division between upper and lower bands of different opacity levels.

[0116] Other types of XR images (virtual content) that may be displayed on display screen 1302 may include, but are not limited to, any one or more of the following:

[0117] I) 2D axial, sagittal and / or coronal views of the patient’s anatomy;

[0118] 2) an overlay of the planned tool and surgical implant positions over the currently tracked tool and surgical implant positions;

[0119] 3) Preoperative image library;

[0120] 4) Video feeds from microscopes and other similar systems or remote video conferencing;

[0121] 5) Options and configuration settings and buttons;

[0122] 6) A floating 3D model of the patient’s anatomy with surgical planning information;

[0123] 7) Real-time tracking of surgical instruments relative to floating patient anatomy;

[0124] 8) Enhanced overlay of patient anatomy with instructions and guidance; and

[0125] 9) Enhanced overlay of surgical equipment.

[0126] Example configuration of cameras for tracking system components

[0127] Fig.17An example configuration of an auxiliary tracking rod 46 with two pairs of stereo navigation cameras configured according to some embodiments of the present disclosure is shown. The auxiliary tracking rod 46 is Figure 3A , 3B and 3C camera tracking system assembly. According to one embodiment, the stereo navigation camera includes a pair of stereo spaced visible light capture cameras and another pair of stereo spaced near infrared capture cameras. Alternatively, only a pair of stereo visible light capture cameras or only a pair of stereo near infrared capture cameras may be used in the auxiliary tracking pole 46. Any number of near infrared and / or visible light cameras may be used.

[0128] Attitude measurement link

[0129] As explained above, navigated surgery can include computer vision tracking and determination of the pose (e.g., location and orientation in a six-degree-of-freedom coordinate system) of a surgical instrument, such as by determining the pose of an attached DRA that includes spaced-apart fiducials (e.g., disks or spheres) arranged in a known manner. The computer vision uses spaced-apart navigation cameras (e.g., stereo cameras) configured to capture near-infrared and / or visible light. In this scenario, there are three parameters that compete for optimization during a surgical procedure: (1) accuracy, (2) robustness, and (3) ergonomics.

[0130] Some further aspects of the present disclosure relate to computer operations that combine (link) measured poses in a manner that can improve optimization of one or more of the three parameters described above by incorporating additional navigation cameras mounted to one or more XR headsets. Fig.17 As shown, according to some embodiments of the present disclosure, a pair of stereoscopic visible light tracking cameras and another pair of stereoscopic near-infrared tracking navigation cameras can be attached to an auxiliary tracking pole of a camera tracking system assembly. An operating algorithm is disclosed that analyzes a fully observed or partially observed pose of a DRA (e.g., when less than all of the fiducials of the DRA are viewed by a pair of stereo cameras) and combines the observed poses or partial poses in a manner that can improve accuracy, robustness, and / or ergonomics during a navigated surgical procedure.

[0131] As explained above, the XR head-mounted device can be configured to enhance the real world scene with a computer-generated XR image. The XR head-mounted device can be configured to provide an XR viewing environment by displaying a computer-generated XR image on a perspective display screen that allows light from the real world scene to pass through for combined viewing by the user. Alternatively, the XR head-mounted device can be configured to provide a VR viewing environment by preventing or substantially preventing light from the real world scene from being directly viewed by the user along the viewing path of the displayed XR image. The XR head-mounted device can be configured to provide both an AR viewing environment and a VR viewing environment. In one embodiment, both the AR viewing environment and the VR viewing environment are provided by lateral bands with significantly different opacities arranged between the perspective display screen and the real world scene, so that a VR viewing environment is provided for an XR image aligned with a high opacity band and an AR viewing environment is provided for an XR image aligned with a low opacity band. In another embodiment, both the AR viewing environment and the VR viewing environment are provided by a computer-adjustable control of an opacity filter that variably limits the amount of light from the real world scene passing through the perspective display screen to be combined with the XR image viewed by the user. Therefore, an XR head-mounted device may also be referred to as an AR head-mounted device or a VR head-mounted device.

[0132] As explained above, the XR head-mounted device may include a near infrared tracking camera and / or a visible light tracking camera configured to track the fiducials of a DRA connected to a surgical instrument, a patient anatomy, one or more other XR head-mounted devices, and / or a robotic end effector. The use of near infrared tracking and / or visible light tracking on the XR head-mounted device provides additional tracking volume coverage beyond what a camera on a single auxiliary tracking pole can provide. Adding a near infrared tracking camera to an existing auxiliary tracking pole allows the head-mounted device position to be tracked more robustly but less accurately than in visible light. Using stereo matching to jointly identify the pose of the DRA fiducials between the visible tracking coordinate system and the near infrared tracking coordinate system enables mechanical calibration of the visible tracking coordinate system and the near infrared tracking coordinate system so that the coordinate systems can be fully aligned to perform a 3D DRA fiducial triangulation operation. Using both visible and near-infrared tracking coordinate systems can achieve any one or more of the following: (a) identification of tools that cannot be identified using a single coordinate system; (b) improved pose tracking accuracy; (c) enabling a wide range of motion without losing tracking of the surgical instrument, patient anatomy, and / or robotic end effector; and (d) natural tracking of the XR headset in the same coordinate system as the surgical instrument being navigated.

[0133] Virtual models of planned instrument attachments to reduce incorrect selection

[0134] During surgical operation, instrument attachments can be attached to instruments. Instrument attachments can be implants (e.g., pedicle screws or expandable cages) or tool enhancers (e.g., curved tips). In some instances, the surgeon selects instrument attachments and attaches instrument attachments to instruments. In additional or alternative instances, the surgeon requests instrument attachments, and an assistant (e.g., a technician or nurse) provides instrument attachments and / or instruments with instrument attachments attached to the surgeon or surgical robot. In additional or alternative instances, the surgical plan instructs the surgeon, assistant, or surgical robot to use instrument attachments, and the corresponding surgeon, assistant, or surgical robot selects instrument attachments. During this operation (request / select instrument attachments), incorrect transmission may result in incorrect instrument attachments being provided. For example, an implant of the wrong size (e.g., length, width, or diameter) may be provided. Implants have many different options in size and product line, so nurses or technicians may select and prepare the wrong implant. In addition, in some instances, the surgeon or surgical robot may not know that the instrument attachment provided is incorrect.

[0135] Detecting an incorrect device attachment prior to insertion into a patient but after the sterility of the device attachment has been compromised can result in wasted (e.g., unusable) device attachments. More serious complications may result if insertion or use of the device attachment is attempted. For example, an incorrectly sized implant may cause injury to a patient when attempted implantation is attempted.

[0136] Various embodiments herein describe a virtual model of a planned instrument attachment that is displayed to a surgeon by an XR headset to allow the surgeon to verify that a physical (e.g., real-world) instrument attachment that has been provided is the planned instrument attachment. Displaying a virtual model of a planned instrument attachment can reduce the likelihood of using the wrong instrument attachment (e.g., placing the wrong implant into a patient).

[0137] Fig.18A -C shows an example of a view through the XR headset 920. Fig.18A In the example, a virtual model 1830 of a planned instrument attachment is displayed at a predetermined offset and / or pose relative to the XR headset 920. The surgeon can select a planned instrument through the XR headset 920 or based on a predetermined surgical plan. In this example, the physical instrument attachment 1820 (e.g., a real-world instrument attachment) is also visible. The virtual model 1830 includes a three-dimensional representation of the planned instrument attachment. Fig.18BIn the embodiment of the present invention, the virtual model 1830 includes virtual text describing the features of the planned instrument attachment (e.g., size, thread type, shape, text). In some examples, the virtual model includes both the three-dimensional representation and the virtual text describing the features of the planned instrument attachment. A surgeon viewing the physical instrument attachment 1820 and the virtual model 1830 through the XR headset 920 can determine whether the planned physical instrument has been provided (e.g., whether the physical instrument 1820 matches the planned instrument).

[0138] In some embodiments, the virtual model 1830 of the planned instrument accessory is displayed based on the pose of the XR head-mounted device 920 relative to the pose of the physical instrument accessory 1820. In some instances, the XR head-mounted device determines the scale and / or orientation of the virtual model 1830 of the planned instrument accessory based on the pose of the XR head-mounted device 920 relative to the pose of the physical instrument accessory 1820. Thus, the virtual model 1830 of the planned instrument accessory is displayed to the user in the same scale and / or orientation as a physical version of the planned instrument accessory would appear if it were in the pose of the physical instrument accessory 1820.

[0139] exist Fig.18C , a virtual model 1830 of a planned instrument attachment is displayed as an overlay of a physical instrument attachment 1820. In this example, the virtual model 1830 is partially transparent so that the physical instrument attachment 1820 is visible through the virtual model 1830, and the surgeon can observe whether there are any differences between the virtual model 1830 and the physical instrument attachment 1820. In additional or alternative examples, the virtual model 1830 can be opaque (e.g., a solid color) and displayed as appearing directly in front of the physical instrument attachment 1820, so that if any portion of the physical instrument attachment 1820 is visible to the surgeon, the surgeon can confirm that the physical instrument attachment 1820 is incorrect. In additional or alternative examples, the virtual model 1830 can be opaque and displayed as appearing directly behind the physical instrument attachment 1820, so that if any portion of the virtual model is visible, the surgeon can confirm that the physical instrument attachment 1820 is incorrect. The surgeon can control different poses of the virtual model 1830 relative to the physical instrument attachment 1820 (and switch between them).

[0140] Fig.19A -C shows an example of a view through an XR headset 920, where a physical device attachment 1820 is visible, but is not attached to a physical (real-world) device 1950. Fig.19A In FIG. 1 , the instrument 1950 is visible through the display of the XR headset 920. The surgeon may provide user input to the XR headset 920 or the XR headset controller to Fig.19BThe virtual model 1830 of the planned instrument accessory is shown as being attached to the instrument 1950. The instrument 1950 may include markers tracked by the camera tracking system, and the virtual model 1830 of the planned instrument accessory may be displayed based on the pose of the instrument 1950 relative to the XR headset 920 so that the virtual model 1830 appears to extend from the end of the instrument 1950 to which the planned instrument accessory will be attached.

[0141] In some examples, the user input includes moving the instrument 1950 to a particular real-world position or orientation relative to the XR headset 920. In additional or alternative examples, the user input is during the planning stage and involves assigning a planned instrument accessory to the instrument 1950, such that the XR headset 920 automatically associates the planned instrument accessory with the instrument 1950.

[0142] exist Fig.19C , the virtual model 1830 is shown attached to the instrument 1950, but in the orientation of the physical instrument attachment 1820. The camera tracking system can determine the pose of the physical instrument attachment 1820, and the XR head mounted device can display the virtual model 1830 based on the pose of the physical instrument attachment 1820 and the instrument 1950 relative to the pose of the XR head mounted device 920.

[0143] In some embodiments, the physical instrument attachment 1820 may be attached to the instrument 1950 and / or the surgeon may position the physical instrument attachment 1820 to be attached to the instrument 1950. If the virtual model 1830 and the physical instrument attachment 1820 are aligned, the surgeon has an additional degree of certainty that the correct selection has been made.

[0144] Fig. 20A -C shows an example of a view through the XR headset 920, where the virtual model 1830 is displayed as attached to the instrument 1950 to which the physical instrument attachment 1820 is currently attached. Fig. 20A , the virtual model 1830 is partially transparent, and the physical instrument 1820 is partially visible through the virtual model 1830. The surgeon wearing the XR head mounted device 920 can determine that the physical instrument attachment 1820 is correct based on comparing the size and features of the physical instrument attachment 1820 and the virtual model 1830.

[0145] exist Fig. 20B , the virtual model 1830 is opaque (e.g., a solid color) and appears to appear directly in front of the physical instrument attachment 1820. The surgeon wearing the XR headset 920 can determine that the physical instrument attachment 1820 is correct based on the inability to observe that the physical instrument 1820 surrounds a portion of the virtual model 1830.

[0146] exist Fig. 20C , the virtual model 1830 is opaque (e.g., a solid color) and appears to appear directly behind the physical instrument attachment 1820. The surgeon wearing the XR headset 920 can determine that the physical instrument attachment 1820 is correct based on the inability to observe that the virtual model 1830 surrounds a portion of the physical instrument attachment 1820.

[0147] In some embodiments, the surgeon can switch between displaying the virtual model 1830 as partially transparent, opaque and in front of the physical instrument attachment 1820, and opaque and behind the physical instrument attachment 1820 in order to determine whether the physical instrument attachment 1820 matches the planned instrument attachment.

[0148] exist Fig.18A In the examples shown in FIGS. 1-C, 19A-C, and 20A-C, the physical instrument attachment 1820 appears to match the virtual model 1830, and thus, the surgeon can determine that the physical instrument attachment 1820 is the correct and / or intended instrument attachment. Fig.21A -C, the physical instrument attachment 1820 does not appear to match the virtual model 2130 of the planned instrument attachment (eg, has a different length, width, shape, and thread pattern).

[0149] Fig.21A -C shows an example of a view through the XR headset 920, where the virtual model 2130 is shown attached to the instrument 1950 to which the physical instrument attachment 1820 is currently attached. Fig.21A , the virtual model 2130 is partially transparent, and the physical instrument 1820 is partially visible through the virtual model 2130. The surgeon wearing the XR head mounted device 920 can determine that the physical instrument attachment 1820 is incorrect based on being able to observe the difference between the physical instrument attachment 1820 and the virtual model 2130.

[0150] exist Fig.21B , the virtual model 2130 is opaque (e.g., a solid color) and appears to appear directly in front of the physical instrument attachment 1820. The surgeon wearing the XR headset 920 can determine that the physical instrument attachment 1820 is incorrect based on being able to observe that the physical instrument 1820 surrounds a portion of the virtual model 2130.

[0151] exist Fig. 21C , the virtual model 2130 is opaque (e.g., a solid color) and is displayed as appearing directly behind the physical instrument attachment 1820. The surgeon wearing the XR headset 920 can determine that the physical instrument attachment 1820 is incorrect based on being able to observe that the virtual model 2130 surrounds a portion of the physical instrument attachment 1820.

[0152] In some embodiments, the surgeon can switch between displaying the virtual model 2130 as partially transparent, opaque and in front of the physical instrument attachment 1820, and opaque and behind the physical instrument attachment 1820 in order to determine whether the physical instrument attachment 1820 matches the planned instrument attachment.

[0153] In some embodiments, surgical operation comprises inserting implant into patient's body.Implant is a piece of hardware (e.g., pedicle screw or expandable cage) to be inserted into patient's body.Surgeon selects implant based on various factors (including patient's age, bone health, vertebra size and surgeon's preference).In additional or alternative embodiments, surgical operation comprises using instrument with instrument extension attached.Instrument extension is a piece of hardware (e.g., curved tip) attached to instrument to allow surgeon to perform specific surgical operation.Surgeon selects instrument extension based on various factors (including surgical operation type, surgical operation position and surgeon's preference).Below provides further description of the process for ensuring that instrument attachment is correct about surgical operation involving implantation, but similar process also applies to surgical operation involving instrument extension.

[0154] Once the surgeon selects which implant to use, the surgeon communicates this selection via navigation software or the attending medical staff. Communication of the selection can be complicated, especially in the stressful and often noisy environment of the operating room, which can lead to incorrect delivery.

[0155] To help prevent this potentially catastrophic situation, the surgeon can use an XR headset. The XR headset can first show a projection of the screw the surgeon has selected and the appropriate size to confirm that the correct selection was made and delivered.

[0156] The surgeon will then confirm that the correct implant was selected. Once the implant has been attached to the instrument and handed to the surgeon, the XR headset can track the instrument and overlay a virtual image of the implant on top of where it should be in reality. If the virtual image and the real-world implant do not seem to match in size, shape, etc., the surgeon will know that the wrong implant was selected.

[0157] In additional or alternative embodiments, the process of displaying the virtual model involves performing a system check prior to the surgical procedure. For example, displaying the virtual model includes verifying that the camera is correctly tracking the instrument and the XR headset, that the XR headset is correctly positioned on the surgeon's head, and that the virtual content is correctly rendered.

[0158] Fig. 22is a flow chart of a process performed by a surgical system, such as surgical system 900. At block 2210, the camera tracking system 6, 6' determines a pose of the instrument relative to the XR head mounted device.

[0159] At box 2220, the XR headset controller 1430 generates a shape and pose of a virtual model of the planned instrument attachment. In some embodiments, the XR headset controller 1430 generates the shape and pose of the virtual model based on predetermined information associated with the planned instrument attachment and based on the pose of the instrument relative to the XR headset. The predetermined information may include characteristics of the planned instrument attachment, including at least one of size and shape. In additional or alternative embodiments, the planned instrument attachment is an implant (e.g., a pedicle screw or an expandable cage). The XR headset controller 1430 generates the virtual model of the planned instrument attachment to have graphical features representing the implant. In additional or alternative embodiments, the planned instrument attachment is an instrument extender. The XR headset controller 1430 generates the virtual model of the planned instrument attachment to have graphical features representing the instrument extender. In additional or alternative embodiments, the XR headset controller 1430 generates the virtual model to include virtual text indicating the characteristics of the planned instrument attachment.

[0160] In some embodiments, the XR head mounted device controller 1430 determines the pose of the virtual model of the planned instrument attachment as if it were attached to a certain location on the instrument when viewed through the see-through display screen. The real world scene may include a physical instrument attachment. In some embodiments, the physical instrument attachment is attached to a location on the instrument.

[0161] In additional or alternative embodiments, the XR head mounted device controller 1430 is further configured to generate the virtual model of the planned instrument accessory to have partially transparent graphical features, at least one of the partially transparent graphical features being posed to be substantially aligned with at least one corresponding feature of the physical instrument accessory when the virtual model of the planned instrument accessory is overlaid on the physical instrument accessory when viewed through the see-through display.

[0162] In additional or alternative embodiments, the XR head mounted device controller 1430 is further configured to generate the virtual model of the planned instrument accessory to have opaque graphical features, at least one of the opaque graphical features being posed to be substantially aligned with at least one corresponding feature of the physical instrument accessory when the virtual model of the planned instrument accessory is overlaid on the physical instrument accessory when viewed through the see-through display.

[0163] In additional or alternative embodiments, the XR head mounted device controller 1430 is further configured to generate the virtual model of the planned instrument accessory to have opaque graphical features, at least one of the opaque graphical features being posed to be substantially aligned with at least one corresponding feature of the physical instrument accessory when the virtual model of the planned instrument accessory is positioned behind the physical instrument accessory when viewed through the see-through display.

[0164] In additional or alternative embodiments, the XR head mounted device controller 1430 is further configured to selectively render the graphical features of the virtual model of the planned instrument accessory as one of partially transparent and opaque based on the user input. In additional or alternative embodiments, the XR head mounted device controller 1430 is further configured to selectively determine to display the pose of the virtual model of the planned instrument accessory as one of: an overlay on the physical instrument accessory, behind the physical instrument accessory, and separate from the physical instrument accessory based on the user input.

[0165] In additional or alternative embodiments, the XR head mounted device controller 1430 is further configured to scale the shape of the virtual model of the planned instrument accessory based on a distance determined based on a pose of the physical instrument accessory relative to the XR head mounted device.

[0166] At block 2230, the XR headset 920 displays a virtual model of the planned instrument attachment on a see-through display screen of the XR headset. In some embodiments, the XR headset 920 is configured to be worn by a user during a surgical procedure, and the see-through display screen allows at least a portion of a real-world scene to pass through the see-through display screen for viewing by the user. The real-world scene may include instruments and / or physical instrument attachments.

[0167] In some embodiments, displaying the virtual model as partially transparent and overlaid on the physical instrument attachment allows the user to determine whether the physical instrument attachment is different than the planned instrument attachment based on the virtual model of the planned instrument attachment and the physical instrument attachment being simultaneously visible.

[0168] In additional or alternative embodiments, displaying the virtual model as opaque and overlaid on the physical instrument accessory allows the user to determine whether the physical instrument accessory is different than the planned instrument accessory based on whether a portion of the physical instrument accessory is visible to the user.

[0169] In additional or alternative embodiments, displaying the virtual model as opaque and behind the physical instrument attachment allows the user to determine whether the physical instrument attachment is different than the planned instrument attachment based on whether a portion of the virtual model is visible to the user.

[0170] Regarding some embodiments of surgical systems and related methods, from Fig. 22 Various operations of the flowcharts may be optional.

[0171] Fig.23 is a flow chart of a process performed by a surgical system (e.g., surgical system 900). At box 2210, the camera tracking system 6, 6' determines the pose of a physical instrument attachment relative to the XR head-mounted device. In some embodiments, the physical instrument attachment is an implant (e.g., a pedicle screw or an expandable cage). In additional or alternative embodiments, the physical instrument attachment is an instrument expander. In additional or alternative embodiments, the camera tracking system 6, 6' determines the pose of the instrument relative to the XR head-mounted device. The physical instrument attachment can be attached to a location on the instrument.

[0172] At box 2220, the XR head mounted device controller 1430 generates a shape and a pose of the virtual model of the planned instrument accessory. In some embodiments, the shape and the pose of the virtual model of the planned instrument accessory are based on predetermined information of the planned instrument accessory and based on the pose of the physical instrument accessory relative to the XR head mounted device. In additional or alternative embodiments, the XR head mounted device controller 1430 generates the pose of the virtual model of the planned instrument accessory in a predetermined pose relative to the pose of the physical instrument accessory. In additional or alternative embodiments, the XR head mounted device controller 1430 adjusts the orientation of the virtual model of the planned instrument accessory relative to the XR head mounted device to match the orientation of the physical instrument accessory. In additional or alternative embodiments, the XR head mounted device controller scales the shape of the virtual model of the planned instrument accessory based on a distance determined according to the pose of the physical instrument accessory relative to the XR head mounted device.

[0173] In some embodiments, the XR head mounted device controller selectively determines based on the user input to display the pose of the virtual model of the planned instrument accessory as one of: an overlay on the physical instrument accessory, behind the physical instrument accessory, and separate from the physical instrument accessory. In additional or alternative embodiments, the XR head mounted device controller generates the virtual model to include virtual text indicating characteristics of the planned instrument accessory based on predetermined information associated with the planned instrument accessory.

[0174] At block 2230, the XR headset 920 displays a virtual model of the planned instrument attachment on a see-through display screen of the XR headset. In some embodiments, the XR headset is configured to be worn by a user during a surgical procedure, and the see-through display screen allows at least a portion of a real-world scene to pass through the see-through display screen for viewing by the user. The real-world scene may include physical instrument attachments and / or instruments.

[0175] Regarding some embodiments of surgical systems and related methods, from Fig. 22Various operations of the flowcharts may be optional.

[0176] Additional limitations and embodiments:

[0177] In the above description of each embodiment of the inventive concept, it should be understood that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the inventive concept. Unless otherwise specified, all terms used herein (including technical terms and scientific and technological terms) have the same meanings commonly understood by those skilled in the art to which the inventive concept belongs. It should be further understood that the terms as defined in the commonly used dictionary should be interpreted as having the meaning consistent with its meaning in the context of this specification and in the relevant field, and are clearly defined in this article, and will not be interpreted in an idealized or overly formal sense.

[0178] When an element is referred to as being "connected to" or "coupled to" "in response to" another element or its variant, it may be directly connected to, coupled to or responsive to another element, or there may be an intermediate element. In contrast, when an element is referred to as being "directly connected to", "directly coupled to", "directly responsive to" another element or its variant, there is no intermediate element. Similar elements are referred to with similar numbers throughout the text. In addition, "connection", "connection", "response" or its variants as used herein may include wireless connection, connection or response. Unless the context clearly indicates otherwise, as used herein, the singular forms "a, an" and "the" are intended to include plural forms. For the sake of brevity and / or clarity, well-known functions or configurations may not be described in detail. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0179] It should be understood that, although the terms first, second, third, etc. can be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish an element / operation from another element / operation. Therefore, without departing from the teaching of the present invention concept, the first element / operation in some embodiments can be referred to as the second element / operation in other embodiments. Throughout this specification, the same reference numerals or the same figure designators represent the same or similar elements.

[0180] As used herein, the terms "comprise / comprising / comprises", "include / including / includes", "have / has / having" or variations thereof are open ended and include one or more stated features, integers, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or combinations thereof. In addition, as used herein, the general abbreviation "such as", derived from the Latin phrase "exempli gratia", may be used to introduce or specify one or more general examples of previously mentioned items and is not intended to limit such items. The general abbreviation "that is", derived from the Latin phrase "id est", may be used to specify a specific item from a more general statement.

[0181] Example embodiments are described herein with reference to block diagrams and / or flow diagrams of computer-implemented methods, devices (systems and / or apparatuses), and / or computer program products. It should be understood that the blocks of the block diagrams and / or flow diagrams and the combination of blocks in the block diagrams and / or flow diagrams can be implemented by computer program instructions executed by one or more computer circuits. These computer program instructions can be provided to processor circuits of general-purpose computer circuits, special-purpose computer circuits, and / or other programmable data processing circuits to produce a machine so that instructions executed by processors of computers and / or other programmable data processing devices convert and control transistors, values ​​stored in memory locations, and other hardware components within such circuit systems to implement the functions / actions specified in the block diagrams and / or one or more flow diagram blocks, and thereby create a device (function) and / or structure for implementing the functions / actions specified in the block diagrams and / or one or more flow diagram blocks.

[0182] These tangible computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing device to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture containing instructions for implementing the functions / actions specified in the block diagram or one or more flowchart blocks. Thus, embodiments of the inventive concept may be embodied in hardware and / or software (including firmware, resident software, microcode, etc.) that runs on a processor such as a digital signal processor, which may be collectively referred to as a "circuitry system," "module," or variations thereof.

[0183] It should also be noted that, in some alternative implementations, the function / action marked in the frame may not occur in the order marked in the flow chart. For example, according to the function / action involved, the two frames shown continuously can actually be performed substantially simultaneously, or the frame can sometimes be performed in reverse order. In addition, the function of a given frame of a flow chart and / or block diagram can be divided into multiple frames, and / or the function of two or more frames of a flow chart and / or block diagram can be integrated at least in part. Finally, without departing from the scope of the inventive concept, other frames can be added / inserted between the frames shown, and / or frame / operation can be omitted. In addition, although some figures in the figure include arrows on the communication path to illustrate the main direction of communication, it should be understood that communication can occur in the direction opposite to the arrows depicted.

[0184] Without substantially departing from the principle of the inventive concept, many changes and modifications may be made to the embodiments. All such variants and changes are intended to be included in the scope of the inventive concept herein. Therefore, the subject matter disclosed above should be considered illustrative rather than restrictive, and the examples of the attached embodiments are intended to cover all such modifications, enhancements and other embodiments that fall within the spirit and scope of the inventive concept. Therefore, in order to be allowed to the greatest extent by law, the scope of the inventive concept will be determined by the most widely allowed interpretation of the present disclosure, which includes the examples of the following embodiments and their equivalents, and should not be limited to or confined to the aforementioned specific detailed description.

Claims

1. A surgical system, comprising: Devices with traceability markings; a physical instrument attachment (1820) configured to be attached to the instrument; a camera tracking system (1440) configured to track the instrument via the trackable marker; an extended reality head mounted device (920) configured to be worn by a user during a surgical procedure and comprising a see-through display screen (1302) configured to display a virtual model (1830) of a planned instrument attachment and to allow at least a portion of a real world scene to pass through the see-through display screen for viewing by the user, the real world scene comprising the instrument; as well as an extended reality head mounted device controller (1430) configured to generate a shape and a pose of the virtual model (1830) of the planned device accessory based on predetermined information associated with the planned device accessory and based on a pose of the device relative to the extended reality head mounted device (920), Wherein the extended reality head mounted device controller (1430) is further configured to selectively determine, based on user input, to display the pose of the virtual model of the planned instrument accessory as one of: an overlay on the physical instrument accessory, behind the physical instrument accessory, and separate from the physical instrument accessory.

2. A surgical system according to claim 1, wherein the extended reality head-mounted device controller is further configured to determine the pose of the virtual model of the planned instrument attachment to appear to be attached to a certain position on the instrument when viewed through the perspective display screen.

3. The surgical system of claim 2, wherein the real-world scene further comprises the physical instrument attachment attached to the location on the instrument.

4. A surgical system according to claim 3, wherein the extended reality head-mounted device controller is further configured to generate the virtual model of the planned instrument accessory to have partially transparent graphic features, and at least one of the partially transparent graphic features is posed so that when viewed through the perspective display screen, the virtual model of the planned instrument accessory is overlaid on the physical instrument accessory to be substantially aligned with at least one corresponding feature of the physical instrument accessory.

5. A surgical system according to claim 1, wherein the extended reality head-mounted device controller is further configured to generate the virtual model of the planned instrument accessory to have opaque graphic features, and at least one of the opaque graphic features is posed so that when viewed through the perspective display screen, the virtual model of the planned instrument accessory is substantially aligned with at least one corresponding feature of the physical instrument accessory when overlaid on the physical instrument accessory.

6. A surgical system according to claim 1, wherein the extended reality head-mounted device controller is further configured to generate the virtual model of the planned instrument accessory to have opaque graphic features, and at least one of the opaque graphic features is posed so that when viewed through the perspective display screen, the virtual model of the planned instrument accessory is substantially aligned with at least one corresponding feature of the physical instrument accessory when positioned behind the physical instrument accessory.

7. A surgical system according to claim 1, wherein the extended reality head mounted device controller is further configured to selectively render the graphical features of the virtual model of the planned instrument attachment as one of partially transparent and opaque based on user input.

8. A surgical system according to claim 1, wherein the extended reality head-mounted device controller is further configured to scale the shape of the virtual model (1830) of the planned instrument attachment (1820) based on a distance determined according to the posture of the physical instrument attachment (1820) relative to the extended reality head-mounted device (920).

9. The surgical system of claim 1, wherein the real-world scene further comprises physical instrument attachments (1820), and wherein The extended reality headset controller (1430) is further configured to generate the virtual model (1830) of the planned instrument attachment (1820) having a graphical feature representing at least one of a pedicle screw or an expandable cage.

Images