Registering surgical tools to a reference array tracked by an extended reality headset's camera for assisted navigation during surgery

Abstract

The present application relates to registering surgical tools with reference arrays tracked by cameras of an extended reality headset for assisted navigation during surgery. The present invention discloses a camera tracking system for computer-assisted navigation during surgery. The camera tracking system is configured to identify a reference array tracked by a set of tracking cameras attached to an extended reality (XR) headset and determine whether the reference array is registered to be paired with a characteristic of one of a plurality of surgical tools defined in a surgical tool database. The camera tracking system is further configured to, based on the reference array being determined to be unregistered and receiving user input, register the reference array to be paired with a characteristic of one of the plurality of surgical tools selected based on the user input. The camera tracking system is further configured to provide a representation of the characteristic to a display device of the XR headset for display to a user.

Description

Technical Field

[0001] The present disclosure relates to medical devices and systems, and more particularly, to a camera tracking system for computer-assisted navigation during surgery. Background Art

[0002] Computer-assisted navigation during surgery can provide surgeons with a computerized visualization of the current position of surgical tools relative to medical images of the patient's anatomical structures. Camera tracking systems for computer-assisted navigation typically use a set of cameras to track a set of reference points attached to the surgical tools, which are positioned by the surgeon or other user during surgery. A set of reference points, also known as a dynamic reference array or dynamic reference library (DRB), allows the camera tracking system to determine the position of the surgical tool relative to the anatomical structures in the medical image and relative to the patient for display to the surgeon. The surgeon can thus use real-time position feedback to navigate the surgical tool during the surgical procedure.

[0003] Extended reality (XR) headsets are used in conjunction with camera tracking systems to enable surgeons to view real-time pose feedback in the form of graphical objects overlaid on the patient. There is an ongoing need to enable surgeons wearing XR headsets to use multiple types of surgical tools while reducing errors in tool selection and reducing interruptions to the surgeon's attention during surgical procedures. Summary of the invention

[0004] Various embodiments disclosed herein relate to improvements in computer-assisted navigation during surgery. An XR headset is used to assist in registering the characteristics of a surgical tool with a reference array identified by a camera tracking system. A representation of the characteristics may be displayed on the XR headset to enable a user to confirm the correctness of the registration process. Using an XR headset in the registration process may provide surgeons and other medical personnel (users) with a more intuitive, time-saving, and reliable process for registering surgical tools with a camera tracking system before and / or during a surgical procedure.

[0005] In one embodiment, a camera tracking system for computer-assisted navigation during surgery is configured to identify a reference array tracked by a set of tracking cameras attached to an XR headset. The camera tracking system determines whether the reference array is registered to be paired with a characteristic of one of a plurality of surgical tools defined in a surgical tool database. Based on the reference array being determined to be unregistered and receiving user input, the camera tracking system registers the reference array to be paired with a characteristic of one of a plurality of surgical tools selected based on the user input. The camera tracking system then provides a representation of the characteristic to a display device of the XR headset for display to the user.

[0006] Related computer program products and methods for camera tracking systems are disclosed.

[0007] Other camera tracking systems, computer program products, and methods according to the embodiments will be or become apparent to those skilled in the art upon review of the following drawings and detailed description. It is contemplated that all such camera tracking systems, computer program products, and methods are included in this specification, within the scope of the present disclosure, and protected by the accompanying claims. Furthermore, it is contemplated that all embodiments disclosed herein may be implemented separately or combined in any manner and / or combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings illustrate certain non-limiting embodiments of the inventive concept, and are included to provide a further understanding of the present disclosure, and are incorporated in and constitute a part of this application. In the drawings:

[0009] Figure 1 An embodiment of a surgical system according to some embodiments of the present disclosure is shown;

[0010] Figure 2 Some embodiments of the present disclosure are shown Figure 1 Surgical robotic components of surgical systems;

[0011] Figure 3A Some embodiments of the present disclosure are shown Figure 1 A camera tracking system component of a surgical system;

[0012] Figure 3B and 3C It shows that some embodiments of the present disclosure can be used with Figure 1 a front view and an isometric view of another camera tracking system component used in conjunction with the surgical system;

[0013] Figure 4 An embodiment of an end effector connectable to a robotic arm and configured in accordance with some embodiments of the present disclosure is shown;

[0014] Figure 5 shows a medical procedure in which a surgical robot and a camera system are positioned around a patient;

[0015] Figure 6 Shown for medical operation Figure 5 A block diagram view of components of a surgical system;

[0016] Figure 7 shows that when using the navigation function of the surgical system, Figure 5 and 6 Various display screens displayed on the monitor;

[0017] Figure 8 A block diagram showing some electrical components of a surgical robot according to some embodiments of the present disclosure;

[0018] Fig. 9 A block diagram showing components of a surgical system including an imaging device connected to a computer platform that may be operatively connected to a camera tracking system and / or a surgical robot according to some embodiments of the present disclosure;

[0019] Fig.10 An embodiment of a C-arm imaging device that may be used in combination with a surgical robot according to some embodiments of the present disclosure is shown;

[0020] Fig.11 An embodiment of an O-arm imaging device that may be used in combination with a surgical robot according to some embodiments of the present disclosure is shown;

[0021] Fig.12 shows a block diagram view of components of a surgical system including a pair of XR headsets and auxiliary tracking poles operating according to some embodiments of the present disclosure;

[0022] Fig.13 An XR headset configured according to some embodiments of the present disclosure is shown;

[0023] Fig.14 shows electrical components of an XR headset that may be operably connected to a computer platform, one or more imaging devices, and / or a surgical robot according to some embodiments of the present disclosure;

[0024] Fig.15 shows a block diagram illustrating an arrangement of optical components of an XR headset according to some embodiments of the present disclosure;

[0025] Fig.16 shows an example view of a display screen through an XR headset for providing navigation assistance to manipulate a surgical tool during a medical procedure in accordance with some embodiments of the present disclosure;

[0026] Fig.17 shows an exemplary configuration of an auxiliary tracking pole with two pairs of stereo cameras configured in accordance with some embodiments of the present disclosure;

[0027] Fig.18 shows a block diagram view of components of a surgical system including a tracking camera in a pair of XR headsets and auxiliary tracking poles operating together according to some embodiments of the present disclosure;

[0028] Fig.19shows a graphical representation of surgical tool characteristics displayed by an XR headset when a reference array is registered to the surgical tool for use in computer-assisted navigation of the surgical tool during surgery in accordance with some embodiments of the present disclosure;

[0029] Fig. 20 The camera tracking system is shown to provide a Fig.19 A flowchart of a registration operation of a graphical representation of surgical tool characteristics;

[0030] Fig.21 shows a graphical representation of a registered, actively used surgical tool displayed by an XR headset in accordance with some embodiments of the present disclosure;

[0031] Fig. 22 The camera tracking system is shown to provide a Fig.21 a flowchart of the operation of a graphical representation of a registered surgical tool in active use;

[0032] Fig.23 shows a graphical representation of tracking accuracy information of a reference array for registration displayed by an XR headset while a user is viewing a surgical tool in accordance with some embodiments of the present disclosure;

[0033] Fig.24 The camera tracking system is shown to provide a Fig.23 A flowchart of the operation of a graphical representation of tracking accuracy information;

[0034] Fig.25 A flowchart illustrating operations performed by a camera tracking system to selectively allow a user to perform registration of a reference array based on whether the reference array is within a detection area defined relative to an XR headset in accordance with some embodiments of the present disclosure;

[0035] Fig.26 A flowchart illustrating operations performed by a camera tracking system to selectively display characteristics of a surgical tool registered with a reference array based on whether the reference array is within a detection area defined relative to an XR headset in accordance with some embodiments of the present disclosure;

[0036] Fig. 27 shows a surgical tool viewed through an XR headset with graphical indicia displayed as identified poses of fiducials of a reference array at least partially overlapping and graphical indicia displayed as estimated poses of missing fiducials at least partially overlapping, in accordance with some embodiments of the present disclosure;

[0037] Fig.28The method performed by the camera tracking system to display, for example, Fig. 27 A flowchart of the operation of the graphical markings shown in FIG. 1 relative to the identified and missing reference points;

[0038] Fig.29 A flowchart illustrating operations performed by a camera tracking system to selectively allow receipt of gesture input based on whether a user's hand is at least partially within a hand-user interface exclusion zone determined relative to a reference array in accordance with some embodiments of the present disclosure; and

[0039] Fig.30 A hand-user interface exclusion zone determined relative to a reference array according to some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0040] The inventive concept will now be described more fully below with reference to the accompanying drawings, in which examples of embodiments of the inventive concept are shown. The inventive concept can be implemented in many different forms and should not be construed as being limited to the embodiments listed herein. Rather, these embodiments are provided to make the disclosure more complete and complete, and to fully convey the scope of each inventive concept to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. A component in one embodiment may be defaulted to being present in or used in another embodiment.

[0041] Various embodiments disclosed herein relate to improvements in computer-assisted navigation during surgery. An extended reality (XR) headset is operably connected to a surgical system and is configured to provide an interactive environment through which surgeons, assistants, and / or other personnel can view and select from patient images, view and select from computer-generated surgical navigation information, and / or control surgical equipment in an operating room. As described below, the XR headset can be configured to augment a real-world scene with a computer-generated XR image. The XR headset can be configured to provide an augmented reality (AR) viewing environment by displaying a computer-generated XR image on a see-through display screen that allows light from a real-world scene to pass through for combined viewing by a user. Alternatively, the XR headset can be configured to provide a virtual reality (VR) viewing environment by preventing or substantially preventing light from a real-world scene from being directly viewed by a user when the user views a computer-generated AR image on a display screen. The XR headset can be configured to provide both AR and VR viewing environments. In one embodiment, both the AR viewing environment and the VR viewing environment are provided by lateral bands of significantly different opacities arranged between the see-through display screen and the real-world scene, such that a VR viewing environment is provided for XR images aligned with the high opacity bands and an AR viewing environment is provided for XR images aligned with the low opacity bands. In another embodiment, both the AR viewing environment and the VR viewing environment are provided by computer-adjustable control of an opacity filter that variably limits the amount of light from the real-world scene that passes through the see-through display screen to be combined with the XR image viewed by the user. Therefore, the XR head-mounted device may also be referred to as an AR head-mounted device or a VR head-mounted device.

[0042] Figure 1 An embodiment of a surgical system 2 according to some embodiments of the present disclosure is depicted. Before performing an orthopedic or other surgical procedure, one may use, for example, Fig.10 C-arm imaging device 104 or Fig.11 A three-dimensional ("3D") image scan of the planned surgical area of ​​the patient is taken from an O-arm imaging device 106, or from another medical imaging device such as a computed tomography (CT) image or MRI. Such a scan may be performed preoperatively (e.g., most commonly a few weeks prior to the procedure) or intraoperatively. However, any known 3D or 2D image scan may be used in accordance with various embodiments of surgical system 2. The image scan is sent to a computer platform in communication with surgical system 2, such as computer platform 910 ( Fig. 9 ), the surgical system may include a camera tracking system component 6, a surgical robot 4 (e.g., Figure 12), imaging devices (e.g., C-arm 104, O-arm 106, etc.), and an image database 950 for storing image scans of patients. View computer platform 910 ( Fig. 9 ) generates a surgical plan defining target poses of surgical tools to be used during a surgical procedure performed on the patient's anatomy. Example surgical tools, also referred to as tools, may include, but are not limited to, drills, screwdrivers, saws, retractors, and implants such as screws, spacers, intervertebral fusion devices, plates, rods, etc. In some embodiments, the surgical plan defining the target plane is planned on the 3D image scan displayed on the display device.

[0043] As used herein, the term "pose" refers to the position and / or rotation angle of an object (e.g., a dynamic reference array, an end effector, a surgical tool, an anatomical structure, etc.) relative to another object and / or a defined coordinate system. Therefore, the pose can be defined based only on the multi-dimensional position of an object relative to another object and / or a defined coordinate system, based only on the multi-dimensional rotation angle of the object relative to another object and / or a defined coordinate system, or a combination of the multi-dimensional position and the multi-dimensional rotation angle. Therefore, the term "pose" is used to refer to the position, the rotation angle, or a combination thereof.

[0044] Figure 1 The surgical system 2 can assist the surgeon during a medical procedure by, for example, providing for use of holding tools, alignment tools, application tools, guide tools, and / or positioning tools. In some embodiments, the surgical system 2 includes a surgical robot 4 and a camera tracking system assembly 6. The ability to mechanically couple the surgical robot 4 and the camera tracking system assembly 6 can allow the surgical system 2 to be manipulated and moved as a single unit, and allows the surgical system 2 to have a small footprint in an area, allowing for easier movement through narrow passages and around turns, and allowing for storage within a smaller area.

[0045] The surgical procedure can begin with the surgical system 2 being moved from a medical storage room to a medical procedure room. The surgical system 2 can be manipulated through doorways, halls, and elevators to reach the medical procedure room. Within the medical procedure room, the surgical system 2 can be physically divided into two separate and distinct systems (the surgical robot 4 and the camera tracking system assembly 6). The surgical robot 4 can be positioned adjacent to the patient in any suitable location to appropriately assist the medical staff. The camera tracking system assembly 6 can be positioned at the bottom of the patient, at the patient's shoulder, or at any other location suitable for tracking the current posture and posture movement of the surgical robot 4 and the patient's track portion. The surgical robot 4 and the camera tracking system assembly 6 can be powered by an onboard power supply and / or plugged into an external wall socket.

[0046] The surgical robot 4 may be used to assist the surgeon by holding and / or using tools during a medical procedure. In order to properly utilize and hold the tools, the surgical robot 4 may rely on a plurality of motors, computers, and / or actuators to function properly. Figure 1 As shown, the robot body 8 can serve as a structure in which multiple motors, computers, and / or actuators can be secured within the surgical robot 4. The robot body 8 can also provide support for the robotic telescopic support arm 16. The size of the robot body 8 can provide a stable platform to support attached components, and can house, conceal, and protect multiple motors, computers, and / or actuators that can operate the attached components.

[0047] The robot base 10 can act as a lower support for the surgical robot 4. In some embodiments, the robot base 10 can support the robot body 8 and can attach the robot body 8 to a plurality of powered wheels 12. This attachment to the wheels can allow the robot body 8 to move efficiently in space. The robot base 10 can run along the length and width of the robot body 8. The robot base 10 can be about two inches to about 10 inches high. The robot base 10 can cover, protect and support the powered wheels 12.

[0048] In some embodiments, Figure 1 As shown, at least one powered wheel 12 can be attached to the robot base 10. The powered wheel 12 can be attached to the robot base 10 at any position. Each individual powered wheel 12 can rotate around a vertical axis in any direction. The motor can be placed above, inside or adjacent to the powered wheel 12. Such a motor can allow the surgical system 2 to be manipulated to any position and stabilize and / or level the surgical system 2. A rod located inside or adjacent to the powered wheel 12 can be pressed into a surface by a motor. The rod not shown can be made of any suitable metal to lift the surgical system 2. The rod can lift the powered wheel 10 (which can lift the surgical system 2) to any height required to level or otherwise fix the surgical system 2 relative to the patient's orientation. The weight of the surgical system 2 is supported by a small contact area of ​​the rod on each wheel, preventing the surgical system 2 from moving during the medical procedure. This rigid positioning can prevent objects and / or people from accidentally moving the surgical system 2.

[0049] The robot track 14 can be used to facilitate moving the surgical system 2. The robot track 14 provides a person with the ability to move the surgical system 2 without grasping the robot body 8. Figure 1 As shown, the length of the robot track 14 can be the same length as the robot body 8, shorter than the robot body 8, and / or longer than the robot body 8. The robot track 14 can further provide protection to the robot body 8, thereby preventing the subject and / or medical personnel from contacting, hitting, or bumping into the robot body 8.

[0050] The robot body 8 may provide support for a selectively compliant articulated robotic arm, hereinafter referred to as a "SCARA". Due to the repeatability and compactness of the robotic arm, the use of the SCARA 24 within the surgical system 2 may be advantageous. The compactness of the SCARA may provide additional space within the medical procedure, which may allow a medical professional to perform the medical procedure without excessive clutter and restricted areas. The SCARA 24 may include a robotic telescopic support 16, a robotic support arm 18, and / or a robotic arm 20. The robotic telescopic support 16 may be positioned along the robot body 8. As shown in FIG. Figure 1 As shown, the robotic telescopic support 16 can provide support for the SCARA 24 and the display 34. In some embodiments, the robotic telescopic support 16 can extend and retract in a vertical direction. The body of the robotic telescopic support 16 can be any width and / or height configured to support the stress and weight placed thereon.

[0051] In some embodiments, the medical staff can move the SCARA 24 via commands submitted by the medical staff. As will be explained in further detail below, the commands can originate from a display 34, a tablet computer, and / or an XR headset (e.g., Fig. 9 The XR headset may eliminate the need for the medical staff to refer to any other display, such as the display 34 or a tablet computer, which enables the SCARA 24 to be configured without the display 34 and / or the tablet computer. As will be explained in further detail below, the command may be generated by pressing a switch and / or pressing multiple switches, and / or may be generated based on gesture commands and / or voice commands sensed by the XR headset.

[0052] like Figure 5 As shown in , the activation assembly 60 may include a switch and / or multiple switches. The activation assembly 60 may be operable to transmit movement commands to the SCARA 24, thereby allowing an operator to manually manipulate the SCARA 24. When the switch or multiple switches are pressed, the medical staff may have the ability to move the SCARA 24 through the applied hand movement. Alternatively or additionally, as will be explained in further detail below, the operator may control the movement of the SCARA 24 through gesture commands and / or voice commands sensed by the XR headset. Additionally, when the SCARA 24 does not receive a command to move, the SCARA 24 may be locked in place to prevent accidental movement by medical staff and / or other objects. By locking in place, the SCARA 24 provides a stable platform through which the end effector 26 can guide surgical tools during a medical procedure.

[0053] The robotic support arm 18 may be connected to the robotic telescopic support 16 by various mechanisms. Figure 1 and 2 , the robotic support arm 18 can rotate in any direction relative to the robotic telescopic support 16. The robotic support arm 18 can rotate three hundred and sixty degrees around the robotic telescopic support 16. The robotic arm 20 can be connected to the robotic support arm 18 in any suitable position and by various mechanisms that enable rotation in any direction relative to the robotic support arm 18. In one embodiment, the robotic arm 20 can rotate three hundred and sixty degrees relative to the robotic support arm 18. This free rotation allows the operator to position the robotic arm 20 according to the surgical plan.

[0054] Figure 4 and 5 The end effector 26 in the example is attached to the robotic arm 20 at any suitable position. The end effector 26 is configured to be attached to the end effector coupling 22 of the robotic arm 20 positioned by the surgical robot 4. The example end effector 26 includes a tubular guide that guides the movement of an inserted surgical tool relative to the anatomical structure on which the surgical procedure is to be performed.

[0055] In some embodiments, a dynamic reference array 52 is attached to the end effector 26. The dynamic reference array (also referred to herein as a "DRA" and "reference array") can be a rigid body, a marker or other marking attachable or formed on one or more XR headsets worn by personnel in the operating room, an end effector, a surgical robot, a surgical tool during navigation, and a patient's anatomical structure (e.g., bone). The computer platform 910, in combination with the camera tracking system assembly 6 or other 3D positioning system, is configured to track the pose (e.g., position and rotational orientation) of the DRA in real time. The DRA can include reference points, such as the illustrated arrangement of a sphere. Such tracking of the 3D coordinates of the DRA can allow the surgical system 2 to determine the DRA's relative position to the end effector. Figure 5 The pose in any multi-dimensional space of the target anatomical structure of the patient 50.

[0056] like Figure 1 As shown, the light indicator 28 can be positioned on top of the SCARA 24. The light indicator 28 can be any type of light that illuminates to indicate the "condition" in which the surgical system 2 is currently operating. In some embodiments, the light can be generated by LED lights that form a ring around the light indicator 28. The light indicator 28 can include a completely permeable material that allows light to pass through the entire light indicator 28. The light indicator 28 can be attached to the lower display support 30. As shown in FIG. Figure 2As shown, the lower display support 30 can allow the operator to manipulate the display 34 to any suitable position. The lower display support 30 can be attached to the light indicator 28 by any suitable mechanism. In some embodiments, the lower display support 30 can rotate about the light indicator 28 or be rigidly attached to the light indicator. The upper display support 32 can be attached to the lower display support 30 by any suitable mechanism.

[0057] In some embodiments, a tablet computer can be used in conjunction with and / or without the display 34. The tablet computer can be placed on the upper display support 32 in place of the display 34 and can be removable from the upper display support 32 during a medical procedure. In addition, the tablet computer can communicate with the display 34. The tablet computer can be connected to the surgical robot 4 via any suitable wireless and / or wired connection. In some embodiments, the tablet computer can program and / or control the surgical system 2 during a medical procedure. When the surgical system 2 is controlled by the tablet computer, all input and output commands can be replicated on the display 34. Using a tablet computer can allow the operator to manipulate the surgical robot 4 without having to move around the patient 50 and / or the surgical robot 4.

[0058] As will be explained below, in some embodiments, the surgeon and / or other personnel may wear an XR headset that can be used in conjunction with the display 34 and / or a tablet computer, or one or more XR headsets may eliminate the need to use the display 34 and / or a tablet computer.

[0059] like Figure 3A and 5 As shown, the camera tracking system assembly 6 works in combination with the surgical robot 4 via a wired or wireless communication network. Figure 1 , 3 and 5, the camera tracking system assembly 6 can include some components similar to those of the surgical robot 4. For example, the camera body 36 can provide the functions found in the robot body 8. The robot body 8 can provide an auxiliary tracking rod on which the camera 46 is mounted. The structure within the robot body 8 can also provide support for electronic equipment, communication devices and power supplies for operating the camera tracking system assembly 6. The camera body 36 can be made of the same material as the robot body 8. The camera tracking system assembly 6 can communicate directly with the XR headset, tablet computer and / or display 34 via a wireless and / or wired network to enable the XR headset, tablet computer and / or display 34 to control the functions of the camera tracking system assembly 6.

[0060] The camera body 36 is supported by a camera base 38. The camera base 38 may be used as the robot base 10. Figure 1In the embodiment of the present invention, the camera base 38 can be wider than the robot base 10. The width of the camera base 38 can allow the camera tracking system assembly 6 to be connected to the surgical robot 4. Figure 1 As shown, the width of the camera mount 38 can be large enough to fit outside of the robotic base 10. The additional width of the camera mount 38 can allow the surgical system 2 to provide additional maneuverability and support for the surgical system 2 when the camera tracking system assembly 6 is coupled to the surgical robot 4.

[0061] As with the robot base 10, a plurality of powered wheels 12 may be attached to the camera base 38. Similar to the operation of the robot base 10 and powered wheels 12, the powered wheels 12 may allow the camera tracking system assembly 6 to be stabilized and leveled or set to a fixed orientation relative to the patient 50. Such stabilization may prevent the camera tracking system assembly 6 from moving during a medical procedure and may prevent the camera 46 on the auxiliary tracking pole from losing track of a DRA connected to the XR headset and / or surgical robot 4 and / or losing track of a DRA connected to the XR headset and / or surgical robot 4. Figure 3A and 5 The tracking of one or more DRAs 52 of the anatomical structure 54 and / or tool 58 within the designated area 56 is shown. This stability and maintenance of tracking enhances the ability of the surgical robot 4 to operate effectively with the camera tracking system assembly 6. Additionally, the wide camera base 38 can provide additional support for the camera tracking system assembly 6. Specifically, as Figure 3A and 5 As shown, the wide camera base 38 can prevent the camera tracking system assembly 6 from tipping over when the camera 46 is positioned over the patient.

[0062] The camera telescopic support 40 can support a camera 46 on an auxiliary tracking pole. In some embodiments, the telescopic support 40 can move the camera 46 higher or lower in a vertical direction. The camera handle 48 can be attached to the camera telescopic support 40 at any suitable location and is configured to allow an operator to move the camera tracking system assembly 6 to a planned location prior to a medical procedure. In some embodiments, the camera handle 48 can be used to lower and raise the camera telescopic support 40. The camera handle 48 can perform raising and lowering of the camera telescopic support 40 by pressing a button, a switch, a lever, and / or any combination thereof.

[0063] The lower camera support arm 42 may be attached to the camera telescoping support 40 at any suitable location, in embodiments such as Figure 1As shown, the lower camera support arm 42 can rotate three hundred and sixty degrees around the telescopic support 40. This free rotation can allow the operator to position the camera 46 in any suitable position. The lower camera support arm 42 can be connected to the telescopic support 40 by any suitable mechanism. The lower camera support arm 42 can be used to provide support for the camera 46. The camera 46 can be attached to the lower camera support arm 42 by any suitable mechanism. The camera 46 can pivot in any direction at the attachment area between the camera 46 and the lower camera support arm 42. In an embodiment, the curved track 44 can be placed on the lower camera support arm 42.

[0064] The curved track 44 may be placed at any suitable location on the lower camera support arm 42. Figure 3A As shown, the curved track 44 can be attached to the lower camera support arm 42 by any suitable mechanism. The curved track 44 can be of any suitable shape, and suitable shapes can be crescent, circular, flat, oval, and / or any combination thereof. The camera 46 can be movably positioned along the curved track 44. The camera 46 can be attached to the curved track 44 by, for example, rollers, brackets, supports, motors, and / or any combination thereof. Motors and rollers not shown can be used to move the camera 46 along the curved track 44. As shown in FIG. Figure 3A As shown, during a medical procedure, if an object blocks the camera 46 from viewing one or more tracked DRAs, the motor can responsively move the camera 46 along the curved track 44. This motorized movement can allow the camera 46 to move to a new position that is no longer blocked by the object without moving the camera tracking system component 6. When the camera 46 is blocked from viewing one or more tracked DRAs, the camera tracking system component 6 can send a stop signal to the surgical robot 4, the XR headset, the display 34, and / or the tablet computer. The stop signal can prevent the SCARA 24 from moving until the camera 46 has reacquired the tracked DRA 52 and / or can warn the operator to wear the XR headset and / or watch the display 34 and / or the tablet computer. The SCARA 24 can be configured to respond to the receipt of the stop signal by stopping further movement of the base and / or the end effector coupling 22 until the camera tracking system can resume tracking of the DRA.

[0065] Figure 3B and 3C Shows that it can be Figure 1 The front view and isometric view of another camera tracking system assembly 6' that can be used together with the surgical system of the surgical robot or can be used independently of the surgical robot. For example, the camera tracking system assembly 6' can be used to provide navigated surgery without the use of robotic guidance. Figure 3B and3C The camera tracking system component 6' and Figure 3A One of the differences between the camera tracking system components 6 is Figure 3B and 3C The camera tracking system assembly 6' includes a housing for transporting a computer platform 910. The computer platform 910 can be configured to: perform camera tracking operations to track the DRA; perform guided surgical operations that provide surgical navigation information to a display device (e.g., an XR headset and / or other display device); and perform other computing operations disclosed herein. Thus, the computer platform 910 can include a navigation computer, such as Fig.14 One or more navigation computers of the navigation computer.

[0066] Figure 6 Depicted is a medical procedure Figure 5 A block diagram view of the components of a surgical system. Figure 6 , the tracking camera 46 on the auxiliary tracking pole has a navigation field of view 600 in which the reference array 602 attached to the patient, the reference array 604 attached to the surgical instrument, and the pose (e.g., position and orientation) of the robotic arm 20 are tracked. The tracking camera 46 may be a computer platform 910 that includes a computer platform configured to perform the operations described below. Figure 3B and Figure 3C The reference array 602 is part of the camera tracking system assembly 6'. The reference array achieves tracking by reflecting light in a known pattern, which is decoded by the tracking subsystem of the surgical robot 4 to determine the corresponding pose of the reference array. If the line of sight between the patient reference array 602 and the tracking camera 46 on the auxiliary tracking pole is blocked (e.g., by medical personnel, instruments, etc.), further navigation of the surgical instruments may not be possible, and responsive notifications can temporarily stop further movement of the robotic arm 20 and the surgical robot 4, display a warning on the display 34, and / or provide an audible warning to the medical staff. The display 34 is available for use by the surgeon 610 and assistant 612, but viewing requires turning the head away from the patient and changing the eye focus to different distances and positions. The navigation software can be controlled by a technician 614 based on voice commands from the surgeon.

[0067] Figure 7 It shows that when using the navigation function of the surgical system 2, the surgical robot 4 can Figure 5 and 6The display screens may include, but are not limited to, a patient radiograph with an overlaid graphical representation of an instrument model positioned in the display screen relative to the anatomy based on the developed surgical plan and / or based on the pose of the tracked reference array, various user-selectable menus for controlling different stages of the surgical procedure, and virtually projected dimensional parameters of the implant (e.g., length, width, and / or diameter).

[0068] For navigated surgery, various processing components (e.g., computer platform 910) and associated software described below are provided that enable preoperative planning of a surgical procedure (e.g., implant placement) and electronic delivery of the plan to a computer platform 910 for providing navigation information to one or more users during the planned surgical procedure.

[0069] For robotic navigation, various processing components (e.g., computer platform 910) and associated software described below are provided that enable preoperative planning of a surgical procedure (e.g., implant placement) and electronic transmission of the plan to the surgical robot 4. The surgical robot 4 uses the plan to guide the robotic arm 20 and the connected end effector 26 to provide target poses of the surgical tools relative to the patient's anatomy for the steps of the planned surgical procedure.

[0070] The following embodiments relate to the use of one or more XR headsets that can be worn by a surgeon 610, assistant 612 and / or other medical personnel to provide an improved user interface for receiving information from and / or providing control commands to a surgical robot, camera tracking system components 6 / 6' and / or other medical equipment in an operating room.

[0071] Figure 8 FIG. 4 is a block diagram showing some electrical components of the surgical robot 4 according to some embodiments of the present disclosure. Figure 8, a load cell (not shown) can be configured to track the force applied to the end effector coupling 22. In some embodiments, the load cell can communicate with multiple motors 850, 851, 852, 853 and / or 854. When the load cell senses a force, information about the amount of force applied can be distributed from the switch array and / or multiple switch arrays to the controller 846. The controller 846 can obtain the force information from the load cell and process it with a switching algorithm. The controller 846 uses the switching algorithm to control the motor driver 842. The motor driver 842 controls the operation of one or more motors 850, 851, 852, 853 and / or 854. The motor driver 842 can direct a specific motor to generate an equal amount of force, such as measured by the load cell through the motor. In some embodiments, as indicated by the controller 846, the force generated can come from multiple motors, such as 850-854. Additionally, the motor driver 842 can receive input from the controller 846. The controller 846 can receive information about the direction of the force sensed by the force sensor from the force sensor. The controller 846 can use a motion controller algorithm to process this information. The algorithm can be used to provide information to a specific motor driver 842. In order to replicate the direction of the force, the controller 846 can activate and / or deactivate certain motor drivers 842. The controller 846 can control one or more motors, such as one or more of 850-854, to induce the movement of the end effector 26 in the direction of the force sensed by the force sensor. This force-controlled movement can allow the operator to move the SCARA 24 and the end effector 26 effortlessly and / or with very little resistance. The movement of the end effector 26 can be performed to position the end effector 26 in any suitable posture (i.e., relative to the position and angle orientation of the defined three-dimensional (3D) orthogonal reference axis) for use by medical personnel.

[0072] Figure 5 As best shown, the activation assembly 60 can be in the form of a bracelet that wraps around the end effector coupling 22. The activation assembly 60 can be positioned on any portion of the SCARA 24, on any portion of the end effector coupling 22, can be worn by a medical professional (and communicate wirelessly), and / or any combination thereof. The activation assembly 60 can include a primary button and a secondary button.

[0073] Pressing the master button may allow the operator to move the SCARA 24 and the end effector coupling 22. According to one embodiment, once set in position, the SCARA 24 and the end effector coupling 22 may not move until the operator programs the surgical robot 4 to move the SCARA 24 and the end effector coupling 22, or uses the master button to move. In some instances, it may be necessary to press at least two non-adjacent master activation switches before the SCARA 24 and the end effector coupling 22 will respond to operator commands. Pressing at least two master activation switches may prevent accidental movement of the SCARA 24 and the end effector coupling 22 during a medical procedure.

[0074] Activated by the main button, the force sensor can measure the magnitude and / or direction of the force applied by the operator (i.e., medical personnel) on the end effector coupling 22. This information can be transmitted to one or more motors within the SCARA 24, such as one or more of 850-854, which can be used to move the SCARA 24 and the end effector coupling 22. Information about the magnitude and direction of the force measured by the force sensor can cause one or more motors (e.g., one or more of 850-854) to move the SCARA 24 and the end effector coupling 22 in the same direction as sensed by the force sensor. This force-controlled movement can allow the operator to easily move the SCARA 24 and the end effector coupling 22, and because the motor moves the SCARA 24 and the end effector coupling 22 while the operator moves the SCARA 24 and the end effector coupling 22, a lot of effort is not required.

[0075] In some instances, the operator can use the secondary button as a "select" device. During a medical operation, the surgical robot 4 can notify medical personnel of certain situations through one or more XR headsets 920, displays 34, and / or light indicators 28. One or more XR headsets 920 are each configured to display images on a perspective display screen to form an extended reality image overlaid on a real-world object that can be viewed through the perspective display screen. The surgical robot 4 can prompt medical personnel to select functions, modes, and / or evaluate the situation of the surgical system 2. Pressing the secondary button once can activate certain functions, modes, and / or confirm information transmitted to medical personnel through one or more XR headsets 920, displays 34, and / or light indicators 28. In addition, pressing the secondary button multiple times in rapid succession can activate additional functions, modes, and / or select information transmitted to medical personnel through one or more XR headsets 920, displays 34, and / or light indicators 28.

[0076] Further references Figure 8, the electrical components of the surgical robot 4 include a platform subsystem 802, a computer subsystem 820, a motion control subsystem 840, and a tracking subsystem 830. The platform subsystem 802 includes a battery 806, a power distribution module 804, a connector panel 808, and a charging station 810. The computer subsystem 820 includes a computer 822, a display 824, and a speaker 826. The motion control subsystem 840 includes a drive circuit 842, motors 850, 851, 852, 853, 854, stabilizers 855, 856, 857, 858, an end effector connector 844, and a controller 846. The tracking subsystem 830 includes a position sensor 832 and a camera converter 834. The surgical robot 4 may also include a removable foot pedal 880 and a removable tablet computer 890.

[0077] The input power is supplied to the surgical robot 4 through a power supply that can be provided to the power distribution module 804. The power distribution module 804 receives the input power and is configured to generate different power supply voltages provided to other modules, components and subsystems of the surgical robot 4. The power distribution module 804 can be configured to provide different voltage supplies to the connector panel 808, which can be provided to other components (such as a computer 822, a display 824, a speaker 826, a drive 842) to, for example, power the motors 850-854 and the end effector connector 844, and to the camera converter 834 and other components for the surgical robot 4. The power distribution module 804 can also be connected to a battery 806, which acts as a temporary power source when the power distribution module 804 does not receive power from the input power source. At other times, the power distribution module 804 can be used to charge the battery 806.

[0078] The connector panel 808 can be used to connect different devices and components to the surgical robot 4 and / or associated components and modules. The connector panel 808 may contain one or more ports for receiving lines or connectors from different components. For example, the connector panel 808 may have a grounding port for grounding the surgical robot 4 to other equipment, a port for connecting a foot pedal 880, and a port for connecting to a tracking subsystem 830, which may include a position sensor 832, a camera converter 834, and a DRA tracking camera 870. The connector panel 808 may also include other ports to allow USB, Ethernet, HDMI communication with other components (such as a computer 822). According to some embodiments, the connector panel 808 may include a wired and / or wireless interface for operably connecting one or more XR head-mounted devices 920 to the tracking subsystem 830 and / or the computer subsystem 820.

[0079] The control panel 816 may provide various buttons or indicators for controlling the operation of the surgical robot 4 and / or providing information from the surgical robot 4 for the operator to observe. For example, the control panel 816 may include buttons for turning the surgical robot 4 on or off, raising or lowering the vertical column 16, and raising or lowering the stabilizers 855-858, which may be designed to engage the casters 12 to lock the surgical robot 4 without physically moving. Other buttons may stop the surgical robot 4 in the event of an emergency, which may remove all motor power and apply mechanical brakes to stop all movement from occurring. The control panel 816 may also have indicators (such as a line power indicator) that notify the operator of certain system conditions or the charge status of the battery 806. According to some embodiments, one or more XR headsets 920 may communicate, for example, via a connector panel 808 to control the operation of the surgical robot 4 and / or receive and display information generated by the surgical robot 4 for observation by a person wearing the XR headset 920.

[0080] The computer 822 of the computer subsystem 820 contains an operating system and software for operating the designated functions of the surgical robot 4. The computer 822 can receive and process information from other components (e.g., the tracking subsystem 830, the platform subsystem 802, and / or the motion control subsystem 840) to display information to the operator. Further, the computer subsystem 820 can provide output to the operator through the speaker 826. The speaker can be part of the surgical robot, part of the XR headset 920, or in another component of the surgical system 2. The display 824 can correspond to Figure 1 and 2 A display 34 is shown.

[0081] Tracking subsystem 830 may include position sensor 832 and camera converter 834. Tracking subsystem 830 may correspond to Figure 3A The DRA tracking camera 870 operates in conjunction with the position sensor 832 to determine the pose of the DRA 52. This tracking may be performed in a manner consistent with the present disclosure including the use of infrared or visible light techniques that track the position of active or passive elements of the DRA 52, such as LEDs or reflective fiducials (also referred to as markers), respectively.

[0082] The functional operations of the tracking subsystem 830 and the computer subsystem 820 may be included in the Figure 3A and 3BThe camera tracking system component 6' is transmitted to the computer platform 910. The tracking subsystem 830 can be configured to determine a pose, such as the position and angular orientation of the tracked DRA. The computer platform 910 can also include a navigation controller configured to use the determined pose to provide navigation information to the user to guide them to move the tracked tool relative to the registered patient image and / or the tracked anatomical structure during the planned surgical procedure. The computer platform 910 can be Figure 3B and 3C The computer platform 910 can be configured to communicate with the computer subsystem 820 and the computer subsystem 820 when used with a surgical robot. Figure 8 The computer platform 910 may communicate with other subsystems of the computer platform 910 to control movement of the end effector 26. For example, as will be explained below, the computer platform 910 may generate a graphical representation of a patient's anatomy, surgical tools, a user's hand, etc., having a displayed size, shape, color, and / or pose controlled based on the determined pose of one or more tracked DRAs, and the displayed graphical representation may be dynamically modified to track changes in the determined pose over time.

[0083] The motion control subsystem 840 can be configured to physically move the vertical column 16, the upper arm 18, the lower arm 20, or to rotate the end effector coupling 22. The physical movement can be performed by using one or more motors 850-854. For example, the motor 850 can be configured to vertically raise or lower the vertical column 16. Figure 2 As shown, the motor 851 can be configured to move the upper arm 18 laterally about the point of engagement with the vertical column 16. Figure 2 As shown, motor 852 may be configured to move lower arm 20 laterally about a joint with upper arm 18. Motors 853 and 854 may be configured to move end effector coupling 22 to provide translational movement along and rotational movement about three-dimensional axes. Fig. 9 The illustrated computer platform 910 can provide control inputs to the controller 846 that direct the movement of the end effector coupling 22 to position the passive end effector connected thereto in a planned pose (i.e., position and angular orientation relative to defined 3D orthogonal reference axes) relative to the anatomical structure to be operated on during the planned surgical procedure. The motion control subsystem 840 can be configured to measure the position of the end effector coupling 22 and / or the end effector 26 using an integrated position sensor (e.g., an encoder).

[0084] Fig. 9 A block diagram of components of a surgical system according to some embodiments of the present disclosure is shown, the surgical system including a device connected to a camera tracking system component 6 ( Figure 3A ) or 6'( Figure 3B , 3C ) and / or an imaging device of the computer platform 910 of the surgical robot 4 (e.g., C-arm 104, O-arm 106, etc.). Alternatively, at least some of the operations performed by the computer platform 910 disclosed herein may additionally or alternatively be performed by components of the surgical system.

[0085] refer to Fig. 9 , the computer platform 910 includes a display 912, at least one processor circuit 914 (also referred to as a processor for brevity), at least one memory circuit 916 (also referred to as a memory for brevity) containing computer readable program code 918, and at least one network interface 902 (also referred to as a network interface for brevity). The display 912 may be part of an XR head-mounted device 920 according to some embodiments of the present disclosure. The network interface 902 may be configured to connect to Fig.10 The C-arm imaging device 104, Fig.11 The O-arm imaging device 106, another medical imaging device, an image database 950 containing medical images of patients, components of the surgical robot 4 and / or other electronic devices.

[0086] When used with the surgical robot 4, the display 912 may correspond to Figure 2 The display 34 and / or Figure 8 The tablet computer 890 and / or the XR head-mounted device 920 operably connected to the surgical robot 4, the network interface 902 may correspond to Figure 8 The platform network interface 812, and the processor 914 may correspond to Figure 8 The network interface 902 of the XR headset 920 can be configured to communicate according to one or more wireless communication protocols (e.g., WLAN, 3GPP 4G and / or 5G (New Radio) cellular communication standards, etc.) through a wired network (e.g., thin-wire Ethernet) and / or through a wireless RF transceiver link.

[0087] The processor 914 may include one or more data processing circuits, such as general and / or special purpose processors, such as microprocessors and / or digital signal processors. The processor 914 is configured to execute computer readable program code 918 in the memory 916 to perform operations, which may include some or all of the operations described herein as being performed for surgical planning, navigated surgery, and / or robotic surgery.

[0088] The computer platform 910 can be configured to provide surgical planning functionality. The processor 914 can be operated to display images of anatomical structures (e.g., vertebrae) received from one of the imaging devices 104 and 106 and / or from the image database 950 via the network interface 920 on the display device 912 and / or on the XR headset 920. The processor 914 receives an operator's definition of a location where a surgical procedure (e.g., screw placement) is to be performed on the anatomical structure shown in one or more images, such as by the operator touching the display 912 to select a location for the planned surgical procedure or using a mouse-based cursor to define the location for the planned surgical procedure. As will be explained in further detail below, when the image is displayed in the XR headset 920, the XR headset can be configured to sense gesture-based commands formed by the wearer and / or sense voice-based commands spoken by the wearer, which can be used to control selection between menu items and / or control how objects are displayed on the XR headset 920.

[0089] The computer platform 910 can be configured to be able to perform anatomical measurements that may be particularly useful for knee surgery, similar to the measurement of various angles to determine the center of the hip, the center of the angle, natural landmarks (e.g., the transepicondylar line, the white side line, the posterior condylar line, etc.). Some measurements can be automatic, while some other measurements can involve human input or assistance. The computer platform 910 can be configured to allow the operator to input the selection of the correct implant for the patient, including the selection of size and alignment. The computer platform 910 can be configured to perform automatic or semi-automatic (involving human input) segmentation (image processing) of CT images or other medical images. The patient's surgical plan can be stored in a cloud-based server that can correspond to a database 950 for retrieval by the surgical robot 4.

[0090] For example, during orthopedic surgery, a surgeon can select a site to be cut (e.g., posterior femur, proximal tibia, etc.) using a computer screen (e.g., a touch screen) or extended reality (XR) interaction (e.g., gesture-based commands and / or voice-based commands) through, for example, an XR headset 920. The computer platform 910 can generate navigation information that provides visual guidance to the surgeon to perform the surgical procedure. When used with the surgical robot 4, the computer platform 910 can provide guidance that allows the surgical robot 4 to automatically move the end effector 26 to a target pose so that the surgical tool is aligned with the target position to perform a surgical procedure on the anatomical structure.

[0091] In some embodiments, surgical system 900 can use two DRAs to track patient anatomical locations, such as a DRA connected to the patient's tibia and a DRA connected to the patient's femur. System 900 can use standard navigated instruments for registration and inspection (e.g., a pointer similar to the pointer used in the Globus Excelsius GPS system for spinal surgery).

[0092] A particularly challenging task in navigation surgery is how to plan the location of implants in the spine, knee, and other anatomical structures, where the surgeon struggles to perform the task on a computer screen, which is a 2D representation of the 3D anatomical structure. The system 900 can solve this problem by using an XR headset 920 to display three-dimensional (3D) computer-generated representations of anatomical structures and candidate implants. Under the guidance of the computer platform 910, the computer-generated representations are scaled and posed relative to each other on the display screen and can be manipulated by the surgeon while viewing it through the XR headset 920. The surgeon can, for example, use gesture-based commands and / or voice-based commands sensed by the XR headset 920 to manipulate the displayed computer-generated representations of anatomical structures, implants, surgical tools, etc.

[0093] For example, the surgeon can watch the displayed virtual handle on the virtual implant, and can manipulate (e.g., grasp and move) the virtual handle to move the virtual implant to the desired position and adjust the planned implant placement relative to the graphical representation of the anatomical structure. Afterwards, during surgery, the computer platform 910 can display navigation information that helps the surgeon to more accurately follow the surgical plan to insert the implant and / or perform another surgical procedure on the anatomical structure through the XR headset 920. When the surgical procedure involves bone removal, the progress of bone removal (e.g., cutting depth) can be displayed in real time through the XR headset 920. Other features that can be displayed by the XR headset 920 may include but are not limited to gaps or ligament balance along the range of joint motion, contact lines along the range of joint motion on the implant, ligament tension and / or laxity rendered by color or other graphics, etc.

[0094] In some embodiments, the computer platform 910 can allow planning for the use of standard surgical tools and / or implants, such as posterior-stabilized implants and cruciate-retaining implants, cemented and cementless implants, and revision systems for surgeries associated with, for example, total or partial knee and / or hip replacements and / or trauma.

[0095] The automated imaging system may be used in conjunction with the computer platform 910 to acquire pre-operative, intra-operative, post-operative, and / or real-time image data of anatomical structures. Fig.10 and 11An example automatic imaging system is shown. In some embodiments, the automatic imaging system is a C-arm 104 ( Fig.10 ) imaging device or 106( Fig.11 (Medtronic Navigation, Inc., with a location in Louisville, Colorado, USA, owns O ). It may be desirable to x-ray a patient from many different positions without the need for frequent manual repositioning of the patient, as may be required in an x-ray system. C-arm 104 x-ray diagnostic equipment can solve the problem of frequent manual repositioning and is well known in the medical field for surgery and other interventional procedures. Fig.10 As shown, the C-arm includes an elongated C-shaped member terminating in a "C" shaped opposite distal end 112. The C-shaped member is attached to an x-ray source 114 and an image receptor 116. The space within the C-arm 104 of the arm provides space for the physician to care for the patient substantially uninterrupted by the x-ray support structure.

[0096] The C-arm is mounted so that the arm is capable of rotational movement in two degrees of freedom (i.e., about two perpendicular axes in a spherical motion). The C-arm is slidably mounted to the x-ray support structure, which allows for orbital rotational movement of the C-arm about its center of curvature, which can allow for selective vertical and / or horizontal orientation of the x-ray source 114 and image receptor 116. The C-arm may also be rotatable laterally (i.e., in a perpendicular direction relative to the direction of orbital motion to enable selective adjustment of the positioning of the x-ray source 114 and image receptor 116 relative to the width and length of the patient). The spherical rotational aspect of the C-arm device allows the physician to x-ray the patient at an optimal angle determined relative to the specific anatomy being imaged.

[0097] Fig.11 O shown in 106 includes a gantry housing 124, which can enclose an image capture portion not shown. The image capture portion includes an x-ray source portion and / or an x-ray emitting portion and an x-ray receiving portion and / or an image receiving portion, which can be arranged about one hundred and eighty degrees apart from each other and mounted on a rotor (not shown) relative to the track of the image capture portion. The image capture portion can be operatively rotated three hundred and sixty degrees during image acquisition. The image capture portion can rotate about a center point and / or an axis, thereby allowing image data of the patient to be acquired from multiple directions or in multiple planes.

[0098] O with stand housing 124 The gantry housing 106 has a central opening for positioning around the object to be imaged, a radiation source that can be rotated around the interior of the gantry housing 124, and the radiation source can be adapted to project radiation from a plurality of different projection angles. The detector system is adapted to detect the radiation at each projection angle, thereby acquiring an image of the object from a plurality of projection planes in a quasi-simultaneous manner. The gantry can be attached to the support structure O in a cantilevered manner. A support structure, such as a wheeled mobile cart with wheels. The positioning unit preferably translates and / or tilts the gantry to a planned position and orientation under the control of a computerized motion control system. The gantry may include a source and a detector positioned relative to each other on the gantry. The source and the detector may be fixed to a motorized rotor that can rotate the source and the detector in combination with each other around the interior of the gantry. The source can be pulsed in multiple positions and orientations in a partial and / or full three hundred and sixty degree rotation to perform multi-planar imaging of a target object positioned inside the gantry. The gantry may further include a track and bearing system for guiding the rotor as it rotates, the track and bearing system can carry the source and the detector. Both and / or one of the surgical system 106 and the C-arm 104 may be used as an automated imaging system to scan the patient and send the information to the surgical system 2 .

[0099] The images captured by the imaging system may be displayed on the XR headset 920 and / or another display device of the computer platform 910, the surgical robot 4, and / or another component of the surgical system 900. The XR headset 920 may be connected to one or more of the imaging devices 104 and / or 106 and / or the image database 950, for example, through the computer platform 910, to display images therefrom. The user may provide control inputs through the XR headset 920, for example, gesture-based and / or voice-based commands to control the operation of one or more of the imaging devices 104 and / or 106 and / or the image database 950.

[0100] Fig.12 A block diagram of components of a surgical system including a pair of XR head-mounted devices 920 (head-mounted displays HMD1 and HMD2) that may correspond to Fig.13 The XR headset 920 shown in FIG. 1 and operates according to some embodiments of the present disclosure.

[0101] refer to Fig.12 In the example scenario of FIG. 6 , both assistant 612 and surgeon 610 wear XR headsets 920, respectively. Assistant 612 optionally wears XR headset 920. As will be further described below, XR headset 920 is configured to provide the wearer with an interactive environment through which information related to the surgical procedure can be viewed and interacted with. This interactive XR-based environment can eliminate the need for the wearer to be in the operating room. Figure 6 The need for technician 614 shown in FIG. 6A and the need for using Figure 6 Each XR headset 920 may include one or more cameras configured to provide an additional source for tracking DRAs or other reference arrays attached to surgical tools, patient anatomy, end effector 26, and / or other equipment. Fig.12 In the example, the XR headset 920 has a field of view (FOV) 1202 for tracking the DRA and other objects, the XR headset 920 has a FOV 1212 for tracking the DRA and other objects that partially overlaps with the FOV 1202, and the tracking camera 46 has another FOV 600 for tracking the DRA and other objects that partially overlaps with the FOVs 1202 and 1212.

[0102] If one or more cameras are blocked from viewing a DRA attached to a tracked object (e.g., a surgical tool), but the DRA is in the view of one or more other cameras, the tracking subsystem 830 and / or navigation controller 828 can continue to track the object seamlessly without losing navigation. Additionally, if a DRA is partially obscured from the view of one camera, but the entire DRA is visible through multiple camera sources, the tracking inputs of the cameras can be combined to continue navigating the DRA. One of the XR headsets and / or tracking cameras 46 can view and track a DRA on another XR headset to allow the computer platform 910 ( Fig. 9 and Fig.14 ), the tracking subsystem 830 and / or another computing component is capable of determining the pose of, for example, the XR headset 920, the DRA of the tracking camera 46 relative to one or more defined coordinate systems and / or another coordinate system defined for the patient, table and / or room.

[0103] The XR headset 920 may be operably connected to view video, pictures, and / or other received information and / or provide commands to control various equipment in the operating room, including but not limited to neuromonitoring, microscopes, cameras, and anesthesia systems. Data from the various equipment may be processed and displayed within the headset, such as displaying a patient's vital signs or a microscope feed.

[0104] Example XR headset components and integration with guided surgery, surgical robots, and other devices

[0105] Fig.13An XR headset 920 configured in accordance with some embodiments of the present disclosure is shown. The XR headset includes a headband 1306 configured to secure the XR headset to the head of a wearer, an electronics housing 1304 supported by the headband 1306, and a display screen 1302 extending laterally across and downwardly from the electronics housing 1304. The display screen 1302 may be a see-through LCD display device or a semi-reflective lens that reflects an image projected by the display device toward the wearer's eyes. A set of DRA reference points 1310 (e.g., dots) are painted or attached to one or both sides of the headset in a known manner at intervals. The DRA on the headset enables a tracking camera on an auxiliary tracking pole to track the pose of the headset 920 and / or enables another XR headset to track the pose of the headset 920.

[0106] Display screen 1302 operates as a perspective display screen (also referred to as a combiner) that reflects light from a display panel of a display device toward the eyes of a user. The display panel can be positioned between the electronic component housing and the user's head, and is angled to project virtual content toward display screen 1302 to reflect the eyes of the user. Display screen 1302 is semi-transparent and semi-reflective, allowing the user to see the reflected virtual content overlaid on the user's view of the real world scene. Display screen 1302 can have different opacity areas, such as the upper lateral band with a higher opacity than the lower lateral band as shown. The opacity of display screen 1302 can be electrically controlled to adjust the amount of light from the real world scene that passes through to reach the user's eyes. The high opacity configuration of display screen 1302 enables a high contrast virtual image to be overlaid on a dim view of the real world scene. The low opacity configuration of display screen 1302 can enable a more blurred virtual image to be overlaid on a clearer view of the real world scene. The opacity can be controlled by applying an opaque material on the surface of display screen 1302.

[0107] According to some embodiments, the surgical system includes an XR headset 920 and an XR headset controller, such as Fig.14 1430 in the controller. The XR headset 920 is configured to be worn by a user during a surgical procedure and has a see-through display screen 1302 that is configured to display an XR image and allow at least a portion of a real-world scene to pass through for viewing by the user. The XR headset 920 also includes an opacity filter that is positioned between at least one of the user's eyes and the real-world scene when the see-through display screen 1302 is viewed by the user. The opacity filter is configured to provide opacity to light from the real-world scene. The XR headset controller is configured to communicate with a navigation controller (e.g., Fig.14One or more controllers 828A, 828B and / or 828C) in the navigation controller communicate to receive navigation information from the navigation controller to provide guidance to the user during a surgical procedure on the anatomical structure, and is further configured to generate XR images based on the navigation information for display on the perspective display screen 1302.

[0108] The opacity of the display screen 1302 may be configured to have a gradient of opacity that changes more continuously with distance downward from the top portion of the display screen 1302. The darkest point of the gradient may be located at the top portion of the display screen 1302 and become progressively less transparent further down the display screen 1302 until the opacity becomes transparent or non-existent. In further example embodiments, the gradient may change from approximately 90% opacity to fully transparent at approximately mid-eye level of the display screen 1302. With the headset properly calibrated and positioned, the mid-eye level may correspond to the point at which the user would be looking straight ahead, and the end of the gradient would be located at the "horizontal" line of the eye. The darker portion of the gradient will allow for clear and unambiguous vision of the virtual content and help block the intrusive brightness of overhead operating room lights.

[0109] Using an opacity filter in this manner enables the XR headset 920 to provide virtual reality (VR) functionality along the upper portion of the display screen 1302 and AR functionality along the middle or lower portion of the display screen 1302 by substantially or completely blocking light from the real world scene. This allows the user to have semi-transparent AR where it is needed and allows for the use of clear optics for the patient's anatomy during procedures. Configuring the display screen 1302 as a gradient rather than a more constant opacity band can allow the wearer to experience a more natural transition between a more VR-type view and a more AR-type view without experiencing abrupt changes in the brightness of the real world scene and depth of field that can otherwise fatigue the eyes, such as during more rapid shifts between upper and lower views.

[0110] The display panel and display screen 1302 can be configured to provide a wide field of view perspective XR display system. In one exemplary configuration, they provide a user with an 80° diagonal field of view (FOV) and a 55° vertical coverage for viewing virtual content. Other diagonal FOV angles and vertical coverage angles can be provided by display panels of different sizes, lenses of different curvatures, and / or different distances and angular orientations between the display panel and the curved display screen 1302.

[0111] Fig.14The electrical components of an XR head-mounted device 920 according to some embodiments of the present disclosure are depicted, and the XR head-mounted device can be operably connected to a computer platform 910, one or more imaging devices (such as a C-arm imaging device 104, an O-arm imaging device 106) and / or an image database 950 and / or a surgical robot 800.

[0112] The XR headset 920 provides an improved human-machine interface for performing a navigated surgical procedure. The XR headset 920 may be configured, for example, to provide functionality, via the computer platform 910, including but not limited to any one or more of: identifying gesture-based commands and / or voice-based commands, displaying XR graphical objects on the display device 1450. The display device 1450 may be a video projector, a flat panel display, etc. that projects the displayed XR graphical objects onto the display screen 1302. The user may view the XR graphical objects as anchored to the surgical procedure through the display screen 1302 ( Fig.13 ) overlay of a particular real-world object being viewed. The XR headset 920 may additionally or alternatively be configured to display video feeds from cameras mounted to one or more of the XR headsets 920 and other cameras on the display screen 1450.

[0113] The electrical components of the XR headset 920 may include a plurality of cameras 1440, a microphone 1442, a gesture sensor 1444, a pose sensor (e.g., an inertial measurement unit (IMU)) 1446, a display module 1448 including a display device 1450, and a wireless / wired communication interface 1452. As described below, the cameras 1440 of the XR headset may be visible light capture cameras, near infrared capture cameras, or a combination of both.

[0114] The cameras 1440 may be configured to operate as gesture sensors 1444 by capturing user gestures performed within the field of view of one or more cameras 1440 for identification. Alternatively, the gesture sensors 1444 may be proximity sensors and / or touch sensors that sense gestures performed in proximity to the gesture sensors 1444 and / or sense physical contact (e.g., tapping the sensor or housing 1304). The pose sensor 1446 (e.g., an IMU) may include a multi-axis accelerometer, a tilt sensor, and / or another sensor that may sense rotation and / or acceleration of the XR headset 920 along one or more defined coordinate axes. Some or all of these electrical components may be contained in the component housing 1304 or may be contained in another housing that is configured to be worn elsewhere, such as on the hip or shoulder.

[0115] As explained above, the surgical system 2 includes a camera tracking system component 6 / 6' and a tracking subsystem 830, which can be part of the computer platform 910. The surgical system can include an imaging device (e.g., a C-arm 104, an O-arm 106, and / or an image database 950) and / or a surgical robot 4. The tracking subsystem 830 is configured to determine the pose of a DRA attached to an anatomical structure, an end effector, a surgical tool, etc. The navigation controller 828 is configured to determine the position of the DRA attached to the anatomical structure based on a surgical plan (e.g., based on a surgical plan defined by the surgical tool) where the surgical procedure is to be performed on the anatomical structure. Fig. 9 The navigation controller 828 may be configured to determine a target pose of the surgical tool relative to the anatomical structure based on the pose of the anatomical structure determined by the tracking subsystem 830. The navigation controller 828 may be further configured to generate steering information based on the target pose of the surgical tool, the pose of the anatomical structure, and the pose of the surgical tool and / or the end effector, wherein the steering information indicates where the surgical tool and / or the end effector of the surgical robot should be moved to execute the surgical plan. The various cameras 1440 of the XR headset 920 may be connected to the camera tracking system component 6 / 6' to track the pose of the DRA, the user's hand, etc.

[0116] The electrical components of the XR headset 920 may be operably connected to the electrical components of the computer platform 910 via the wired / wireless interface 1452. The electrical components of the XR headset 920 may be operably connected to various imaging devices (e.g., C-arm imaging device 104, I / O-arm imaging device 106, image database 950) and / or other medical devices, for example, via the computer platform 910 or directly connected via the wired / wireless interface 1452.

[0117] The surgical system 2 further includes at least one XR headset controller 1430 (also referred to as an "XR headset controller" for brevity), which may reside in the XR headset 920, the computer platform 910, and / or in another system component connected by a wired cable and / or a wireless communication link. The software executed by the XR headset controller 1430 provides various functions. The XR headset controller 1430 is configured to receive navigation information from the navigation controller 828 to provide guidance to the user during a surgical procedure performed on an anatomical structure, and is configured to generate an XR image based on the navigation information for display on the display device 1450 for projection on the perspective display screen 1302.

[0118] The configuration of the display device 1450 relative to the display screen (also referred to as a “see-through display screen”) 1302 is configured to display the XR images in a manner such that the XR images appear to be in the real world when viewed through the display screen 1302 by a user wearing the XR headset 920. The display screen 1302 may be positioned in front of the user's eyes by a headband 1306.

[0119] The XR headset controller 1430 may be within a housing configured to be worn on the user's head or other part of the user's body while viewing the display screen 1302, or may be positioned away from the user viewing the display screen 1302 while communicatively connected to the display screen 1302. The XR headset controller 1430 may be configured to operatively process signaling from the camera 1440, the microphone 142, and / or the pose sensor 1446, and to connect to display an XR image on the display device 1450 for viewing by the user on the display screen 1302. Thus, the XR headset controller 1430, illustrated as a circuit box within the XR headset 920, should be understood to be operatively connected to the other illustrated components of the XR headset 920, but need not reside in a common housing (e.g., Fig.13 For example, the AR headset controller 1430 may reside in the computer platform 910, which in turn may reside in the computer platform 910. Figure 3B and Figure 3C The computer tracking system 6' is shown inside the housing.

[0120] Example XR headset component optical layout

[0121] Fig.15 FIG. 9 is a block diagram showing an arrangement of optical components of an XR headset 920 according to some embodiments of the present disclosure. Fig.15 , the display device 1450 is configured to display the XR image generated by the XR headset controller 1430, and the light from the XR image is projected onto the display screen 1302 as the XR image 1500. The display screen 1302 is configured to combine the light of the XR image 1500 and the light from the real world scene 1502 into a combined augmented view 1504, which is directed to the user's eye 1510. The display screen 1302 configured in this manner acts as a see-through display screen. The XR headset 920 can include any number of tracking cameras 1440. The camera 1440 can be a visible light capture camera, a near infrared capture camera, or a combination of both.

[0122] Example user view through an XR headset

[0123] The XR headset operation can display both 2D images and 3D models on the display screen 1302. The 2D images can be preferably displayed in the less transparent band (upper band) of the display screen 1302, and the 3D models can be more preferably displayed in the more transparent band (otherwise referred to as the environment area) (bottom band) of the display screen 1302. Below the lower band where the display screen 1302 ends, the wearer can have an unobstructed view of the operating room. Note that the location where the XR content is displayed on the display screen 1302 can be fluid. The location where the 3D content is displayed may move to the opaque band depending on the positioning of the headset relative to the content, and the location where the 2D content is displayed can be placed in the transparent band and stabilized relative to the real world. Additionally, the entire display screen 1302 can be darkened under electronic control to convert the headset into virtual reality for surgical planning or completely transparent during medical procedures. As explained above, the XR headset 920 and associated operations not only support navigated procedures, but can also be performed in conjunction with robotic-assisted procedures.

[0124] Fig.16 An example view of a display screen 1302 of an XR headset 920 for providing navigation assistance to a user manipulating a surgical tool 1602 during a medical procedure is shown in accordance with some embodiments of the present disclosure. Fig.16 , when the surgical tool 1602 is brought into proximity with the tracked anatomical structure so that the dynamic reference arrays 1630 and 1632 connected to the surgical tool 1602 become in the camera 1440 ( Fig.15 ) and / or 46( Figure 6 ), a graphical representation of a tool 1600 may be displayed in 2D and / or 3D images relative to a graphical representation of an anatomical structure 1610. The user may use the viewed graphical representation to adjust a trajectory 1620 of a surgical tool 1602, which may be shown extending from the graphical representation of the tool 2000 through the graphical representation of the anatomical structure 1610. The XR headset 920 may also display textual information and other objects 1640. A dashed line 1650 extending across the viewed display screen represents an example division between upper and lower bands of different opacity levels.

[0125] Other types of XR images (virtual content) that may be displayed on display screen 1302 may include, but are not limited to, any one or more of the following:

[0126] I) 2D axial, sagittal and / or coronal views of the patient’s anatomy;

[0127] 2) an overlay of the planned tool and surgical implant positions over the currently tracked tool and surgical implant positions;

[0128] 3) Preoperative image library;

[0129] 4) Video feeds from microscopes and other similar systems or remote video conferencing;

[0130] 5) Options and configuration settings and buttons;

[0131] 6) A floating 3D model of the patient’s anatomy with surgical planning information;

[0132] 7) Real-time tracking of surgical instruments relative to the floating patient anatomy;

[0133] 8) Enhanced overlay of patient anatomy with instructions and guidance; and

[0134] 9) Enhanced overlay of surgical equipment.

[0135] Example configuration of cameras for tracking system components

[0136] Fig.17 An exemplary configuration of an auxiliary tracking rod 46 with two pairs of stereo tracking cameras configured according to some embodiments of the present disclosure is depicted; the auxiliary tracking rod 46 is Figure 3A , Figure 3B and Figure 3C The stereo tracking camera is part of a camera tracking system assembly. According to one embodiment, the stereo tracking camera includes a pair of stereo spaced visible light capture cameras and another pair of stereo spaced near infrared capture cameras. Alternatively, only a pair of visible light capture cameras or only a pair of near infrared capture cameras may be used in the auxiliary tracking rod 46. Any number of near infrared and / or visible light cameras may be used.

[0137] Pose measurement chain

[0138] As explained above, navigated surgery can include computer vision tracking and determining the pose (e.g., position and orientation in a six-degree-of-freedom coordinate system) of a surgical instrument, such as by determining the pose of an attached DRA that includes spaced-apart fiducials, such as disks or balls, arranged in a known manner to a camera tracking system. The computer vision uses spaced-apart tracking cameras, such as stereo cameras, that are configured to capture near-infrared and / or visible light. In this context, there are three parameters competing for optimization: (1) accuracy, (2) stability, and (3) user ergonomics during the surgical procedure.

[0139] Computer operations can combine (link) the measured poses in a way that can improve optimization of one or more of the three parameters mentioned above by incorporating additional tracking cameras mounted to one or more XR headsets. Fig.17As shown, according to some embodiments of the present disclosure, a pair of stereo visible light tracking cameras and another pair of stereo near infrared tracking cameras can be attached to an auxiliary tracking pole of a camera tracking system assembly. An operating algorithm is disclosed that analyzes the pose of a fully or partially observed DRA (e.g., when not all fiducials of the DRA are observed by a pair of stereo cameras) and combines the observed poses or partial poses in a manner that can improve accuracy, robustness, and / or ergonomics during navigation surgery.

[0140] As explained above, the XR headset can be configured to enhance the real world scene with a computer-generated XR image. The XR headset can be configured to provide an XR viewing environment by displaying a computer-generated XR image on a see-through display screen that allows light from the real world scene to pass through for combined viewing by the user. Alternatively, the XR headset can be configured to provide a VR viewing environment by preventing or substantially preventing light from the real world scene from being directly viewed by the user along the viewing path of the displayed XR image. The XR headset can be configured to provide both an AR viewing environment and a VR viewing environment. In one embodiment, both the AR viewing environment and the VR viewing environment are provided by lateral bands with significantly different opacities arranged between the see-through display screen and the real world scene, so that a VR viewing environment is provided for an XR image aligned with a high opacity band and an AR viewing environment is provided for an XR image aligned with a low opacity band. In another embodiment, both the AR viewing environment and the VR viewing environment are provided by a computer-adjustable control of an opacity filter that variably limits the amount of light from the real world scene passing through the see-through display screen to be combined with the XR image viewed by the user. Therefore, an XR headset may also be referred to as an AR headset or a VR headset.

[0141] As explained above, the XR headset may include a near infrared tracking camera and / or a visible light tracking camera configured to track fiducials of a DRA connected to a surgical instrument, a patient anatomy, one or more other XR headsets, and / or a robotic end effector. The use of near infrared tracking and / or visible light tracking on the XR headset provides additional tracking volume coverage beyond what a camera on a single auxiliary tracking pole can provide. Adding a near infrared tracking camera to an existing auxiliary tracking pole allows for more robust but less accurate tracking of the headset position than in visible light. Using stereo matching to co-identify the pose of the DRA fiducials between the visible tracking coordinate system and the near infrared tracking coordinate system enables mechanical calibration of the visible tracking coordinate system and the near infrared tracking coordinate system so that the coordinate systems can be fully aligned to perform a 3D DRA fiducial triangulation operation. Using both visible and near-infrared tracking coordinate systems can achieve any one or more of the following: (a) identifying tools that cannot be identified using a single coordinate system; (b) improving pose tracking accuracy; (c) enabling a wider range of motion without losing track of the surgical instrument, patient anatomy, and / or robotic end effector; and (d) naturally tracking the XR headset in the same coordinate system as the navigated surgical instrument.

[0142] Fig.18 A block diagram view of components of a surgical system is shown, the surgical system including tracking cameras in a pair of XR headsets 920 (head mounted displays HMD1 and HMD2) and tracking cameras in a camera tracking pole in a camera tracking system assembly 6' housing a computer platform 910. The computer platform 910 may include a tracking subsystem 830, a navigation controller 828, and an XR headset controller 1430, as previously described in Fig.14 As shown in .

[0143] refer to Fig.18 In the surgical system, the surgeon and the assistant each wear an XR head-mounted device HMD1 920 and HMD2 920 respectively. If each XR head-mounted device includes Fig.13 Tracking cameras in the shown configuration. The assistant optionally wears an XR headset HMD2 920.

[0144] The combination of the XR head mounted devices HMD1 920 and HMD2 920 and the tracking camera 46 on the auxiliary tracking pole can more robustly track the patient reference array (R), the robot end effector (E), and the surgical tool (T) or instrument instance objects when operating with the computer platform 910. The overlapping views from different perspectives provided by the XR head mounted devices HMD1 920 and HMD2 920 and the tracking camera 46 on the auxiliary tracking pole are shown in FIG. Fig.12 Displayed in.

[0145] Fig.18 Each of the labeled items in represents a unique coordinate system. The coordinate system labels are explained as follows:

[0146] A = visible light coordinate system of the second head mounted device HMD2 920;

[0147] N3 = the near infrared (NIR) coordinate system of the second head mounted device HMD2 920;

[0148] S = visible light coordinate system of the main head mounted device HMD1 920;

[0149] N2 = NIR coordinate system of the primary head mounted device HMD1 920;

[0150] N = NIR coordinate system of the auxiliary navigation rod 46;

[0151] V = the visible light coordinate system of the auxiliary navigation rod 46;

[0152] R = NIR coordinate system of patient reference fiducial array 602;

[0153] T = NIR coordinate system of the tracked tool 604;

[0154] E = the NIR coordinate system of the tracked robot end effector on the robot arm 20; and

[0155] W = Inertial navigation world coordinate system with stable gravity vector.

[0156] During manufacturing, when the device is installed in an operating room, and / or before a surgical procedure is to be performed, the spatial relationships (and by extension, the coordinate systems) of some of these marker objects may be measured and calibrated. In the disclosed system, the following coordinate systems are calibrated: where the term "T" is defined as a six-degree-of-freedom (6DOF) homogeneous transformation between two specified coordinate systems. Thus, for example, the term It is a 6DOF homogeneous transformation between the visible light coordinate system of the primary head mounted device HMD1 920 and the NIR coordinate system of the primary head mounted device HMD1 920 .

[0157] In one embodiment, XR headsets HMD1 920 and HMD2 920 have passive visible light reference points painted or otherwise attached to them (coordinate systems S and A), such as Fig.13 Reference array fiducials 1310 are shown in FIG. The tracking camera is spatially calibrated to these passive fiducials (coordinate systems N2 and N3).

[0158] As described above, the cameras on the XR headsets HMD1 920 and HMD2 920 and the tracking camera 46 on the auxiliary tracking pole have partially overlapping fields of view. If one or more cameras on the XR headset HMD1 920 are blocked from viewing a DRA attached to a tracked object, such as a tracked tool (T), but the DRA is within the field of view of the camera of the other XR headset HMD2 920 and / or the tracking camera 46 on the auxiliary tracking pole, the computer platform 910 can continue to track the DRA seamlessly without losing navigation. In addition, if the DRA is partially obscured from the perspective of the camera on the XR headset HMD1 920, but the entire DRA is visible through the camera of the other XR headset HMD2 920 and / or the tracking camera 46 on the auxiliary tracking pole, the tracking inputs of the cameras can be merged to continue navigating the DRA.

[0159] More specifically, various coordinate systems can be linked together by independently observing the various camera systems provided by the XR headsets HMD1 920 and HMD2 920 and the tracking camera 46 on the auxiliary tracking pole. For example, each of the XR headsets HMD1 920 and HMD2 920 may require virtual augmentation of the robot end effector (E). Although one XR headset HMD1 920 (N2) and the tracking camera 46 on the auxiliary tracking pole (N) can see (E), it may not be seen by another XR headset HMD2 920 (N3). The position of (E) relative to (N3) can still be calculated by one of several different operation methods. The operation according to one embodiment performs the linking of the pose from the patient reference (R). If the patient reference (R) is seen by (N3) and any one of (N) or (N2), the pose of (E) relative to (N3) can be directly solved by any of the following two equations:

[0160] -or-

[0161] The key to such pose chains is to infer the relationship between the frames at the end of each chain (circled below and transmitted). The chain can be arbitrarily long and is implemented by having more than one stereo camera system (e.g., N, N2, N3).

[0162] The camera tracking system can be configured to receive tracking information related to a tracked object from a first tracking camera (e.g., N3) and a second tracking camera (e.g., N2) during a surgical procedure. The camera tracking system can determine a first pose transformation between a coordinate system of the first object (e.g., R) and a coordinate system of the first tracking camera (e.g., N3) based on first object tracking information indicating a pose of the first object (e.g., R) from the first tracking camera (e.g., N3). For example, The camera tracking system may determine a second pose transformation (e.g., a first pose transformation (e.g., a first pose transformation (e.g., a first pose transformation (e.g., a first pose transformation (e.g., a first pose transformation (e.g., a first pose transformation (e.g., a first pose transformation (e.g., a first pose transformation (e.g., a first pose transformation (e.g., a second ... The camera tracking system may determine a third pose transformation (e.g., a first pose transformation) between a coordinate system of the second object (e.g., E) and a coordinate system of the second tracking camera (e.g., N2) based on second object tracking information indicating a pose of the second object (e.g., E) from a second tracking camera (e.g., N2). )). The camera tracking system can determine a fourth pose transformation between the second object (e.g., E) coordinate system and the first tracking camera (e.g., N3) coordinate system based on combining the first, second, and third pose transformations. For example, ).

[0163] In some other embodiments, the camera system may further determine the pose of the second object (eg, E) with respect to the first tracking camera system (eg, N3) coordinate system based on processing the tracking information through a fourth pose transformation.

[0164] Due to the overlapping fields of view of the various camera systems, when the first camera is blocked from seeing the second object (e.g., E), the camera tracking system is able to determine the pose of the second object (e.g., E) relative to the first tracking camera (e.g., N3). For example, in some embodiments, the camera tracking system is further configured to determine a fourth pose transformation (e.g., N4) between the coordinate system of the second object (e.g., E) and the coordinate system of the first tracking camera (e.g., N3). ) without using any tracking information from the first tracking camera (e.g., N3) indicating the pose of the second object (e.g., E).

[0165] The camera tracking system can achieve higher tracking accuracy by merging synchronized images from multiple camera systems. For example, the camera tracking system can determine the pose of the second object (e.g., E) relative to the first tracking camera (e.g., N3) by merging synchronized images of the second object (e.g., E) from multiple perspectives (first and second tracking cameras), and can use weighting that can be determined based on the accuracy specifications of the corresponding cameras. More specifically, the camera tracking system can be further configured to determine a fourth pose transformation (e.g., N3) between the second object (e.g., E) coordinate system and the first tracking camera (e.g., N3) coordinate system based on second object tracking information indicating the pose of the second object (e.g., E) from the first tracking camera (e.g., N3) and further based on the combined results of the first, second and third pose transformations. ).

[0166] The surgical system may be configured to display an XR image having a posture determined based on the fourth posture transformation on the perspective display screen of the XR head-mounted device. The camera tracking system may be further configured to generate an XR image as a graphical representation of a second object (e.g., E) placed on the perspective display screen based on processing the first object tracking information from the first and second tracking cameras and the second object tracking information from the second tracking camera for the fourth posture transformation.

[0167] As described above, the camera tracking system can include a navigation controller 828 that is communicatively connected to the first tracking camera (e.g., N3) and the second tracking camera (e.g., N2) to receive tracking information and is configured to determine first, second, third, and fourth pose transforms.

[0168] Registering surgical tool features to a reference array identified by a camera tracking system

[0169] During the surgical procedure, the camera tracking system can simultaneously track the pose of a surgical tool held or supported within the field of view of a set of tracking cameras, and can continue to track the surgical tool as it moves from outside to inside (e.g., after being picked up again). Many surgical tools require software configuration for proper tracking. Because the camera tracking system tracks the pose of reference arrays attached to or on the surgical tool, the camera tracking system should be informed which surgical tool features are registered to which tracked reference arrays. Navigation controller 828 ( Fig.14) can thus operate with knowledge of specific characteristics of the surgical tool. For example, registration of the surgical tool characteristic of the distance from the identified reference array to the tip of the surgical tool enables the navigation controller 828 to guide the surgeon's movement of the tool tip during the surgical procedure. Similarly, registration of the curvature direction of the surgical tool relative to the identified reference array enables the navigation controller 828 to display an accurate graphical representation of the surgical tool through the XR headset 920 and accurately position it relative to the tracked anatomical structure during the surgical procedure.

[0170] According to some embodiments, the registration process is also referred to as a pairing process, during which a user fixes a surgical tool with a reference array in the field of view of a set of tracking cameras used to identify the reference array, and the user then defines the characteristics of the surgical tool. The registration process is repeated for each combination of reference array and surgical tool characteristics that will be tracked during the surgical procedure, and the registration process can be further repeated when the reference array is detached from one type of surgical tool and attached to a different type of surgical tool. It is critical to enable the surgeon or other medical personnel wearing the XR headset to perform the registration process on a set of surgical tools in a timely and efficient manner to assist the surgeon in avoiding errors in registering the surgical tool characteristics with the reference array to reduce interruptions to the surgeon's attention before and during the surgical procedure.

[0171] Some other embodiments of the present disclosure relate to using an XR headset in a registration process to register an identified reference array to features of a surgical tool and displaying a representation of those features through the XR headset so that a user can verify the correctness of the registration. Using an XR headset in a registration process can provide surgeons and other medical personnel (users) with a more intuitive, time-saving, and reliable process for registering surgical tools with a camera tracking system before and / or during a surgical procedure.

[0172] The registration process may be automatically initiated in response to a reference array entering a field of view of a set of tracking cameras, and the camera tracking system determines that the identified reference array has not yet been registered to pair with the characteristics of the surgical tool. Fig.19 A graphical representation 1910 of surgical tool characteristics is shown that may be displayed by an XR headset when a reference array 1902 is being registered to the characteristics of the surgical tool 1900 for use in computer-assisted navigation of the surgical tool 1900 during surgery in accordance with some embodiments of the present disclosure. Fig. 20 A camera tracking system (e.g., Fig.18 6) is a flowchart of the registration operation performed by the system in FIG. 6, and the registration operation can provide Fig.19 Graphical representation of surgical tool characteristics in .

[0173] refer to Fig.19 and 20 , an example XR head mounted device includes a set of tracking cameras, which may be included, for example, in Fig.13 The electronic component housing 1304 of the XR head mounted device 920 shown in FIG. Fig.18 The system 6) identifies 2000 a reference array 1902 that is tracked by a set of tracking cameras, for example by identifying the reference array in a video stream from the tracking cameras. As described above, the reference array can be uniquely identified based on different orientations between a set of reference points that form the corresponding reference array. The camera tracking system determines 2002 whether the reference array 1902 identified is aligned to be paired with a characteristic of one of a plurality of surgical tools defined in a surgical tool database. Based on the reference array 1902 being determined 2002 to be unaligned and based on receiving user input, the camera tracking system aligns the reference array 1902 to be paired 2004 with a characteristic of a surgical tool 1900 selected from a plurality of surgical tools based on the user input. The camera tracking system then provides 2006 a representation of the characteristic to a display device of the XR head mounted device 920 for display to the user.

[0174] Therefore, in Fig.19 In the example illustration of , a user may begin registration by raising a surgical tool 1900 into the field of view of a set of tracking cameras attached to an XR headset 920. When an unregistered reference array enters the field of view of the tracking cameras, the camera tracking system may cause a visual prompt to be displayed through the XR headset 920, prompting the user to register the reference array to the characteristics of the associated surgical tool. The camera tracking system receives video from the tracking camera, identifies 2000 the reference array based on spacing and relativity, and determines 2002 that the identified reference array has not been registered to be paired with any defined characteristics of the surgical tool defined in a surgical tool database. The camera tracking system receives user input identifying the characteristics of the surgical tool and registers 2004 the reference array to be paired with the characteristics of the surgical tool 1900. Camera tracking system.

[0175] Example surgical tool characteristics may include, but are not limited to, structural and / or operational characteristics of drills, saws, retractors, and implants such as screws, spacers, intervertebral fusion devices, plates, rods, and the like.

[0176] By providing a list of surgical tools in the surgical tool database for display through the XR headset 920 for the user to select to align with the reference array 1902, the camera tracking system can facilitate the user's definition of surgical tool characteristics to be aligned with the identified reference array 1902. For example, based on the reference array 1902 being determined to be unaligned, the camera tracking system can provide a list of at least some of the plurality of surgical tools defined in the surgical tool database to the display device of the XR headset 920 for the user to select one of the surgical tools to align with the reference array 1902.

[0177] Some surgical tools are asymmetric and require additional configuration of features during registration. Fig.19 In a specific example of the present invention, the tip of the surgical tool 1900 is bent to one side, and the user is required to select one of four modes displayed as a list 1910 during registration based on the direction in which the tool tip is bent relative to the reference array 1902. The four modes include mode A corresponding to +X curvature, mode B corresponding to +Z curvature, mode C corresponding to -X curvature, and mode D corresponding to -Z curvature. Using the XR headset 920 during such surgical tool registration can intuitively help the user to register the surgical tool features and provide safeguards to ensure that the correct features are registered to the correct reference array 1902.

[0178] The camera tracking system can be configured to track the movement of the user's hand during the tool registration process and identify gestures as user input. Fig.19 In an example of , the user may be able to virtually touch select one of the modes (modes AD) displayed in the virtual space so that the surgical tool characteristics defined for the selected mode become registered to the reference array 1902. More specifically, the camera tracking system may be configured to determine the user's selection in the displayed list 1910 based on tracking information from a set of tracking cameras indicating the position of the user's hand in the XR space relative to the displayed list 1910.

[0179] The camera tracking system can display the characteristics and other information so that when the surgeon views through the XR headset 920, the characteristics and other information appear near the surgical tool 1900, such as shown in the list 1910, which is graphically displayed near the surgical tool 1900 when the surgeon views through the XR headset 920. In one embodiment, the camera tracking system receives tracking information indicating the pose of the reference array 1902 relative to the XR headset 920 from a set of tracking cameras, and determines a pose for how to display a representation of the characteristic (e.g., tool information) relative to the reference array based on the tracking information. The camera tracking system then controls the XR headset 920 to display the representation of the characteristic at the determined pose relative to the reference array 1902.

[0180] Once the surgical tool 1900 has been registered to the reference array 1902, the camera tracking system can display visual feedback of the registration characteristics to enable the surgeon to verify the accuracy of the registration. Displaying information descriptions about the registered surgical tool 1900 provides a level of verification to ensure that the correct surgical tool has been registered. Another level of verification can include displaying a 2D or 3D graphical representation of the characteristics of the surgical tool that has been or is being registered to the reference array 1902.

[0181] Fig.21 A graphical representation 2112 of the registered, actively used surgical tool 1900 displayed by the XR headset 920 is shown in accordance with some embodiments of the present disclosure. Fig. 22 The camera tracking system is shown to provide a Fig.21 A flowchart of the operation of a graphical representation 2112 of an aligned surgical tool 1900 in active use.

[0182] refer to Fig.21 and Fig. 22 , the camera tracking system obtains 2200 a graphical representation 2112 of the surgical tool 1900 in at least 2 dimensions (i.e., a 2D model or a 3D model), and receives 2202 tracking information from a set of tracking cameras indicating a pose of the reference array 1902 relative to the XR head mounted device 920. The camera tracking system determines 2204 a pose for the graphical representation 2112 based on the tracking information, and provides 2206 the graphical representation 2112 as a representation of the feature to a display device of the XR head mounted device 920 for display at the determined pose.

[0183] exist Fig.21In the example shown, the camera tracking system has obtained a 3D representation 2112 of the registered surgical tool 1900 and displays the 3D representation 2112 in a feedback window 2110 adjacent to the surgical tool 1900 and having a pose that matches the current pose of the surgical tool 1900. This shows the surgeon holding the surgical tool 1900 in front of the XR headset 920 that a particular feature of the surgical tool has been registered to the reference array 1902, which enables the surgeon to visually compare the displayed 3D representation 2112 with the actual surgical tool 1900 to verify the correctness of the registration. These operations can, for example, enable the surgeon to confirm that the tool tip bending direction shown in the graphical representation 2112 is in the list 1910 ( Fig.19 ) in the dialog box.

[0184] When a set of tracking cameras identifies a reference array connected to a surgical tool, the camera tracking system can determine how accurately the tracking cameras are tracking the reference array, a measure of tracking accuracy (also referred to as tracking quality). Tracking accuracy can be reduced when the tracking cameras are not correctly positioned relative to the surgical tool 2100, when the position of one or more fiducials has moved through deformation of the fiducial support structure, and / or when the fiducials are damaged and / or covered with bodily fluids or other materials. Therefore, it is important to enable the surgeon to observe the accuracy of the tracking while viewing the surgical tool 2100 through the XR headset 920.

[0185] Fig.23 A graphical representation 2312 of tracking accuracy information of a reference array 2102 for registration displayed by an XR headset 920 while a user is viewing a surgical tool 2100 is shown in accordance with some embodiments of the present disclosure. Fig.24 The camera tracking system is shown to provide a Fig.23 A flowchart of the operation of graphically representing 2312 the tracking accuracy information.

[0186] refer to Fig.23 and 24 , the camera tracking system obtains 2400 tracking accuracy information characterizing the accuracy of a set of tracking cameras currently tracking the pose of the reference array 2102. The camera tracking system then provides 2402 an indication 2312 of the tracking accuracy information to a display device of the XR headset for display to the user. Fig.23In the illustrative example of , the camera tracking system has displayed tracking accuracy in the form of a graphical sliding scale within a tracking quality window 2310 viewable through the XR headset 920, but other graphical indicators, text descriptions (e.g., poor tracking, good tracking, perfect tracking), and / or colored graphical markers may be displayed.

[0187] Tracking accuracy may be determined based on how closely the measured positions of the fiducials forming the reference array 2102 match the positions of fiducials that have been previously defined for the reference array 2102 (e.g., a model defined for the identified reference array 2101) and / or how well the shapes of the fiducials identified in the video from the tracking camera fit the defined shape of the fiducials (e.g., a defined elliptical shape). Alternatively or in addition, tracking accuracy may be determined based on comparing the pose of the reference array 2102 to an expected pose when a defined position (e.g., a tip) of a surgical tool touches a known position (e.g., a calibration point of another registered reference array). A graphical representation of tracking accuracy is generated to visually indicate the certainty or accuracy with which the tracking camera is tracking the reference array 2102.

[0188] The camera tracking system may be configured to display other information associated with the surgical tool 2100. In order to ensure that the surgical tool is able to operate properly during the surgical procedure, such as by not having structural bending or not having an appropriate range of extension and / or angular movement, it can be checked regularly. For example, it can be determined whether the surgical tool has a bent tip by positioning the tool tip at a known location (e.g., a known calibration point) to the camera tracking system to confirm that the reference array tracked when the tool tip is at a known location has the expected pose. The results of this verification process can be stored in the surgical database and displayed near the surgical tool to ensure that the user is informed whether the surgical tool should not be used during the surgical procedure and / or whether enough time has passed since the last verification so that the surgical tool should be re-verified before use. The camera tracking system may be configured to obtain a last verification date indicating the time when the surgical tool 2100 was last verified to have no structural deformation, and provide an indication of the last verification date to the display device of the XR head-mounted device 920 for display to the user.

[0189] Such graphical features of a surgical tool may be selectively displayed depending on whether the surgical tool is within a detection area defined relative to the position of the XR headset 920. For example, when a surgeon secures a surgical tool so that it is above the surgeon's chest or neck within a defined detection area, the camera tracking system may respond by displaying registration features of the surgical tool. In contrast, when the surgeon secures the surgical tool below the detection area, the camera tracking system may prevent display of the registration features of the surgical tool. This enables the surgeon to quickly check the display features that have been registered to a particular surgical tool by securing it within the detection area, and then have those display features disappear from view as the surgical tool moves outside the detection area, thereby, for example, avoiding interference with the surgeon's view of the surgical site during use of the tool during surgery.

[0190] The camera tracking system may compare the shape of a portion of the tool (e.g., the shape of the shaft of the tool 2100) to a defined template shape of the tool 2100. A difference in the compared shapes exceeding a defined threshold may result in a notification being displayed via the XR headset 920. For example, a graphical highlight may be displayed overlaid with a portion of the shaft of the tool 2100 that exhibits a deviation from the defined template shape in order to visually notify the user of a potential problem that may require replacement of the shaft and / or tool 2100.

[0191] Fig.26 A flowchart illustrating operations performed by a camera tracking system to selectively display characteristics of a surgical tool registered with a reference array based on whether the reference array is within a detection area defined relative to an XR headset in accordance with some embodiments of the present disclosure is shown.

[0192] refer to Fig.26 , the camera tracking system is configured to determine 2600 when the reference array enters a detection area defined relative to the position of the XR headset. In response to determining that the reference array has entered the detection area, the camera tracking system begins 2602 providing a representation of the characteristic to a display device of the XR headset for display to the user. In contrast, in response to determining that the reference array has left the detection area, the camera tracking system stops 2604 any ongoing provision of a representation of the characteristic to the display device of the XR headset for display to the user.

[0193] A detection region may similarly be defined to selectively enable a registration process to be performed on a surgical tool. Fig.25 A flow diagram illustrating operations performed by a camera tracking system to selectively allow a user to register a reference array based on whether the reference array is within a detection area defined relative to an XR headset in accordance with some embodiments of the present disclosure.

[0194] refer to Fig.25, the camera tracking system is configured to determine 2500 when the reference array enters a detection area defined relative to the position of the XR headset. In response to determining that the reference array has entered the detection area, the camera tracking system begins 2502 determining whether the reference array is aligned to be paired with a characteristic of one of a plurality of surgical tools defined in a surgical tool database. In contrast, the camera tracking system is configured to limit 2504 receiving user input to align the reference array to be paired with a characteristic of one of a plurality of surgical tools defined in the surgical tool database to only when the reference array remains within the detection area.

[0195] During a surgical procedure, one or more fiducials arranged to form a reference array may become obscured or otherwise not viewed by a set of tracking cameras because, for example, they have been damaged and / or become covered with bodily fluids or other materials during the surgical procedure. Some embodiments relate to displaying information showing which fiducials in the reference array are tracked by the tracking cameras. For example, when tracking fails due to obstruction of one or more fiducials, the remaining fiducials may be reported as stray. In this case, the camera tracking system may identify the pose (e.g., position) of some fiducials, but there is not enough information to match the identified fiducials with one of a plurality of predefined reference arrays. However, the pose of the stray fiducials may still be used to inform the surgeon about which fiducials are being tracked. A virtual representation of each tracked fiducial may be displayed by an XR headset in its pose (e.g., position) in the real world as seen by the XR headset. In this way, the surgeon will know which fiducials are obscured or need to be adjusted or replaced.

[0196] refer to Fig. 27 In an example embodiment, according to some embodiments of the present disclosure, a surgical tool 2100 having five reference points forming a reference array 2102 is viewed through an XR headset 920 having graphic marks 2700, 2702, 2704, 2706 that are shown as at least partially overlapping with the identified poses of four reference points of the reference array 2102, and another graphic mark 2710 that is shown as at least partially overlapping with the estimated pose of the missing reference points.

[0197] Fig.28 The method performed by the camera tracking system to display, for example, Fig. 27 Flowchart of the operations shown in FIG. 1 with respect to the graphical marking of identified and missing reference points.

[0198] refer to Fig. 27 and 28, the camera tracking system is further configured to identify 2800 a pose of a plurality of fiducials in the reference array 2102. For each of the plurality of fiducials in the reference array 2102 having the identified pose, the camera tracking system provides 2802 a graphical marker (e.g., 2700, 2702, 2704, and 2706) for display by the display of the XR headset 920 as at least partially overlapping the identified pose of the fiducial. The camera tracking system may be further configured to respond when one of the fiducials in the reference array 2102 is a missing fiducial that does not have an identifiable pose by estimating 2804 a pose of the missing fiducial relative to the plurality of fiducials having the identified poses, and providing 2804 another graphical marker 2710 displayed by the display of the XR headset 920 as at least partially overlapping the estimated pose of the missing fiducial. The process of estimating 2804 the pose of the missing fiducials may be performed based on a comparison of the currently identified poses of the fiducials 2700-2706 with the identified relative poses of the reference array 2102 to estimate the current poses of the missing fiducials. In this way, the surgeon may intuitively identify which fiducials are being correctly tracked by the tracking camera, and may also identify which fiducials, if any, are not being correctly tracked. For example, when the missing fiducials are excessively obscured by bodily fluids or other materials, the surgeon may remedy the situation to achieve accurate tracking of the reference array 2102.

[0199] Some additional embodiments are directed to avoiding interpreting movements of a user's hand while holding a surgical tool as the user attempting to provide input to a hand tracking input interface. This may be particularly important when a surgeon is holding a surgical tool because the tracking camera may not be able to maintain an adequate field of view of the hand during movement of the surgical tool, and this may result in inaccurate tracking of the hand and cause unexpected interactions with the user interface. According to these additional embodiments, a hand-user interface exclusion zone is defined relative to a reference array. Although it is determined that the user's hand is at least partially within the hand-user interface exclusion zone, the hand tracking information is not used as input from the user to the user interface.

[0200] Fig.29 A flow diagram is shown of operations performed by a camera tracking system to selectively allow receipt of gesture input based on whether a user's hand is at least partially within a hand-user interface exclusion zone determined relative to a reference array in accordance with some embodiments of the present disclosure. Fig.30 A hand-user interface exclusion zone 3000 is shown determined relative to a reference array 2102 connected to a surgical tool 2100 in accordance with some embodiments of the present disclosure.

[0201] refer to Fig.29, the camera tracking system is configured to receive 2900 tracking information indicating a position of a reference array 2102 relative to the XR headset and a position of a user's hand from a set of tracking cameras. The camera tracking system determines 2902 a hand-user interface exclusion area 3000 based on the tracking information relative to the position of the reference array 2102. When it is determined that the user's hand is completely outside the hand-user interface exclusion area 3000, the camera tracking system allows 2904 to receive gesture input from the user based on the tracked position of the hand indicated by the tracking information. In contrast, when it is determined that the user's hand is at least partially within the hand-user interface exclusion area 3000, the camera tracking system prohibits 2906 from receiving gesture input from the user based on the tracked position of the hand indicated by the tracking information. Therefore, when the user's hand is outside the exclusion area 3000, the camera tracking system may interpret the movement of the user's hand as the provided gesture input, for example, in a display such as Fig.19 1900. In contrast, when the user's hand is at least partially within the exclusion zone 3000, such as when holding the surgical tool 2100, the camera tracking system will prevent the user's hand movements from being interpreted as gesture input to the system, and will therefore, for example, prevent the user from using gestures to select from the displayed list 1910 of modes.

[0202] Although various embodiments have been described in the context of using a set of tracking cameras and an XR headset, these and other embodiments may be used with any form of tracking cameras, such as a set of tracking cameras on an auxiliary tracking pole and / or on another XR headset. Thus, in some embodiments, when the camera tracking system is receiving tracking information from the set of tracking cameras, the set of tracking cameras is separated and spaced apart from the XR headset. The various operations for pose linking disclosed herein may be used to track the pose of a reference array and / or a user's hand.

[0203] Other definitions and embodiments:

[0204] In the above description of each embodiment of the inventive concept, it should be understood that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the inventive concept. Unless otherwise defined, all terms used herein (including technical terms and scientific and technological terms) have the same meanings commonly understood by those skilled in the art to which the inventive concept belongs. It should be further understood that the terms as defined in the commonly used dictionary should be interpreted as having the meaning consistent with its meaning in the context of this specification and in the relevant field, and are clearly defined in this article, and will not be interpreted in an idealized or overly formal sense.

[0205] When an element is referred to as being "connected to" or "coupled to" "in response to" another element or its variant, it may be directly connected to, coupled to or responsive to another element, or there may be an intermediate element. In contrast, when an element is referred to as being "directly connected to", "directly coupled to", "directly responsive to" another element or its variant, there is no intermediate element. Similar elements are referred to with similar numbers throughout the text. In addition, "connection", "connection", "response" or its variants as used herein may include wireless connection, connection or response. Unless the context clearly indicates otherwise, as used herein, the singular forms "a, an" and "the" are intended to include plural forms. For the sake of brevity and / or clarity, well-known functions or configurations may not be described in detail. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0206] It should be understood that, although the terms first, second, third, etc. can be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish an element / operation from another element / operation. Therefore, without departing from the teaching of the present invention concept, the first element / operation in some embodiments can be referred to as the second element / operation in other embodiments. Throughout this specification, the same reference numerals or the same figure designators represent the same or similar elements.

[0207] As used herein, the terms "comprise / comprising / comprises", "include / including / includes", "have / has / having" or variations thereof are open ended and include one or more stated features, integers, elements, steps, components, or functions, but do not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or combinations thereof. In addition, as used herein, the general abbreviation "such as", derived from the Latin phrase "exempli gratia", may be used to introduce or specify one or more general examples of previously mentioned items and is not intended to limit such items. The general abbreviation "that is", derived from the Latin phrase "id est", may be used to specify a specific item from a more general statement.

[0208] Example embodiments are described herein with reference to block diagrams and / or flow diagrams of computer-implemented methods, devices (systems and / or apparatuses), and / or computer program products. It should be understood that the blocks shown in the block diagrams and / or flow diagrams and the combination of blocks shown in the block diagrams and / or flow diagrams can be implemented by computer program instructions executed by one or more computer circuits. These computer program instructions can be provided to processor circuits of general-purpose computer circuits, special-purpose computer circuits, and / or other programmable data processing circuits to produce a machine so that instructions executed by processors of computers and / or other programmable data processing devices convert and control transistors, values ​​stored in memory locations, and other hardware components within such circuit systems to implement the functions / actions specified in the block diagrams and / or one or more flow diagram blocks, and thereby create a device (function) and / or structure for implementing the functions / actions specified in the block diagrams and / or one or more flow diagram blocks.

[0209] These tangible computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing device to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture containing instructions for implementing the functions / actions specified in the block diagram and / or one or more flowchart blocks. Thus, embodiments of the inventive concept may be embodied in hardware and / or software (including firmware, resident software, microcode, etc.) that runs on a processor such as a digital signal processor, which may be collectively referred to as a "circuitry system," "module," or variations thereof.

[0210] It should also be noted that, in some alternative implementations, the function / action marked in the frame may not occur in the order marked in the flow chart. For example, according to the function / action involved, the two frames shown in succession can actually be performed substantially simultaneously, or the frame can sometimes be performed in reverse order. In addition, the function of a given frame of a flow chart and / or block diagram can be divided into multiple frames, and / or the function of two or more frames of a flow chart and / or block diagram can be integrated at least in part. Finally, without departing from the scope of the inventive concept, other frames can be added / inserted between the frames shown, and / or frame / operation can be omitted. In addition, although some figures in the figure include arrows on the communication path to illustrate the main direction of communication, it should be understood that communication can occur in the direction opposite to the arrows depicted.

[0211] Without substantially departing from the principle of the inventive concept, many changes and modifications may be made to the embodiments. All such variants and changes are intended to be included in the scope of the inventive concept herein. Therefore, the subject matter disclosed above should be considered illustrative rather than restrictive, and the examples of the attached embodiments are intended to cover all such modifications, enhancements and other embodiments that fall within the spirit and scope of the inventive concept. Therefore, in order to be allowed to the greatest extent by law, the scope of the inventive concept will be determined by the most widely allowed interpretation of the present disclosure, which includes the examples of the following embodiments and their equivalents, and should not be limited to or confined to the aforementioned specific detailed description.

Claims

1. A camera tracking system for computer-assisted navigation during surgery, the camera tracking system being configured to: The registration process begins automatically when a reference array of surgical tools is brought into the field of view of a set of tracking cameras attached to an extended reality XR headset. automatically identifying the reference array of surgical tools for tracking by the set of tracking cameras attached to the XR headset adapted to be worn by a user, the set of tracking cameras of the XR headset adapted to track a plurality of surgical tools and supplement a primary navigation system having a plurality of cameras for tracking the same plurality of surgical tools as well as anatomy of a patient; automatically determining whether the identified reference array captured by the tracking camera of the XR headset is registered to pair with a characteristic of one of the plurality of surgical tools defined in a surgical tool database; Based on the reference array being determined to be unregistered, providing a list of at least some of the plurality of surgical tools defined in the surgical tool database to a display device of the XR headset for the user to select the one of the plurality of surgical tools to be registered to pair with the reference array; Based on the reference array being determined to be misregistered and receiving user input, registering the reference array to pair with a characteristic of one of the plurality of surgical tools selected based on the user input, wherein the user input is in the form of one or more gestures recognizable by the set of tracking cameras of the XR headset; as well as A representation of the characteristic is provided to the display device of the XR headset for display to the user.

2. The camera tracking system according to claim 1, further configured to: A selection of the user in the displayed list is determined based on tracking information from the set of tracking cameras indicating a pose of the user's hand in XR space relative to the displayed list of one of the plurality of surgical tools to be registered with the reference array.

3. The camera tracking system according to claim 1, further configured to: receiving tracking information from the set of tracking cameras indicating a pose of the reference array relative to the XR headset; determining, based on the tracking information, a pose of a position at which the representation of the characteristic is to be displayed relative to the reference array; as well as The XR headset is controlled to display the representation of the characteristic at the determined pose relative to the reference array.

4. The camera tracking system according to claim 1, further configured to: Obtaining tracking accuracy information characterizing the accuracy with which the set of tracking cameras currently tracks the pose of the reference array; and An indication of the tracking accuracy information is provided to the display device of the XR headset for display to the user.

5. The camera tracking system according to claim 1, further configured to: obtaining a graphical representation of the one of the plurality of surgical tools in at least 2 dimensions; receiving tracking information from the set of tracking cameras indicating a pose of the reference array relative to the XR headset; determining a pose for the graphical representation based on the tracking information; as well as The graphical representation as the representation of the characteristic is provided to the display device of the XR headset for display at the determined pose.

6. The camera tracking system according to claim 1, further configured to: obtaining a last verification date indicating when the one of the plurality of surgical tools was last verified to be free of structural deformation; and An indication of the last validation date is provided to the display device of the XR headset for display to the user.

7. The camera tracking system according to claim 1, further configured to: determining when the reference array has entered a detection area defined relative to the position of the XR headset; In response to determining that the reference array has entered the detection area, initiating automatic determination of whether the identified reference array captured by the tracking camera of the XR headset is registered to be paired with a characteristic of one of the plurality of surgical tools defined in the surgical tool database; and Receiving user input to register the reference array to pair with a characteristic of one of the plurality of surgical tools defined in the surgical tool database is limited to only while the reference array remains within the detection region.

8. The camera tracking system according to claim 1, further configured to: determining when the reference array has entered a detection area defined relative to the position of the XR headset; In response to determining that the reference array has entered the detection area, commencing said providing the representation of the characteristic to the display device of the XR headset for display to the user; and In response to determining that the reference array has left the detection area, ceasing any ongoing provision of the representation of the characteristic for display to the user to the display device of the XR headset.

9. The camera tracking system according to claim 1, further configured to: identifying poses of a plurality of fiducials in the reference array; and For each of the plurality of fiducials in the reference array having an identified pose, a graphical indicia is provided for display by the display device to at least partially overlap the identified pose of the fiducial.

Images