Camera tracking rod of a camera tracking system for computer-aided navigation during surgery

Abstract

A camera tracking rod for a camera tracking system for computer - assisted navigation during surgery. The camera tracking rod includes a first set of stereo tracking cameras having a first resolution, a first field of view, and spaced apart on the camera tracking rod by a first baseline distance. The camera tracking rod further includes a second set of stereo tracking cameras having a second resolution, a second field of view, and spaced apart on the camera tracking rod by a second baseline distance that is less than the first baseline distance. The second set of stereo tracking cameras is positioned between the first set of stereo tracking cameras, and the resolution and / or field of view of the second set of stereo tracking cameras is different from the resolution and / or field of view of the first set of stereo tracking cameras. A communication interface provides a camera video stream to a camera tracking subsystem.

Description

Technical Field

[0001] The present disclosure relates to medical devices and systems, and more particularly, to a camera tracking system for computer-aided navigation during surgery. Background Art

[0002] Computer-aided navigation during surgery can provide a surgeon with a computerized visualization of the current pose of a surgical tool relative to a medical image of a patient's anatomy. A camera tracking system for computer-aided navigation uses one or more stereoscopic camera systems to track a set of fiducial points that are attached to a surgical tool that is positioned by a surgeon or other user during surgery. The set of fiducial points, also referred to as a dynamic reference array, allows the camera tracking system to determine the pose of the surgical tool relative to the anatomical structures in the medical image and relative to the patient for display to the surgeon. Thus, the surgeon can use real-time pose feedback to navigate the surgical tool during the surgical procedure.

[0003] When the object being tracked is moved outside the tracking area of the camera system or is obstructed from view by an intervening person and / or instrument, the navigation of the surgical procedure using an existing navigation system is prone to events that trigger intermittent pauses. There is also a need to improve the tracking accuracy of the navigation system. Summary of the Invention

[0004] Various embodiments disclosed herein relate to improvements in computer-aided navigation during surgery.

[0005] Some embodiments are directed to a camera tracking pole for a camera tracking system for computer-aided navigation during surgery. The camera tracking pole includes a first set of stereoscopic tracking cameras having a first resolution, a first field of view, and spaced apart on the camera tracking pole by a first baseline distance. The camera tracking pole further includes a second set of stereoscopic tracking cameras having a second resolution, a second field of view and spaced apart on the camera tracking pole by a second baseline distance that is less than the first baseline distance. The second set of stereoscopic tracking cameras is positioned between the first set of stereoscopic tracking cameras, and the resolution and / or field of view of the second set of stereoscopic tracking cameras is different from the resolution and / or field of view of the first set of stereoscopic tracking cameras. The camera tracking pole further includes a communication interface configured to provide video streams from the first set of stereoscopic tracking cameras and the second set of stereoscopic tracking cameras to a camera tracking subsystem.

[0006] Related embodiments of operations and methods performed by the camera tracking system are disclosed.

[0007] After reviewing the following drawings and detailed description, other camera tracking poles, camera tracking systems, computer program products, and methods according to the embodiments will be or become apparent to those skilled in the art. It is contemplated that all such camera tracking poles, camera tracking systems, computer program products, and methods are included in this specification, within the scope of the present disclosure, and are protected by the appended claims. Additionally, it is intended that all embodiments disclosed herein can be implemented singly or in combination in any manner and / or combination. Description of the Drawings

[0008] The drawings illustrate some non - limiting embodiments of the inventive concept and are included to provide a further understanding of the present disclosure and are incorporated in and form a part of this application. In the drawings:

[0009] Figure 1 An embodiment of a surgical system according to some embodiments of the present disclosure is shown;

[0010] Figure 2 An embodiment of a surgical robot assembly of a surgical system according to some embodiments of the present disclosure is shown; Figure 1 of a surgical system

[0011] Figure 3A An embodiment of a camera tracking system assembly of a surgical system according to some embodiments of the present disclosure is shown; Figure 1 of a surgical system

[0012] Figure 3B and 3C A front view and an isometric view of another camera tracking system assembly that can be used with a surgical system according to some embodiments of the present disclosure are shown; Figure 1 of a surgical system

[0013] Figure 4 An embodiment of an end - effector that can be connected to a robotic arm and configured according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A medical operation is shown in which a surgical robot and a camera system are positioned around a patient;

[0015] Figure 6 A block diagram view of components of a surgical system for a medical operation is shown; Figure 5 of a surgical system

[0016] Figure 7 A variety of display screens that can be displayed on a display when using the navigation function of a surgical system are shown; Figure 5 and Figure 6 of a display

[0017] Figure 8Shows a block diagram of some electrical components of a surgical robot according to some embodiments of the present disclosure;

[0018] Figure 9 Shows a block diagram of components of a surgical system according to some embodiments of the present disclosure, the surgical system including an imaging device connected to a computer platform that can be operatively connected to a camera tracking system and / or a surgical robot;

[0019] Figure 10 Shows an embodiment of a C-arm imaging device that can be used in combination with a surgical robot according to some embodiments of the present disclosure;

[0020] Figure 11 Shows an embodiment of an O-arm imaging device that can be used in combination with a surgical robot according to some embodiments of the present disclosure;

[0021] Figure 12 Shows a block diagram view of components of a surgical system including a pair of XR head-mounted devices and an auxiliary tracking rod operating according to some embodiments of the present disclosure;

[0022] Figure 13 Shows an XR head-mounted device configured according to some embodiments of the present disclosure;

[0023] Figure 14 Shows the electrical components of an XR head-mounted device that can be operatively connected to a computer platform, one or more imaging devices, and / or a surgical robot according to some embodiments of the present disclosure;

[0024] Figure 15 Shows a block diagram showing the arrangement of the optical components of an XR head-mounted device according to some embodiments of the present disclosure;

[0025] Figure 16 Shows an example view through the display screen of an XR head-mounted device for providing navigation assistance to manipulate a surgical tool during a medical procedure according to some embodiments of the present disclosure;

[0026] Figure 17 Shows an exemplary configuration of an auxiliary tracking rod with two pairs of stereo cameras configured according to some embodiments of the present disclosure;

[0027] Figure 18 Shows a block diagram view of components of a surgical system including a pair of XR head-mounted devices and an auxiliary tracking rod including a tracking camera operating together according to some embodiments of the present disclosure;

[0028] Figure 19 Shows some embodiments of the main components of the disclosed camera tracking rod according to some embodiments of the present disclosure.

[0029] Figure 20 and Figure 21 shows a tracking volume generated by a first set of stereo tracking cameras according to some embodiments of the present disclosure;

[0030] Figure 22 and Figure 23 shows a tracking volume generated by a second set of stereo tracking cameras according to some embodiments of the present disclosure;

[0031] Figure 24 and Figure 25 shows the simulated triangulation accuracy of the first set of stereo tracking cameras and the second set of stereo tracking cameras according to some embodiments of the present disclosure;

[0032] Figure 26 shows a dual-pass filter applied to video frames output by the first set and the second set of stereo tracking cameras according to some embodiments of the present disclosure;

[0033] Figure 27 shows a retroreflective tracking array having retroreflective spheres thereon according to some embodiments of the present invention; and

[0034] Figure 28 and Figure 29 shows a flowchart of operations performed by a camera tracking subsystem according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0035] The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which examples of embodiments of the inventive concept are shown. The inventive concept may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of each inventive concept to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components in one embodiment may be tacitly present in or used in another embodiment.

[0036] The various embodiments disclosed herein relate to improvements in computer - assisted navigation during surgery. A camera tracking pole of a camera tracking system is disclosed, which includes a first set of stereo tracking cameras having a first resolution, a first field of view, and spaced apart on the camera tracking pole by a first baseline distance. The camera tracking pole further includes a second set of stereo tracking cameras having a second resolution, a second field of view, and spaced apart on the camera tracking pole by a second baseline distance that is less than the first baseline distance. The second set of stereo tracking cameras is positioned between the first set of stereo tracking cameras, and the resolution and / or field of view of the second set of stereo tracking cameras is different from the resolution and / or field of view of the first set of stereo tracking cameras. The camera tracking pole further includes a communication interface configured to provide video streams from the first set of stereo tracking cameras and the second set of stereo tracking cameras to a camera tracking subsystem. Various camera configurations and operations are disclosed that enable the camera tracking pole to provide improved object tracking for computer - assisted navigation during surgery.

[0037] Figure 1 An embodiment of a surgical system 2 according to some embodiments of the present disclosure is shown. Before performing an orthopedic or other surgical procedure, a three - dimensional (“3D”) image scan of a planned surgical area of a patient can be performed using, for example, Figure 10 a C - arm imaging device 104 or Figure 11 an O - arm imaging device 106, or from another medical imaging device such as a computed tomography (CT) image or an MRI. Such a scan can be performed pre - operatively (e.g., most commonly, a few weeks before the procedure) or intra - operatively. However, any known 3D or 2D image scan can be used according to the various embodiments of the surgical system 2. The image scan is sent to a computer platform in communication with the surgical system 2, such as the computer platform 910 of the surgical system 900 ( Figure 9 ), which surgical system can include a camera tracking system component 6, a surgical robot 4 (e.g., Figure 1 the robot 2 in Figure 9 ), imaging devices (e.g., C - arm 104, O - arm 106, etc.), and an image database 950 for storing the patient's image scan. A surgeon viewing one or more image scans on a display device of the computer platform 910 ( Figure 9 ) generates a surgical plan that defines the target poses of surgical tools to be used during a surgical procedure on the patient's anatomical structure. Example surgical tools, also referred to as tools, can include but are not limited to drills, screwdrivers, saws, retractors, and implants such as screws, spacers, interbody fusion devices, plates, rods, etc. In some embodiments, the surgical plan that defines a target plane is planned on the 3D image scan displayed on the display device.

[0038] As used herein, the term "pose" refers to the position and / or rotational angle of an object (e.g., a dynamic reference array, an end effector, a surgical tool, an anatomical structure, etc.) relative to another object and / or a defined coordinate system. Thus, a pose can be defined based solely on the multi-dimensional position of an object relative to another object and / or a defined coordinate system, solely on the multi-dimensional rotational angle of the object relative to another object and / or a defined coordinate system, or in combination with the multi-dimensional position and the multi-dimensional rotational angle. Thus, the term "pose" is used to refer to a position, a rotational angle, or a combination thereof.

[0039] Figure 1 The surgical system 2 can assist a surgeon during a medical procedure by, for example, holding tools, alignment tools, use tools, guiding tools, and / or positioning tools for use. In some embodiments, the surgical system 2 includes a surgical robot 4 and a camera tracking system component 6. The ability to mechanically couple the surgical robot 4 and the camera tracking system component 6 can allow the surgical system 2 to be manipulated and moved as a single unit, and allows the surgical system 2 to have a small footprint in the area, allowing for easier movement through narrow passages and around turns, and allowing storage within a smaller area.

[0040] The surgical procedure can begin with the surgical system 2 being moved from a medical storage room to a medical procedure room. The surgical system 2 can be maneuvered through doorways, halls, and elevators to reach the medical procedure room. Within the medical procedure room, the surgical system 2 can be physically separated into two separate and distinct systems (the surgical robot 4 and the camera tracking system component 6). The surgical robot 4 can be positioned adjacent to the patient at any suitable location to appropriately assist medical personnel. The camera tracking system component 6 can be positioned at the bottom of the patient, at the patient's shoulder, or any other location suitable for tracking the current pose and pose movement of the robotic telescoping support arm 16 of the surgical robot 4 and the patient. The surgical robot 4 and the camera tracking system component 6 can be powered by an on-board power supply and / or plugged into an exterior wall outlet.

[0041] The surgical robot 4 can be used to assist a surgeon by holding and / or using tools during a medical procedure. To appropriately utilize and hold tools, the surgical robot 4 can rely on multiple motors, computers, and / or actuators to function properly. As Figure 1 shown, the robot body 8 can serve as a structure within which multiple motors, computers, and / or actuators can be secured within the surgical robot 4. The robot body 8 can also provide support for the robotic telescoping support arm 16. The size of the robot body 8 can provide a stable platform for supporting the attachment assembly, and can house, conceal, and protect the multiple motors, computers, and / or actuators that can operate the attachment assembly.

[0042] The robotic base 10 can serve as the lower support for the surgical robot 4. In some embodiments, the robotic base 10 can support the robotic body 8 and can attach the robotic body 8 to a plurality of powered wheels 12. This attachment to the wheels can allow the robotic body 8 to move effectively in space. The robotic base 10 can run along the length and width of the robotic body 8. The robotic base 10 can be from about two inches to about 10 inches in height. The robotic base 10 can cover, protect, and support the powered wheels 12.

[0043] In some embodiments, as Figure 1 shown, at least one powered wheel 12 can be attached to the robotic base 10. The powered wheel 12 can be attached to the robotic base 10 at any location. Each individual powered wheel 12 can rotate about a vertical axis in any direction. A motor can be disposed above, within, or adjacent to the powered wheel 12. Such a motor can allow the surgical system 2 to be maneuvered to any position and to stabilize and / or level the surgical system 2. A rod located within or adjacent to the powered wheel 12 can be pressed into a surface by the motor. The rod, not shown, can be made of any suitable metal to lift the surgical system 2. The rod can lift the powered wheel 10, which can lift the surgical system 2, to any height required to level or otherwise fix the orientation of the surgical system 2 relative to the patient. The weight of the surgical system 2 is supported by the small contact area of the rod on each wheel, preventing the surgical system 2 from moving during a medical procedure. This rigid positioning can prevent an object and / or person from accidentally moving the surgical system 2.

[0044] The movement of the mobile surgical system 2 can be facilitated using a robotic track 14. The robotic track 14 gives a person the ability to move the surgical system 2 without grasping the robotic body 8. As Figure 1 shown, the length of the robotic track 14 can be as long as, shorter than, and / or longer than the robotic body 8. The robotic track 14 can further provide protection to the robotic body 8, thereby preventing an object and / or medical staff from contacting, hitting, or bumping into the robotic body 8.

[0045] The robotic body 8 can provide support for a Selective Compliance Assembly Robot Arm, hereinafter referred to as "SCARA". Due to the repeatability and compactness of the robotic arm, it may be advantageous to use a SCARA 24 within the surgical system 2. The compactness of the SCARA can provide additional space within a medical procedure, which can allow medical professionals to perform a medical procedure without excessive clutter and restricted areas. The SCARA 24 can include a robotic telescoping support 16, a robotic support arm 18, and / or a robotic arm 20. The robotic telescoping support 16 can be disposed along the robotic body 8. As Figure 1As shown, the robotic telescoping support 16 can provide support for the SCARA 24 and the display 34. In some embodiments, the robotic telescoping support 16 can extend and contract in the vertical direction. The body of the robotic telescoping support 16 can be any width and / or height configured to support the stresses and weights placed thereon.

[0046] In some embodiments, a healthcare provider can move the SCARA 24 via a command submitted by the healthcare provider. As will be explained in further detail below, the command can originate from an input received on the display 34, a tablet computer, and / or an XR headset (e.g., Figure 9 the headset 920 in ). The XR headset can eliminate the need for the healthcare provider to reference any other display such as the display 34 or the tablet computer, which enables the configuration of the SCARA 24 without the display 34 and / or the tablet computer. As will be explained in further detail below, the command can be generated by pressing a switch and / or pressing multiple switches, and / or can be generated based on gesture commands and / or voice commands sensed by the XR headset.

[0047] As Figure 5 shown, the activation assembly 60 can include a switch and / or multiple switches. The activation assembly 60 can be operable to transmit a movement command to the SCARA 24, thereby allowing an operator to manually manipulate the SCARA 24. When the switch or multiple switches are pressed, the healthcare provider can have the ability to move the SCARA 24 via the applied hand movement. Alternatively or additionally, as will be explained in further detail below, the operator can control the movement of the SCARA 24 via gesture commands and / or voice commands sensed by the XR headset. Additionally, when the SCARA 24 does not receive a command to move, the SCARA 24 can be locked in place to prevent accidental movement by the healthcare provider and / or other objects. By being locked in place, the SCARA 24 provides a stable platform through which the end effector 26 can guide a surgical tool during a medical procedure.

[0048] The robotic support arm 18 can be connected to the robotic telescoping support 16 via various mechanisms. In some embodiments, best seen in Figure 1 and 2, the robotic support arm 18 rotates relative to the robotic telescoping support 16 in any direction. The robotic support arm 18 can rotate three hundred and sixty degrees about the robotic telescoping support 16. The robotic arm 20 can be connected to the robotic support arm 18 at any suitable location and by various mechanisms that enable rotation in any direction relative to the robotic support arm 18. In one embodiment, the robotic arm 20 can rotate three hundred and sixty degrees relative to the robotic support arm 18. This free rotation allows the operator to position the robotic arm 20 according to the surgical plan.

[0049] Figure 4 and Figure 5 The end effector 26 in is attached to the robotic arm 20 at any suitable location. The end effector 26 is configured to be attached to the end effector coupler 22 of the robotic arm 20 positioned by the surgical robot 4. An example end effector 26 includes a tubular guide that guides the movement of an inserted surgical tool relative to the anatomical structure on which the surgical procedure is to be performed.

[0050] In some embodiments, a dynamic reference array 52 is attached to the end effector 26. The dynamic reference array (also referred to herein as "DRA" and "reference array") can be a rigid body, a marker or other marking that can be attached or formed on one or more XR head-mounted devices worn by a person in the operating room, an end effector, a surgical robot, a surgical tool during a navigated surgical procedure, and the anatomical structure of a patient (such as a bone). The combination of the computer platform 910 and the camera tracking system component 6 or other 3D positioning system is configured to track the pose (e.g., position and rotational orientation) of the DRA in real time. The DRA can include fiducial points, as shown in the arrangement of the balls. This tracking of the 3D coordinates of the DRA can allow the surgical system 2 to determine the pose of the DRA in any multi-dimensional space relative to Figure 5 the target anatomical structure of the patient 50 in.

[0051] As Figure 1 shown, the light indicator 28 can be positioned on top of the SCARA 24. The light indicator 28 can be lit as any type of light to indicate the "status" in which the surgical system 2 is currently operating. In some embodiments, the light can be generated by an LED light that can form a ring around the light indicator 28. The light indicator 28 can include a fully permeable material that allows light to pass through the entire light indicator 28. The light indicator 28 can be attached to the lower display support 30. As Figure 2As shown, the lower display support 30 may allow an operator to manipulate the display 34 to any suitable position. The lower display support 30 may be attached to the light indicator 28 by any suitable mechanism. In some embodiments, the lower display support 30 may rotate about the light indicator 28 or be rigidly attached thereto. The upper display support 32 may be attached to the lower display support 30 by any suitable mechanism.

[0052] In some embodiments, a tablet computer may be used in combination with the display 34 and / or used without the display 34. The tablet computer may be positioned on the upper display support 32 in place of the display 34 and may be removable from the upper display support 32 during a medical procedure. Additionally, the tablet computer may communicate with the display 34. The tablet computer may be capable of connecting to the surgical robot 4 via any suitable wireless and / or wired connection. In some embodiments, the tablet computer may be capable of programming and / or controlling the surgical system 2 during a medical procedure. When the surgical system 2 is controlled by the tablet computer, all input and output commands may be replicated on the display 34. Using the tablet computer may allow an operator to manipulate the surgical robot 4 without having to move around the patient 50 and / or the surgical robot 4.

[0053] As will be explained below, in some embodiments, a surgeon and / or other personnel may wear XR head-mounted devices that may be used in combination with the display 34 and / or the tablet computer, or one or more XR head-mounted devices may eliminate the need to use the display 34 and / or the tablet computer.

[0054] As Figure 3A and 5 shown, the camera tracking system assembly 6 works in combination with the surgical robot 4 via a wired or wireless communication network. Referring Figure 1 、 Figure 3A-3C and Figure 5 , the camera tracking system assembly 6 may include some components similar to those found in the surgical robot 4. For example, the camera body 36 may provide functions found in the robot body 8. The robot body 8 may provide an auxiliary tracking rod on which the camera 46 is mounted. The structure within the robot body 8 may also provide support for the electronics, communication devices, and power source used to operate the camera tracking system assembly 6. The camera body 36 may be made of the same material as the robot body 8. The camera tracking system assembly 6 may communicate directly with the XR head-mounted device, the tablet computer, and / or the display 34 via a wireless and / or wired network such that the XR head-mounted device, the tablet computer, and / or the display 34 can control the functions of the camera tracking system assembly 6.

[0055] The camera body 36 is supported by the camera base 38. The camera base 38 can be used as the robot base 10. In Figure 1 the embodiment, the camera base 38 can be wider than the robot base 10. The width of the camera base 38 can allow the camera tracking system assembly 6 to be connected to the surgical robot 4. As Figure 1 shown, the width of the camera base 38 can be large enough to fit over the outside of the robot base 10. When the camera tracking system assembly 6 is connected to the surgical robot 4, the additional width of the camera base 38 can allow the surgical system 2 to provide additional maneuverability and support for the surgical system 2.

[0056] Similar to the robot base 10, a plurality of drive wheels 12 can be attached to the camera base 38. Similar to the operation of the robot base 10 and the drive wheels 12, the drive wheels 12 can allow the camera tracking system assembly 6 to be stabilized and leveled relative to the patient 50 or set to a fixed orientation. This stabilization can prevent the camera tracking system assembly 6 from moving during a medical procedure and can prevent the camera 46 on the auxiliary tracking rod from losing tracking of the DRA connected to the XR head-mounted device and / or the surgical robot 4 and / or losing tracking of one or more DRAs 52 of the anatomical structure 54 and / or the tool 58 within the designated area 56 as Figure 3A and 5 shown. This stability and maintenance of tracking enhances the ability of the surgical robot 4 to operate effectively with the camera tracking system assembly 6. Additionally, the wide camera base 38 can provide additional support for the camera tracking system assembly 6. Specifically, as Figure 3A and 5 shown, when the camera 46 is positioned over the patient, the wide camera base 38 can prevent the camera tracking system assembly 6 from tipping over.

[0057] The camera telescoping support 40 can support the camera 46 on the auxiliary tracking rod. In some embodiments, the telescoping support 40 can move the camera 46 higher or lower in the vertical direction. The camera handle 48 can be attached to the camera telescoping support 40 at any suitable location and is configured to allow an operator to move the camera tracking system assembly 6 to a planned position prior to a medical operation. In some embodiments, the camera handle 48 can be used to lower and raise the camera telescoping support 40. The camera handle 48 can perform the raising and lowering of the camera telescoping support 40 by pressing buttons, switches, levers, and / or any combination thereof.

[0058] The lower camera support arm 42 can be attached to the camera telescoping support 40 at any suitable location. In an embodiment, as Figure 1As shown, the lower camera support arm 42 can rotate 360 degrees about the telescoping support 40. This free rotation can allow the operator to position the camera 46 in any suitable position. The lower camera support arm 42 can be connected to the telescoping support 40 by any suitable mechanism. The lower camera support arm 42 can be used to provide support for the camera 46. The camera 46 can be attached to the lower camera support arm 42 by any suitable mechanism. The camera 46 can pivot in any direction at the attachment area between the camera 46 and the lower camera support arm 42. In an embodiment, the curved track 44 can be disposed on the lower camera support arm 42.

[0059] The curved track 44 can be disposed at any suitable position on the lower camera support arm 42. As Figure 3A shown, the curved track 44 can be attached to the lower camera support arm 42 by any suitable mechanism. The curved track 44 can be of any suitable shape, and suitable shapes can be crescent, circular, flat, oval, and / or any combination thereof. The camera 46 can be movably disposed along the curved track 44. The camera 46 can be attached to the curved track 44 by, for example, rollers, brackets, braces, motors, and / or any combination thereof. Motors and rollers (not shown) can be used to move the camera 46 along the curved track 44. As Figure 3A shown, during a medical procedure, if an object blocks the camera 46 from viewing one or more of the tracked DRAs, the motor can correspondingly move the camera 46 along the curved track 44. This motorized movement can allow the camera 46 to move to a new position that is no longer blocked by the object without moving the components of the camera tracking system 6. When an object blocks the camera 46 from viewing one or more of the tracked DRAs, the camera tracking system component 6 can send a stop signal to the surgical robot 4, the XR head-mounted device, the display 34, and / or the tablet computer. The stop signal can prevent the SCARA 24 from moving until the camera 46 has reacquired the tracked DRA 52 and / or can warn the operator to wear the XR head-mounted device and / or view the display 34 and / or the tablet computer. This SCARA 24 can be configured to respond to the receipt of the stop signal by stopping the further movement of the base and / or the end effector coupler 22 until the camera tracking system can resume tracking the DRA.

[0060] Figure 3B and 3C show a front view and an isometric view of another camera tracking system component 6' that can be used with Figure 1 the surgical system or can be used independently of the surgical robot. For example, the camera tracking system component 6' can be used to provide navigated surgery without the use of robotic guidance. Figure 3B and3C One of the differences between the camera tracking system component 6' of Figure 3A and the camera tracking system component 6 of Figure 3B and 3C the camera tracking system component 6' of Figure 14 includes a housing for transporting the computer platform 910. The computer platform 910 can be configured to: perform camera tracking operations to track the DRA; perform a navigated surgical operation to provide surgical navigation information to a display device (e.g., an XR head-mounted device and / or other display devices); and perform other computing operations disclosed herein. Thus, the computer platform 910 can include a navigation computer, such as

[0061] Figure 6 One or more of the navigation computers in the navigation computer of Figure 5 Fig. Figure 6 shows a block diagram view of the components of a surgical system for a medical operation. Referring to Figure 3B and Figure 3C the tracking camera 46 on the auxiliary tracking rod has a navigation field of view 600 in which the poses (e.g., position and orientation) of the reference array 602 attached to the patient, the reference array 604 attached to the surgical instrument, and the robotic arm 20 are tracked. The tracking camera 46 can be part of the camera tracking system component 6' of

[0062] Figure 7 Fig. Figure 5 and Figure 6Various display screens shown on the display 34. The display screens can include, but are not limited to, patient radiographs with an overlaid graphical representation of an instrument model positioned relative to the anatomical structure based on a developed surgical plan and / or the pose of a tracked reference array, various user-selectable menus for controlling different stages of the surgical procedure, and the dimensional parameters (e.g., length, width, and / or diameter) of a virtual projected implant.

[0063] For navigated surgery, various processing components (e.g., computer platform 910) and associated software are provided as described below that enable preoperative planning of a surgical procedure (e.g., implant placement) and electronic transfer of the plan to a computer platform 910 to provide navigation information to one or more users during the planned surgical procedure.

[0064] For robotic navigation, various processing components (e.g., computer platform 910) and associated software are provided as described below that enable preoperative planning of a surgical procedure (e.g., implant placement) and electronic transfer of the plan to the surgical robot 4. The surgical robot 4 uses the plan to guide the robotic arm 20 and the attached end effector 26 to provide the target pose of the surgical tool relative to the patient's anatomical structure for the steps of the planned surgical procedure.

[0065] The following various embodiments relate to the use of one or more XR head-mounted devices that can be worn by a surgeon 610, an assistant 612, and / or other medical personnel to provide an improved user interface for receiving information and / or providing control commands to / from a surgical robot, a camera tracking system component 6 / 6', and / or other medical devices in the operating room.

[0066] Figure 8 A block diagram of some electrical components of the surgical robot 4 according to some embodiments of the present disclosure is shown. Refer to Figure 8, a load cell (not shown) can be configured to track the force applied to the end effector coupler 22. In some embodiments, the load cell can communicate with multiple motors 850, 851, 852, 853, and / or 854. When the load cell senses a force, information about the amount of the applied force can be distributed from the switch array and / or multiple switch arrays to the controller 846. The controller 846 can obtain the force information from the load cell and process it with a switching algorithm. The controller 846 uses the switching algorithm to control the motor driver 842. The motor driver 842 controls the operation of one or more of the motors 850, 851, 852, 853, and / or 854. The motor driver 842 can direct a particular motor to generate, for example, an equal amount of force as measured by the load cell through the motor. In some embodiments, as indicated by the controller 846, the generated force can come from multiple motors, such as 850 to 854. Additionally, the motor driver 842 can receive an input from the controller 846. The controller 846 can receive information about the direction of the force sensed by the load cell from the load cell. The controller 846 can process this information using a motion controller algorithm. The algorithm can be used to provide information to a particular motor driver 842. To replicate the direction of the force, the controller 846 can activate and / or deactivate certain motor drivers 842. The controller 846 can control one or more of the motors, such as one or more of 850 to 854, to induce movement of the end effector 26 in the direction of the force sensed by the load cell. This force-controlled movement can allow the operator to move the SCARA 24 and the end effector 26 effortlessly and / or with very little resistance. The movement of the end effector 26 can be performed to position the end effector 26 in any suitable pose (i.e., position and angular orientation relative to a defined three-dimensional (3D) orthogonal reference axis) for use by medical personnel.

[0067] Figure 5 Best shown, the activation assembly 60 can be in the form of a bracelet that wraps around the end effector coupler 22. The activation assembly 60 can be located on any part of the SCARA 24, any part of the end effector coupler 22, can be worn (and communicate wirelessly) by medical personnel, and / or any combination thereof. The activation assembly 60 can include a main button and a secondary button.

[0068] Pressing the main button can allow the operator to move the SCARA 24 and the end - effector coupler 22. According to one embodiment, once in place, the SCARA 24 and the end - effector coupler 22 may not move until the operator programs the surgical robot 4 to move the SCARA 24 and the end - effector coupler 22, or uses the main button to move. In some instances, it may be required to press at least two non - adjacent main activation switches before the SCARA 24 and the end - effector coupler 22 will respond to the operator's commands. Pressing at least two main activation switches can prevent accidental movement of the SCARA 24 and the end - effector coupler 22 during a medical procedure.

[0069] Activated by the main button, a force - sensing sensor can measure the magnitude and / or direction of the force applied by the operator (i.e., medical staff) on the end - effector coupler 22. This information can be transmitted to one or more motors within the SCARA 24, such as one or more of 850 - 854, which can be used to move the SCARA 24 and the end - effector coupler 22. Information regarding the magnitude and direction of the force measured by the force - sensing sensor can cause one or more motors (e.g., one or more of 850 - 854) to move the SCARA 24 and the end - effector coupler 22 in the same direction as that sensed by the force - sensing sensor. This force - controlled movement can allow the operator to easily move the SCARA 24 and the end - effector coupler 22, and since the motors move the SCARA 24 and the end - effector coupler 22 while the operator is moving them, it does not require a great deal of effort.

[0070] In some instances, the operator can use a secondary button as a "selection" device. During a medical operation, the surgical robot 4 can notify the medical staff of certain situations through one or more XR head - mounted devices 920, the display 34, and / or the light indicator 28. Each of the one or more XR head - mounted devices 920 is configured to display an image on a see - through display screen to form an extended reality image overlaid on real - world objects viewable through the see - through display screen. The surgical robot 4 can prompt the medical staff to select functions, modes, and / or evaluate the situation of the surgical system 2. Pressing the secondary button once can activate certain functions, modes, and / or confirm the information transmitted to the medical staff through one or more XR head - mounted devices 920, the display 34, and / or the light indicator 28. Additionally, pressing the secondary button multiple times in quick succession can activate additional functions, modes, and / or select the information transmitted to the medical staff through one or more XR head - mounted devices 920, the display 34, and / or the light indicator 28.

[0071] Further reference Figure 8, the electrical components of the surgical robot 4 include a platform subsystem 802, a computer subsystem 820, a motion control subsystem 840, and a tracking subsystem 830. The platform subsystem 802 includes a battery 806, a power distribution module 804, a connector panel 808, and a charging station 810. The computer subsystem 820 includes a computer 822, a display 824, and a speaker 826. The motion control subsystem 840 includes drive circuits 842, motors 850, 851, 852, 853, 854, stabilizers 855, 856, 857, 858, an end effector connector 844, and a controller 846. The tracking subsystem 830 includes a position sensor 832 and a camera converter 834. The surgical robot 4 may also include a removable foot pedal 880 and a removable tablet computer 890.

[0072] Input power is supplied to the surgical robot 4 through a power supply that can be provided to the power distribution module 804. The power distribution module 804 receives the input power and is configured to generate different power supply voltages for other modules, components, and subsystems of the surgical robot 4. The power distribution module 804 may be configured to provide different voltage supplies to the connector panel 808, and the voltage supplies may be provided to other components (such as the computer 822, the display 824, the speaker 826, the driver 842) to power, for example, the motors 850 to 854 and the end effector coupler 844, and to the camera converter 834 and other components of the surgical robot 4. The power distribution module 804 may also be connected to the battery 806, and the battery acts as a temporary power source when the power distribution module 804 does not receive power from the input power source. At other times, the power distribution module 804 may be used to charge the battery 806.

[0073] The connector panel 808 can be used to connect different devices and components to the surgical robot 4 and / or associated components and modules. The connector panel 808 may include one or more ports for receiving lines or connectors from different components. For example, the connector panel 808 may have a ground terminal port for grounding the surgical robot 4 to other devices, a port for connecting the foot pedal 880, a port for connecting to the tracking subsystem 830, which may include a position sensor 832, a camera converter 834, and a camera tracking rod 870. The connector panel 808 may also include other ports to allow USB, Ethernet, and HDMI communication with other components (such as the computer 822). According to some embodiments, the connector panel 808 may include wired and / or wireless interfaces for operably connecting one or more XR head-mounted devices 920 to the tracking subsystem 830 and / or the computer subsystem 820.

[0074] The control panel 816 can provide various buttons or indicators for controlling the operation of the surgical robot 4 and / or providing information from the surgical robot 4 for the operator to observe. For example, the control panel 816 can include buttons for turning the surgical robot 4 on or off, raising or lowering the vertical column 16, and raising or lowering the stabilizers 855 to 858, which can be designed to engage the casters 12 to lock the surgical robot 4 without physically moving it. Other buttons can stop the surgical robot 4 in case of an emergency, which can remove all motor power and apply mechanical brakes to stop all movement from occurring. The control panel 816 can also have indicators (such as a line power indicator) to notify the operator of certain system conditions or the charge status of the battery 806. According to some embodiments, one or more XR head-mounted devices 920 can communicate, for example, through the connector panel 808, to control the operation of the surgical robot 4 and / or receive and display information generated by the surgical robot 4 for the person wearing the XR head-mounted device 920 to observe.

[0075] The computer 822 of the computer subsystem 820 includes an operating system and software for operating the designated functions of the surgical robot 4. The computer 822 can receive and process information from other components (such as the tracking subsystem 830, the platform subsystem 802, and / or the motion control subsystem 840) in order to display information to the operator. Further, the computer subsystem 820 can provide output for the operator through the speaker 826. The speaker can be part of the surgical robot, part of the XR head-mounted device 920, or within another component of the surgical system 2. The display 824 can correspond to Figure 1 and Figure 2 the display 34 shown.

[0076] The tracking subsystem 830 can include a position sensor 832 and a camera converter 834. The tracking subsystem 830 can correspond to Figure 3A the camera tracking system component 6 of. The camera tracking rod 870 operates together with the position sensor 832 to determine the pose of the DRA 52. This tracking can be performed in a manner consistent with the present disclosure, which includes using infrared or visible light techniques that respectively track the positions of active or passive elements (such as LEDs or reflective fiducial points, also known as markers) of the DRA 52.

[0077] The functional operations of the tracking subsystem 830 and the computer subsystem 820 can be included in what can be by Figure 3A and Figure 3Bin the computer platform 910 to which the camera tracking system component 6' transmits. The tracking subsystem 830 can be configured to determine a pose, such as the position and angular orientation of the tracked DRA. The computer platform 910 can also include a navigation controller configured to use the determined pose to provide navigation information to guide the user to move the tracked tool relative to the positioned and registered patient image and / or the tracked anatomical structure during a planned surgical procedure. The computer platform 910 can display information on a display at Figure 3B and Figure 3C and / or to one or more XR headsets 920. When used with a surgical robot, the computer platform 910 can be configured to communicate with the computer subsystem 820 and Figure 8 other subsystems to control the movement of the end effector 26. For example, as will be explained below, the computer platform 910 can generate graphical representations of the patient's anatomical structure, surgical tools, the user's hand, etc., having a displayed size, shape, color, and / or pose that is controlled based on the determined pose of one or more tracked DRAs, and the displayed graphical representations can be dynamically modified to track changes in the determined pose over time.

[0078] The motion control subsystem 840 can be configured to physically move the vertical column 16, the upper arm 18, the lower arm 20, or rotate the end effector coupler 22. The physical movement can be performed by using one or more motors 850 to 854. For example, the motor 850 can be configured to vertically lift or lower the vertical column 16. As Figure 2 shown, the motor 851 can be configured to laterally move the upper arm 18 about the joint with the vertical column 16. As Figure 2 shown, the motor 852 can be configured to laterally move the lower arm 20 about the joint with the upper arm 18. The motors 853 and 854 can be configured to move the end effector coupler 22 to provide translational movement along three-dimensional axes and rotation about them. Figure 9 As shown, the computer platform 910 can provide a control input to the controller 846 to guide the movement of the end effector coupler 22 to position the passive end effector connected thereto in a planned pose (i.e., position and angular orientation relative to defined 3D orthogonal reference axes) relative to the anatomical structure to be operated on during a planned surgical procedure. The motion control subsystem 840 can be configured to use integrated position sensors (e.g., encoders) to measure the position of the end effector coupler 22 and / or the end effector 26.

[0079] Figure 9 shows a block diagram of components of a surgical system according to some embodiments of the present disclosure, the surgical system including a connection to a camera tracking system component 6 (Figure 3A ) or 6'( Figure 3B 、 3C ) and / or an imaging device of the computer platform 910 of the surgical robot 4 (e.g., C-arm 104, O-arm 106, etc.). Alternatively, at least some of the operations performed by the computer platform 910 disclosed herein may be additionally or alternatively performed by components of the surgical system.

[0080] Reference Figure 9 , the computer platform 910 includes a display 912, at least one processor circuit 914 (also referred to as a processor for simplicity), at least one memory circuit 916 containing computer-readable program code 918 (also referred to as a memory for simplicity), and at least one network interface 902 (also referred to as a network interface for simplicity). The display 912 may be part of an XR head-mounted device 920 according to some embodiments of the present disclosure. The network interface 902 may be configured to connect to Figure 10 the C-arm imaging device 104 in Figure 11 the O-arm imaging device 106 in, another medical imaging device, an image database 950 containing patient medical images, components of the surgical robot 4, and / or other electronic devices.

[0081] When used with the surgical robot 4, the display 912 may correspond to Figure 2 the display 34 of Figure 8 and / or Figure 8 the tablet computer 890 of Figure 8 and / or the XR head-mounted device 920 operably connected to the surgical robot 4. The network interface 902 may correspond to

[0082] the platform network interface 812 of

[0083] The computer platform 910 can be configured to provide surgical planning functionality. The processor 914 can operate to display on the display device 912 and / or on the XR headset 920 images of anatomical structures (e.g., vertebrae) received via the network interface 920 from one of the imaging devices 104 and 106 and / or from the image database 950. The processor 914 receives an operator's definition of the location for a surgical procedure (e.g., screw placement) to be performed on the anatomical structure shown in one or more images, such as by the operator touching to select the location on the display 912 for the planned surgical procedure or using a mouse-based cursor to define the location for the planned surgical procedure. As will be explained in further detail below, when the image is displayed in the XR headset 920, the XR headset can be configured to sense gesture-based commands formed by the wearer and / or sense voice-based commands spoken by the wearer, and the commands can be used to control the selection between menu items and / or control how objects are displayed on the XR headset 920.

[0084] The computer platform 910 can be configured to be capable of performing anatomical measurements that may be particularly useful for knee surgery, similar to measuring various angles for determining the hip center, the angular center, natural landmarks (e.g., the transepicondylar line, the Whitesides line, the posterior condylar line, etc.). Some measurements can be automatic, while some other measurements can involve human input or assistance. The computer platform 910 can be configured to allow the operator to input the selection of the correct implant for the patient, including the selection of size and alignment. The computer platform 910 can be configured to perform automatic or semi-automatic (involving human input) segmentation (image processing) of CT images or other medical images. The surgical plan for the patient can be stored in a cloud-based server that can correspond to the database 950 for retrieval by the surgical robot 4.

[0085] For example, during orthopedic surgery, the surgeon can use a computer screen (e.g., a touch screen) or extended reality (XR) interactions (e.g., gesture-based commands and / or voice-based commands) via, for example, the XR headset 920 to select the site to be cut (e.g., the posterior femur, the proximal tibia, etc.). The computer platform 910 can generate navigation information that provides visual guidance to the surgeon to perform the surgical procedure. When used with the surgical robot 4, the computer platform 910 can provide guidance that allows the surgical robot 4 to automatically move the end effector 26 to the target pose such that the surgical tool is aligned with the target location to perform the surgical procedure on the anatomical structure.

[0086] In some embodiments, the surgical system 900 may use two DRAs to track the patient's anatomical position, such as a DRA attached to the patient's tibia and a DRA attached to the patient's femur. The system 900 may use standard navigated instruments for registration and inspection (e.g., a pointer similar to those used in the Globus ExcelsiusGPS system for performing spinal surgery).

[0087] A particularly challenging task in navigated surgery is how to plan the position of implants in the spine, knee, and other anatomical structures, where the surgeon endeavors to perform the task on a computer screen, which is a 2D representation of a 3D anatomy. The system 900 can address this issue by using the XR headset 920 to display three-dimensional (3D) computer-generated representations of the anatomical structures and candidate implant devices. Under the guidance of the computer platform 910, the computer-generated representations are scaled and positioned relative to each other on the display screen and can be manipulated by the surgeon when viewed through the XR headset 920. The surgeon can manipulate the displayed computer-generated representations of the anatomical structures, implants, surgical tools, etc., using, for example, gesture-based commands and / or voice-based commands sensed by the XR headset 920.

[0088] For example, the surgeon can view the displayed virtual handles on the virtual implant and can manipulate (e.g., grasp and move) the virtual handles to move the virtual implant to the desired pose and adjust the planned implant placement relative to the graphical representation of the anatomical structure. Thereafter, during the surgery, the computer platform 910 can display, through the XR headset 920, navigation information that facilitates the surgeon's ability to more accurately follow the surgical plan to insert the implant and / or perform another surgical procedure on the anatomical structure. When the surgical procedure involves bone removal, the progress of the bone removal (e.g., cutting depth) can be displayed in real time through the XR headset 920. Other features that can be displayed through the XR headset 920 may include, but are not limited to, the gap or ligament balance along the range of motion of the joint, the contact line along the range of motion of the implant, the ligament tension and / or laxity rendered by color or other graphics, etc.

[0089] In some embodiments, the computer platform 910 may allow for the planning of the use of standard surgical tools and / or implants, such as posterior stabilized implants and cruciate-retaining implants, cemented and non-cemented implants, revision systems for surgeries related to, for example, total knee or partial knee and / or hip replacement and / or trauma.

[0090] An automated imaging system can be used in conjunction with the computer platform 910 to acquire pre-operative, intra-operative, post-operative, and / or real-time image data of the anatomical structure. Figure 10 and Figure 11illustrates an example of an automated imaging system. In some embodiments, the automated imaging system is a C-arm 104( Figure 10 ), an imaging device, or 106( Figure 11 ). (Copyright of is owned by Medtronic Navigation, Inc., with a place of business in Louisville, Colorado, USA). It may be desirable to perform x-ray examinations of a patient from many different positions without the need for frequent manual repositioning of the patient, which may be required in x-ray systems. The C-arm 104 x-ray diagnostic device can address the problem of frequent manual repositioning and is well known in the medical field of surgery and other interventional procedures. As Figure 10 shown, the C-arm includes an elongated C-shaped member that terminates at opposite distal ends 112 of the "C" shape. The C-shaped member is attached to an x-ray source 114 and an image receiver 116. The space within the C-arm 104 of the arm provides a space that is substantially free from interference by the x-ray support structure for the doctor to attend to the patient.

[0091] The C-arm is mounted such that the arm can rotate and move in two degrees of freedom (i.e., in a spherical motion about two perpendicular axes). The C-arm is slidably mounted to the x-ray support structure, which allows the C-arm to orbitally rotate about its center of curvature, which can allow the x-ray source 114 and the image receiver 116 to be selectively oriented vertically and / or horizontally. The C-arm can also be rotatable laterally (i.e., in a direction perpendicular to the direction of orbital travel so as to be able to selectively adjust the positioning of the x-ray source 114 and the image receiver 116 relative to the width and length of the patient). The spherical rotation aspect of the C-arm device allows the doctor to perform x-ray examinations of the patient at an optimal angle determined relative to the specific anatomical condition being imaged.

[0092] Figure 11 shown in 106 includes a gantry housing 124, which may enclose an image capture portion (not shown). The image capture portion includes an x-ray source portion and / or an x-ray emitting portion and an x-ray receiving portion and / or an image receiving portion, which may be disposed approximately one hundred and eighty degrees apart from each other and are mounted on a rotor (not shown) relative to the orbit of the image capture portion. The image capture portion can be operatively rotated three hundred and sixty degrees during image acquisition. The image capture portion can rotate about a center point and / or an axis, thereby allowing image data of the patient to be acquired from multiple directions or in multiple planes.

[0093] With the gantry housing 124 106 has a central opening for positioning around an object to be imaged, and a radiation source that can rotate internally around the gantry housing 124, and the radiation source can be adapted to project radiation from a plurality of different projection angles. The detector system is adapted to detect the radiation at each projection angle, so as to obtain an object image from a plurality of projection planes in a quasi-simultaneous manner. The gantry can be attached to the support structure in a cantilever manner Support structure, such as a wheeled mobile cart with wheels. The positioning unit preferably translates and / or tilts the gantry to a planned position and orientation under the control of a computerized motion control system. The gantry can include a source and a detector disposed opposite each other on the gantry. The source and the detector can be fixed to a motorized rotor, and the motorized rotor can cause the source and the detector to rotate around each other inside the gantry. The source can be pulsed at multiple positions and orientations in a partial and / or complete three-hundred-sixty-degree rotation to perform multi-plane imaging on a target object positioned inside the gantry. The gantry can further include an orbital and bearing system for guiding the rotor when the rotor rotates, and the orbital and bearing system can carry the source and the detector. Both 106 and the C-arm 104 and / or one of them can be used as an automatic imaging system to scan a patient and send information to the surgical system 2.

[0094] The images captured by the imaging system can be displayed on the XR head-mounted device 920 and / or the computer platform 910 of the surgical system 900, the surgical robot 4 and / or another display device of another component. The XR head-mounted device 920 can be connected to one or more imaging devices and / or the image database 950 in the imaging devices 104 and / or 106 through the computer platform 910, for example, to display images therefrom. The user can provide control inputs through the XR head-mounted device 920, for example, gesture- and / or voice-based commands to control the operation of one or more imaging devices and / or the image database 950 in the imaging devices 104 and / or 106.

[0095] Figure 12 A block diagram view of the components of a surgical system including a pair of XR head-mounted devices 920 (head-mounted displays HMD1 and HMD2) is shown, and the XR head-mounted devices can correspond to Figure 13 the XR head-mounted device 920 shown in and operate according to some embodiments of the present disclosure.

[0096] Referring to Figure 12 an example scenario of, both the assistant 612 and the surgeon 610 wear the XR head-mounted device 920 respectively. Optionally, the assistant 612 wears the XR head-mounted device 920. As will be further described below, the XR head-mounted device 920 is configured to provide an interactive environment for the wearer through which the wearer can view and interact with information related to the surgical procedure. This interactive XR-based environment can eliminate the need for the presence in the operating roomFigure 6 the needs of the technician 614 shown in Figure 6 the need for the display 34 shown in Figure 12 In an example, the XR head-mounted device 920 has a field of view (FOV) 1202 for tracking the DRA and other objects, the XR head-mounted device 920 has a FOV 1212 that partially overlaps the FOV 1202 for tracking the DRA and other objects, and the tracking camera 46 has another FOV 600 that partially overlaps the FOV 1202 and 1212 for tracking the DRA and other objects.

[0097] If one or more cameras are blocked from viewing the DRA attached to the tracked object (e.g., a surgical tool), but the DRA is in the view of one or more other cameras, the tracking subsystem 830 and / or the navigation controller 828 can continue to seamlessly track the object without losing navigation. Additionally, if the DRA is partially occluded from the perspective of one camera, but the entire DRA is visible through multiple camera sources, the tracking inputs from the cameras can be combined to continue navigating the DRA. One and / or the tracking camera 46 in the XR head-mounted device can view and track the DRA on another XR head-mounted device so that the computer platform 910 ( Figure 9 and Figure 14 )、the tracking subsystem 830 and / or another computing component can determine the pose of the DRA, such as the XR head-mounted device 920, the tracking camera 46, relative to one or more defined coordinate systems and / or another coordinate system that defines for the patient, the table, and / or the room.

[0098] The XR head-mounted device 920 can be operably connected to view video, pictures, and / or other received information and / or provide commands to control various devices in the operating room, including but not limited to nerve monitoring, microscopes, cameras, and anesthesia systems. Data from various devices can be processed and displayed within the head-mounted device, such as displaying the patient's vital signs or microscope feeds.

[0099] Example XR Head-Mounted Device Components and Integration with Navigated Surgery, Surgical Robots, and Other Devices

[0100] Figure 13An XR head-mounted device 920 configured according to some embodiments of the present disclosure is shown. The XR head-mounted device includes a headband 1306 configured to secure the XR head-mounted device to a wearer's head, an electronic component housing 1304 supported by the headband 1306, and a display screen 1302 that extends laterally across and downward from the electronic component housing 1304. The display screen 1302 can be a see-through LCD display device or a semi-reflective lens that reflects an image projected by a display device toward the wearer's eyes. A set of DRA reference points 1310 (e.g., dot-like portions) are sprayed or attached to one or both sides of the head-mounted device in a spaced-apart and known manner. The DRA on the head-mounted device enables a tracking camera on an auxiliary tracking rod to track the pose of the head-mounted device 920 and / or enables another XR head-mounted device to track the pose of the head-mounted device 920.

[0101] The display screen 1302 operates as a see-through display screen (also referred to as a combiner) that reflects light from a display panel of a display device toward the user's eyes. The display panel can be positioned between the electronic component housing and the user's head and angled to project virtual content toward the display screen 1302 for reflection toward the user's eyes. The display screen 1302 is semi-transparent and semi-reflective, allowing the user to see the reflected virtual content overlaid on the user's view of the real-world scene. The display screen 1302 can have different opacity regions, such as an upper lateral band having a higher opacity than a lower lateral band as shown. The opacity of the display screen 1302 can be electronically controlled to adjust the amount of light from the real-world scene that passes through to reach the user's eyes. The high-opacity configuration of the display screen 1302 enables a high-contrast virtual image to be overlaid on a dim view of the real-world scene. The low-opacity configuration of the display screen 1302 can enable a more blurred virtual image to be overlaid on a clearer view of the real-world scene. The opacity can be controlled by coating an opaque material on the surface of the display screen 1302.

[0102] According to some embodiments, a surgical system includes an XR head-mounted device 920 and an XR head-mounted device controller, such as Figure 14 the controller 1430 in. The XR head-mounted device 920 is configured to be worn by a user during a surgical procedure and has a see-through display screen 1302 configured to display XR images and allow at least a portion of the real-world scene to pass therethrough for the user to view. The XR head-mounted device 920 further includes an opacity filter positioned between at least one of the user's eyes and the real-world scene when the see-through display screen 1302 is being viewed by the user. The opacity filter is configured to provide opacity to light from the real-world scene. The XR head-mounted device controller is configured to communicate with a navigation controller (e.g., Figure 14communicate with one or more of the controllers 828A, 828B, and / or 828C) to receive navigation information for guiding the user during a surgical procedure on an anatomical structure and further configured to generate an XR image based on the navigation information for display on the fluoroscopy display screen 1302.

[0103] The opacity of the display screen 1302 can be configured to have a gradient of opacity that varies more continuously with distance from the top portion of the display screen 1302 downward. The darkest point of the gradient can be located at the top portion of the display screen 1302 and gradually becomes less opaque further down on the display screen 1302 until the opacity becomes transparent or non-existent. In another exemplary embodiment, the gradient can change from approximately 90% opacity to completely transparent at approximately the mid-eye level of the display screen 1302. With the head-mounted device properly calibrated and positioned, the mid-eye level can correspond to the point where the user would look straight ahead, and the end of the gradient will be located on the "horizontal" line of the eyes. The darker portion of the gradient will allow for a clear and distinct visualization of the virtual content and help block the intrusive brightness of the overhead operating room lights.

[0104] Using the opacity filter in this way enables the XR head-mounted device 920 to provide virtual reality (VR) functionality by substantially or completely blocking light from the real-world scene along the upper portion of the display screen 1302 and provide augmented reality (AR) functionality along the middle or lower portion of the display screen 1302. This allows the user to have a translucent AR where needed and during the procedure allows for clear visualization of the patient's anatomy at the location where the clear optics are used. Configuring the display screen 1302 as a gradient rather than a more constant opacity band can enable the wearer to experience a more natural transition between a more VR-type view and a more AR-type view without experiencing the brightness of the real-world scene and the abrupt changes in depth of field that can otherwise cause eye fatigue, such as during a more rapid shift between the up and down views.

[0105] The display panel and the display screen 1302 can be configured to provide a wide field of view fluoroscopy XR display system. In one exemplary configuration, they provide the user with an 80° diagonal field of view (FOV) and a 55° vertical coverage for the user to view virtual content. Other diagonal FOV angles and vertical coverage angles can be provided by different sized display panels, different curvature lenses, and / or different distances and angular orientations between the display panel and the curved display screen 1302.

[0106] Figure 14Shows the electrical components of the XR head-mounted device 920 according to some embodiments of the present disclosure, which may be operably connected to a computer platform 910, one or more imaging devices in an imaging device (such as a C-arm imaging device 104, an O-arm imaging device 106), and / or an image database 950 and / or a surgical robot 800.

[0107] The XR head-mounted device 920 provides an improved human-machine interface for performing a navigated surgical procedure. The XR head-mounted device 920 may be configured to provide functions, for example, through the computer platform 910, which include but are not limited to any one or more of the following: identifying gesture-based commands and / or voice-based commands, and displaying XR graphical objects on the display device 1450. The display device 1450 may be a video projector, a flat panel display, etc. that projects the displayed XR graphical objects onto the display screen 1302. The user may view the XR graphical objects as an overlay anchored to a specific real-world object viewed through the display screen 1302 ( Figure 13 ). The XR head-mounted device 920 may additionally or alternatively be configured to display a video feed from cameras mounted to one or more XR head-mounted devices 920 and other cameras on the display screen 1450.

[0108] The electrical components of the XR head-mounted device 920 may include a plurality of cameras 1440, a microphone 1442, a gesture sensor 1444, a pose sensor (e.g., an inertial measurement unit (IMU)) 1446, a display module 1448 including the display device 1450, and a wireless / wired communication interface 1452. As explained below, the cameras 1440 of the XR head-mounted device may be visible light capture cameras, near-infrared capture cameras, or a combination of both.

[0109] The camera 1440 may be configured to operate as a gesture sensor 1444 by capturing user gestures performed within the field of view of one or more cameras 1440 for identification. Alternatively, the gesture sensor 1444 may be a proximity sensor and / or a touch sensor that senses the proximity of gestures performed by the gesture sensor 1444 and / or senses physical contact (e.g., a tap sensor or the housing 1304). The pose sensor 1446 (e.g., an IMU) may include a multi-axis accelerometer, a tilt sensor, and / or another sensor that can sense the rotation and / or acceleration of the XR head-mounted device 920 along one or more defined coordinate axes. Some or all of these electrical components may be contained within the component housing 1304 or may be contained within another housing configured to be worn elsewhere (such as on the hip or shoulder).

[0110] As explained above, the surgical system 2 includes a camera tracking system component 6 / 6' and a tracking subsystem 830, which can be part of a computer platform 910. The surgical system can include an imaging device (e.g., a C-arm 104, an O-arm 106, and / or an image database 950) and / or a surgical robot 4. The tracking subsystem 830 is configured to determine the pose of a DRA attached to an anatomical structure, an end effector, a surgical tool, etc. The navigation controller 828 is configured to determine a target pose of a surgical tool relative to an anatomical structure based on a surgical plan (e.g., according to a surgical planning function performed by Figure 9 the computer platform 910) that defines positions for performing a surgical procedure on the anatomical structure using the surgical tool and based on the pose of the anatomical structure determined by the tracking subsystem 830. The navigation controller 828 can further be configured to generate steering information based on the target pose of the surgical tool, the pose of the anatomical structure, and the pose of the surgical tool and / or end effector, where the steering information indicates where the surgical tool and / or end effector of the surgical robot should be moved to perform the surgical plan. The various cameras 1440 of the XR head-mounted device 920 can be connected to the camera tracking system component 6 / 6' to track the pose of a DRA, the user's hand, etc.

[0111] The electrical components of the XR head-mounted device 920 can be operatively connected to the electrical components of the computer platform 910 via a wired / wireless interface 1452. The electrical components of the XR head-mounted device 920 can be operatively connected to various imaging devices (e.g., a C-arm imaging device 104, an I / O-arm imaging device 106, an image database 950) and / or other medical devices, for example, via the computer platform 910 or directly via the wired / wireless interface 1452.

[0112] The surgical system 2 further includes at least one XR head-mounted device controller 1430 (also referred to as the "XR head-mounted device controller" for brevity), which can reside in the XR head-mounted device 920, the computer platform 910, and / or in another system component connected via a wired cable and / or a wireless communication link. The software executed by the XR head-mounted device controller 1430 provides various functions. The XR head-mounted device controller 1430 is configured to receive navigation information for guiding a user during a surgical procedure on an anatomical structure from the navigation controller 828, and is configured to generate an XR image based on the navigation information for display on a display device 1450 for projection on a fluoroscopic display screen 1302.

[0113] The configuration of the display device 1450 relative to the display screen (also referred to as a "see-through display screen") 1302 is configured such that XR images are displayed in a manner that appears to be in the real world when a user wearing the XR head-mounted device 920 views through the display screen 1302. The display screen 1302 can be positioned in front of the user's eyes by a headband 1306.

[0114] The XR head-mounted device controller 1430 can be within a housing configured to be worn on the user's head or other parts of the user's body when viewing the display screen 1302, or can be positioned away from the user viewing the display screen 1302 when communicatively connected to the display screen 1302. The XR head-mounted device controller 1430 can be configured to operatively process signaling from the camera 1440, microphone 142, and / or pose sensor 1446, and is connected to display XR images on the display device 1450 for viewing by the user on the display screen 1302. Thus, the XR head-mounted device controller 1430 shown as a circuit block within the XR head-mounted device 920 should be understood to be operatively connected to the other shown components of the XR head-mounted device 920, but not necessarily resident within a common housing (e.g., Figure 13 the electronic component housing 1304) or otherwise transportable by the user. For example, the AR head-mounted device controller 1430 can reside within the computer platform 910, which in turn can reside within Figure 3B and Figure 3C the housing of the computer tracking system 6' shown.

[0115] Exemplary XR Head-Mounted Device Component Optical Arrangement

[0116] Figure 15 A block diagram showing an arrangement of optical components of the XR head-mounted device 920 in accordance with some embodiments of the present disclosure is shown. Referring to Figure 15 , the display device 1450 is configured to display XR images generated by the XR head-mounted device controller 1430, and light from the XR images is projected as XR images 1500 onto the display screen 1302. The display screen 1302 is configured to combine the light of the XR images 1500 and the light from the real-world scene 1502 into a combined enhanced view 1504, which is directed to the user's eyes 1510. The display screen 1302 configured in this manner acts as a see-through display screen. The XR head-mounted device 920 can include any number of tracking cameras 1440. The cameras 1440 can be visible light capture cameras, near-infrared capture cameras, or a combination of both.

[0117] Exemplary User Views Through the XR Head-Mounted Device

[0118] The XR head-mounted device operation can display both 2D images and 3D models on the display screen 1302. The 2D images can preferably be displayed in the less opaque band (upper band) of the display screen 1302, and the 3D models can more preferably be displayed in the more transparent band (otherwise referred to as the ambient area) (bottom band) of the display screen 1302. Below the lower band where the display screen 1302 ends, the wearer can view the operating room unobstructed. Note that the position where XR content is displayed on the display screen 1302 can be fluid. The position for displaying 3D content may move to the opaque band depending on the positioning of the head-mounted device relative to the content, and the position for displaying 2D content can be placed in the transparent band and be stable relative to the real world. Additionally, the entire display screen 1302 can be darkened under electrical control to convert the head-mounted device into virtual reality for surgical planning or be completely transparent during a medical procedure. As explained above, the XR head-mounted device 920 and the associated operations not only support the navigated procedure but can also be performed in combination with a robot-assisted procedure.

[0119] Figure 16 An example view of the display screen 1302 of the XR head-mounted device 920 for providing navigation assistance to a user operating a surgical tool 1602 during a medical procedure is shown according to some embodiments of the present disclosure. Refer Figure 16 , when the surgical tool 1602 is brought near the tracked anatomical structure such that the dynamic reference arrays 1630 and 1632 connected to the surgical tool 1602 become within the field of view of the camera 1440 ( Figure 15 ) and / or 46 ( Figure 6 ), the graphical representation 1600 of the tool can be displayed in 2D and / or 3D images relative to the graphical representation 1610 of the anatomical structure. The user can use the viewed graphical representation to adjust the trajectory 1620 of the surgical tool 1602, which can be shown as extending from the graphical representation 2000 of the tool through the graphical representation 1610 of the anatomical structure. The XR head-mounted device 920 can also display text information and other objects 1640. The dashed line 1650 extending across the viewed display screen represents an example division between the upper and lower bands at different opacity levels.

[0120] Other types of XR images (virtual content) that can be displayed on the display screen 1302 can include, but are not limited to, any one or more of the following:

[0121] I) 2D axial, sagittal, and / or coronal views of the patient's anatomy;

[0122] 2) Overlays of planned tool and surgical implant positions on current tracked tool and surgical implant positions;

[0123] 3) Preoperative image libraries;

[0124] 4) Video feeds from microscopes and other similar systems or remote video conferencing;

[0125] 5) Options and configuration settings, and buttons;

[0126] 6) A floating 3D model of a patient's anatomy with surgical planning information;

[0127] 7) Real-time tracking of surgical instruments relative to the floating patient anatomy;

[0128] 8) An augmented overlay of the patient anatomy with instructions and guidance; and

[0129] 9) An augmented overlay of surgical equipment.

[0130] Exemplary configurations of cameras for tracking system components

[0131] Figure 17 An exemplary configuration of an auxiliary tracking bar 46 with two pairs of stereo tracking cameras configured according to some embodiments of the present disclosure is shown. The auxiliary tracking bar 46 is Figure 3A , Figure 3B and Figure 3C part of the camera tracking system components of. According to one embodiment, the stereo tracking cameras include a pair of stereo-spaced visible light capture cameras and another pair of stereo-spaced near-infrared capture cameras. Alternatively, only one pair of visible light capture cameras or only one pair of near-infrared capture cameras can be used in the auxiliary tracking bar 46. Any number of near-infrared and / or visible light cameras can be used.

[0132] Pose measurement chain

[0133] As explained above, navigated surgery can include computer vision tracking and determining the pose of surgical instruments (e.g., position and orientation in a six-degree-of-freedom coordinate system), e.g., by determining the pose of an attached DRA that includes fiducial points such as disks or balls arranged in a known manner in a camera tracking system. Computer vision uses spaced-apart tracking cameras configured to capture near-infrared light and / or visible light, such as stereo cameras. In this context, three parameters compete for optimization: (1) accuracy, (2) stability, and (3) user ergonomics during the surgical procedure.

[0134] Computer operation can be combined with additional tracking cameras mounted to one or more XR head-mounted devices to combine (link) the measured poses in a manner that can improve the optimization of one or more of the above three parameters. As Figure 17As shown, according to some embodiments of the present disclosure, a pair of stereo visible light tracking cameras and another pair of stereo near-infrared tracking cameras can be attached to the auxiliary tracking rod of the camera tracking system assembly. Operating algorithms are disclosed that analyze the pose of the DRA that is fully observed or partially observed (e.g., when not all fiducial points of the DRA are observable by a pair of stereo cameras), and combine the observed poses or partial poses in a manner that can improve accuracy, robustness, and / or ergonomics during navigational surgery.

[0135] As explained above, the XR head-mounted device can be configured to enhance a real-world scene with computer-generated XR images. The XR head-mounted device can be configured to provide an XR viewing environment by displaying computer-generated XR images on a see-through display screen that allows light from the real-world scene to pass through for the user to view in combination. Alternatively, the XR head-mounted device can be configured to provide a VR viewing environment by preventing or substantially preventing light from the real-world scene from being directly viewed by the user along the viewing path of the displayed XR images. The XR head-mounted device can be configured to provide both an AR viewing environment and a VR viewing environment. In one embodiment, both the AR viewing environment and the VR viewing environment are provided by lateral bands with significantly different opacities arranged between the see-through display screen and the real-world scene, such that a VR viewing environment is provided for XR images aligned with the high-opacity band and an AR viewing environment is provided for XR images aligned with the low-opacity band. In another embodiment, both the AR viewing environment and the VR viewing environment are provided by a computer-adjustable controllable transparency filter that variably restricts how much light from the real-world scene passes through the see-through display screen for combination with the XR images viewed by the user. Thus, the XR head-mounted device can also be referred to as an AR head-mounted device or a VR head-mounted device.

[0136] As explained above, an XR head-mounted device may include a near-infrared tracking camera and / or a visible light tracking camera configured to track fiducial points of a DRA connected to a surgical instrument, a patient anatomy, one or more other XR head-mounted devices, and / or a robotic end effector. Using near-infrared tracking and / or visible light tracking on the XR head-mounted device provides additional tracking volume coverage beyond what can be provided by a camera on a single assistive tracking rod. Adding a near-infrared tracking camera to an existing assistive tracking rod allows for more robust but less accurate tracking of the head-mounted device position compared to visible light. Using stereo matching to jointly identify the pose of the DRA fiducial points between the visible tracking coordinate system and the near-infrared tracking coordinate system mechanically calibrates the visible tracking coordinate system and the near-infrared tracking coordinate system so that the coordinate systems are sufficiently aligned to perform 3D DRA fiducial point triangulation operations. Using both the visible and near-infrared tracking coordinate systems can achieve any one or more of the following: (a) identifying tools that cannot be identified using a single coordinate system; (b) improving pose tracking accuracy; (c) achieving a wider range of motion without losing track of the surgical instrument, patient anatomy, and / or robotic end effector; and (d) naturally tracking the XR head-mounted device in the same coordinate system as the navigated surgical instrument.

[0137] Figure 18 A block diagram view of components of a surgical system is shown, the surgical system including tracking cameras in a pair of XR head-mounted devices 920 (head-mounted displays HMD1 and HMD2) and in a camera tracking rod in a camera tracking system component 6' that houses a computer platform 910. The computer platform 910 may include a tracking subsystem 830, a navigation controller 828, and an XR head-mounted device controller 1430, as previously shown in Figure 14 as presented.

[0138] Referring Figure 18 to the surgical system of Figure 13 as presented, a surgeon and an assistant each wear XR head-mounted devices HMD1 920 and HMD2 920, respectively, if each XR head-mounted device includes a tracking camera that may be configured as presented in

[0139] The combination of XR head-mounted devices HMD1 920 and HMD2 920 and the tracking camera 46 on the assistive tracking rod can more robustly track example objects of a patient reference array (R), a robotic end effector (E), and a surgical tool (T) or instrument when operating with the computer platform 910. The overlapping views from different perspectives provided by XR head-mounted devices HMD1 920 and HMD2 920 and the tracking camera 46 on the assistive tracking rod are shown in Figure 12 as presented.

[0140] Figure 18 Each of the marked items represents a unique coordinate system. The descriptions of the coordinate system labels are as follows:

[0141] A = visible light coordinate system of the second head-mounted device HMD2 920;

[0142] N3 = NIR coordinate system of the second head-mounted device HMD2 920;

[0143] S = visible light coordinate system of the main head-mounted device HMD1 920;

[0144] N2 = NIR coordinate system of the main head-mounted device HMD1 920;

[0145] N = NIR coordinate system of the auxiliary navigation bar 46;

[0146] V = visible light coordinate system of the auxiliary navigation bar 46;

[0147] R = NIR coordinate system of the patient reference fiducial array 602;

[0148] T = NIR coordinate system of the tracked tool 604;

[0149] E = NIR coordinate system of the tracked robotic end effector on the robotic arm 20; and

[0150] W = inertial navigation world coordinate system with a stable gravity vector.

[0151] During the manufacturing process, when the device is installed in the operating room and / or before a surgical procedure is to be performed, the spatial relationships (and by extension, the coordinate systems) of some of these marked objects can be measured and calibrated. In the disclosed system, the following coordinate systems are calibrated: where the term "T" is defined as a six-degree-of-freedom (6DOF) homogeneous transformation between two specified coordinate systems. Thus, for example, the term is the 6DOF homogeneous transformation between the visible light coordinate system of the main head-mounted device HMD1 920 and the NIR coordinate system of the main head-mounted device HMD1 920.

[0152] In one embodiment, the XR head-mounted devices HMD1 920 and HMD2 920 have passive visible light fiducial points sprayed or otherwise attached to them (coordinate systems S and A), such as Figure 13 the reference array fiducial points 1310 shown in. The tracking cameras are spatially calibrated to these passive fiducial points (coordinate systems N2 and N3).

[0153] As explained above, the cameras on the XR head-mounted devices HMD1 920 and HMD2 920 and the tracking camera 46 on the auxiliary tracking rod have partially overlapping fields of view. If one or more cameras on the XR head-mounted device HMD1 920 are blocked from viewing the DRA attached to the tracked object, such as the tracked tool (T), but the DRA is within the field of view of the cameras on another XR head-mounted device HMD2 920 and / or the tracking camera 46 on the auxiliary tracking rod, the computer platform 910 can continue to track the DRA seamlessly without losing navigation. Additionally, if the DRA is partially occluded from the perspective of the cameras on the XR head-mounted device HMD1 920 but the entire DRA is visible through the cameras on another XR head-mounted device HMD2 920 and / or the tracking camera 46 on the auxiliary tracking rod, the tracking inputs from the cameras can be combined to continue navigating the DRA.

[0154] More specifically, by independently observing the various camera systems provided by the XR head-mounted devices HMD1 920 and HMD2 920 and the tracking camera 46 on the auxiliary tracking rod, various coordinate systems can be linked together. For example, each of the XR head-mounted devices HMD1 920 and HMD2 920 may require virtual augmentation of the robotic end effector (E). While one XR head-mounted device HMD1 920 (N2) and the tracking camera 46 on the auxiliary tracking rod (N) can see (E), another XR head-mounted device HMD2 920 (N3) may not be able to. The position of (E) relative to (N3) can still be calculated by one of several different methods of operation. The operation according to one embodiment performs the linking of the poses from the patient reference (R). If the patient reference (R) is seen by either (N3) and (N) or (N2), the pose of (E) relative to (N3) can be directly solved by either of the following two equations:

[0155] - or -

[0156] The key to such a pose chain is to infer the relationship between the frames at each end of the chain (circled and transmitted below). The chain can be of any length and is achieved by having more than one stereo camera system (e.g., N, N2, N3).

[0157] A camera tracking system can be configured to receive tracking information related to a tracked object during a surgical procedure from a first tracking camera (e.g., N3) and a second tracking camera (e.g., N2). The camera tracking system can determine a first pose transformation between a first object (e.g., R) coordinate system and a first tracking camera (e.g., N3) coordinate system based on first object tracking information from the first tracking camera (e.g., N3) indicating the pose of the first object (e.g., R). For example,

[0158] )). The camera tracking system can determine a second pose transformation between a first object (e.g., R) coordinate system and a second tracking camera (e.g., N2) coordinate system based on first object tracking information from the second tracking camera (e.g., N2) indicating the pose of the first object (e.g., R). (e.g., ))). The camera tracking system can determine a third pose transformation between a second object (e.g., E) coordinate system and a second tracking camera (e.g., N2) coordinate system based on second object tracking information from the second tracking camera (e.g., N2) indicating the pose of the second object (e.g., E). (e.g., ))). The camera tracking system can determine a fourth pose transformation between a second object (e.g., E) coordinate system and a first tracking camera (e.g., N3) coordinate system based on combining the first, second, and third pose transformations. For example, )).

[0159] In some other embodiments, the camera system can further determine the pose of the second object (e.g., E) with respect to the first tracking camera system (e.g., N3) coordinate system based on processing the tracking information through the fourth pose transformation.

[0160] Due to the overlapping fields of view of various camera systems, when the first camera is blocked from seeing the second object (e.g., E), the camera tracking system is able to determine the pose of the second object (e.g., E) relative to the first tracking camera (e.g., N3). For example, in some embodiments, the camera tracking system is further configured to determine a fourth pose transformation between a second object (e.g., E) coordinate system and a first tracking camera (e.g., N3) coordinate system (e.g., )) without using any tracking information from the first tracking camera (e.g., N3) indicating the pose of the second object (e.g., E).

[0161] A camera tracking system can achieve high tracking accuracy by combining synchronized images from multiple camera systems. For example, the camera tracking system can determine the pose of a second object (e.g., E) relative to a first tracking camera (e.g., N3) by combining synchronized images of the second object (e.g., E) from multiple viewpoints (the first and second tracking cameras), and can use weights that can be determined based on the accuracy specifications of the respective cameras. More specifically, the camera tracking system can be further configured to determine a fourth pose transformation (e.g., ) between the second object (e.g., E) coordinate system and the first tracking camera (e.g., N3) coordinate system based on second object tracking information indicating the pose of the second object (e.g., E) from the first tracking camera (e.g., N3) and further based on the combined result of the first, second, and third pose transformations.

[0162] The surgical system can be configured to display an XR image having a pose determined based on the fourth pose transformation on the perspective display screen of the XR headset. The camera tracking system can be further configured to generate an XR image that is a graphical representation of the second object (e.g., E) placed on the perspective display screen by processing the fourth pose transformation of the first object tracking information from the first and second tracking cameras and the second object tracking information from the second tracking camera.

[0163] As explained above, the camera tracking system can include a navigation controller 828 that is communicatively connected to the first tracking camera (e.g., N3) and the second tracking camera (e.g., N2) to receive tracking information and is configured to determine the first, second, third, and fourth pose transformations.

[0164] Five camera tracking rods for surgical navigation

[0165] Although there are different stereo camera tracking rods or even tracking rods with three cameras (also called navigation rods) available today, the camera tracking rod described herein can include five cameras that are designed, configured, calibrated, and used in such a way that the camera tracking rod can provide improved object tracking for computer-aided navigation during surgery.

[0166] This includes a high-rate inertial measurement unit (3-axis gyroscope and 3-axis accelerometer), and also allows for other functions to be provided at the current state of the art.

[0167] According to some embodiments, the camera tracking arm uses ambient visible light for instrument, end effector, and direct patient anatomy tracking. The near-infrared (NIR) illuminator circuit is also included in at least one of the five cameras capable of detecting NIR illumination. At very low image sensor exposures, the illumination can assist in tracking in dark environments or differentiating retroreflective surfaces from normal surfaces. Additional cameras (beyond just two stereo cameras) increase the volume in which object tracking can be accomplished. This reduces the impact of the problems of current systems where small tracking volumes force surgical personnel to frequently reposition the camera tracking arm and / or result in loss of object tracking and unexpected interruption of navigated surgical procedures.

[0168] Since image sensors require a high dynamic range, it can be difficult to track visible light from ambient light. Briefly, there is a large difference in illumination between the center of a bright or spotlight and the surrounding area. Having multiple stereo pairs allows for simultaneous optical marker detection and tracking at different exposures. Including more than two stereo cameras can also eliminate the ambiguity of point triangulation, which effectively means there is less "interference".

[0169] Figure 19 Some embodiments showing the main components of the disclosed camera tracking arm are shown.

[0170] In various embodiments of the present disclosure, the camera tracking arm includes a first set of stereo tracking cameras 1900, a second set of tracking cameras 1910, and a communication interface 1970, as Figure 19 shown.

[0171] The first set of stereo tracking cameras 1900 are the outermost stereo cameras. The resolution of the first set of stereo tracking cameras 1900, the field of view of the lenses, the baseline 1905 (the spacing between the first set of stereo tracking cameras), and the mounting angles are all adjusted for the application because each affects accuracy, acuity, and the supported stereo tracking volume. A wider baseline enables better triangulation and depth perception accuracy, but for objects that are farther away, the greater the distance between the cameras, the greater the error.

[0172] In some embodiments, the camera tracking arm further includes a second set of tracking cameras 1910 located inside the original pair. These cameras can have a wider field of view and be angled inwards to detect tracking objects outside the normal tracking area. The resolution of the second set of stereo tracking cameras 1910, the field of view of the lenses, the baseline 1915 (the spacing between the first set of stereo tracking cameras), and the mounting angles are all adjusted for the application because each affects accuracy, acuity, and the supported stereo tracking volume.

[0173] In some embodiments, the camera tracking rod of a camera tracking system for computer - assisted navigation during surgery includes a first set of stereo tracking cameras 1900, which have a first resolution, a first field of view, and are spaced apart on the camera tracking rod by a first baseline distance 1905. The camera tracking rod further includes a second set of stereo tracking cameras 1910, which have a second resolution, a second field of view, and are spaced apart on the camera tracking rod by a second baseline distance 1915 that is less than the first baseline distance 1905. The second set of stereo tracking cameras 1910 is positioned between the first set of stereo tracking cameras 1900, and the resolution and / or field of view of the second set of stereo tracking cameras 1910 is different from the resolution and / or field of view of the first set of stereo tracking cameras 1900. The camera tracking rod further includes a communication interface 1970, which is configured to provide video streams from the first set of stereo tracking cameras 1900 and the second set of stereo tracking cameras 1910 to the camera tracking subsystem.

[0174] In some embodiments, each camera in the first set of stereo tracking cameras 1900 is attached to the surface of the camera tracking rod and is inclined inwardly towards each other at a first angle, and each camera in the second set of stereo tracking cameras 1910 is attached to the surface of the camera tracking rod and is inclined inwardly towards each other at a second angle different from the first angle.

[0175] In some embodiments, a color camera 1920 or a set of color cameras may be included for recording or training purposes. The color camera included herein is well - calibrated with the other four cameras, so that color information can be added to the tracked object to improve the recognition and differentiation of the configuration of the tracked object.

[0176] In some embodiments, the camera tracking rod may further include a high - rate inertial measurement unit (IMU) 1940. The IMU 1940 allows for the automatic differentiation of the motion of the camera and the tracked object. The IMU measures the angular velocity and linear acceleration due to motion and gravity. This IMU can not only be used to differentiate the motion of the camera and the tracked object, but also provide an accurate estimate of the gravity of the camera and all the tracked instruments. The IMU 1940 may include a 3 - axis gyroscope and a 3 - axis accelerometer. The IMU 1940 is configured to output motion data indicating the measured motion of the camera tracking rod, and the communication interface is further configured to provide the motion data to the camera tracking subsystem. The inertial measurement unit may be configured to include in the motion data the measurement of the angular velocity of the 3 - axis gyroscope and the measurement of the linear acceleration of the 3 - axis accelerometer. The camera tracking rod may further include a processor, which is configured to align the orientation of the video stream and the motion data measurements with a gravity reference.

[0177] In some embodiments, the camera tracking pole may further include a microphone array 1950 that includes a plurality of microphones spaced along the camera tracking pole and connected to provide at least one audio stream to a computer platform. The computer platform is configured to perform at least one of the following: record at least one audio stream, identify voice commands included in the at least one audio stream, and selectively trigger defined operations associated with the identified voice commands in the voice commands, measure the ambient noise level included in the at least one audio stream, and triangulate the location of a sound source included in at least two audio streams provided by the plurality of microphones.

[0178] In some embodiments, the microphone array 1950 has been incorporated into the tracking pole to record audio for training, trigger voice commands, measure the ambient noise level, and triangulate the source of the sound. The microphone array 1950 can also be used to collect data for deep learning. Background conversations and equipment noises such as high-speed drills or saws can be measured, detected, and timestamped relative to the overall situation.

[0179] The camera tracking pole can be used to track a disk or a retroreflective sphere on a dynamic reference array. The resolution, field of view, baseline distance, and mounting angle of a single tracking camera are all optimized to detect a disk or a retroreflective sphere with an ideal pixel density, such as a pixel array of at least 10 pixels by 10 pixels and no more than 30 pixels by 30 pixels.

[0180] In some embodiments, the first resolution and the first field of view of the first set of stereo tracking cameras 1900 are configured to output frames in a video stream that image a reference array positioned during surgery, the reference array extending at least 10 pixels by 10 pixels and no more than 30 pixels by 30 pixels.

[0181] In some other embodiments, the second resolution and the second field of view of the second set of stereo tracking cameras 1910 are configured to output frames in a video stream that image a reference array, the reference array extending at least 10 pixels by 10 pixels and no more than 30 pixels by 30 pixels.

[0182] In some other embodiments, the first resolution is 7 megapixels, the first field of view is 50 degrees, the first baseline distance 1905 is 0.5 meters, and the second resolution is 1.5 megapixels, the second field of view is 85 degrees, and the second baseline distance 1915 is 0.4 meters.

[0183] In some embodiments, the camera tracking pole further comprises: a set of near-infrared illuminators positioned around each of the cameras in the first set of stereo tracking cameras, and at least one processor operatively configured to turn on the illuminators when an ambient light level detected by at least one of the cameras in the first set of stereo tracking cameras is below an ambient light threshold, and in response to turning on the near-infrared illuminators, stop processing video from the first set of stereo tracking cameras using values from a visible light camera calibration file and start processing video from the first set of stereo tracking cameras using values from a near-infrared light camera calibration file.

[0184] In some embodiments, the camera tracking pole further comprises: a set of illuminators that primarily emit one color positioned around each of the cameras in the first set of stereo tracking cameras; and at least one processor operatively configured to turn on the illuminators when an ambient light level detected by at least one of the cameras in the first set of stereo tracking cameras is below an ambient light threshold, and in response to turning on the near-infrared illuminators, stop processing video from the first set of stereo tracking cameras using values from an ambient light camera calibration file and start processing video from the first set of stereo tracking cameras using values from a color light camera calibration file.

[0185] In some embodiments, the first set of stereo tracking cameras is configured to have a different sensor exposure speed than the second set of stereo tracking cameras. Additionally, for each pair of tracking cameras 1900 and camera 1910, the exposure of the sensors can be different, thereby increasing the likelihood of detecting everything present and increasing the robustness to shadows and other ambient light aberrations.

[0186] In some embodiments, video frames output by the first set of stereo tracking cameras and the second set of stereo tracking cameras are synchronized by a common synchronization signal output by a floating-point gate array. All of the tracking cameras 1900, 1910, and 1920 are synchronized 1960 via a floating-point gate array FPGA, such that data from different sensors is aligned in time. This synchronization signal is output from the navigation pole so that other external sensors can capture information corresponding to the same moment.

[0187] Figure 20 and Figure 21 Illustrates a tracking volume generated by the first set of stereo tracking cameras 1900. In some embodiments, the first set of stereo tracking cameras 1900 is configured to provide a first tracking volume having a first near-field three-dimensional volume pyramid that is 413 millimeters wide by 427 millimeters high and a first far-field three-dimensional volume pyramid that is 1619 millimeters wide by 1463 millimeters high.

[0188] Figure 22 and Figure 23 shows the tracking volume generated by the second set of stereo tracking cameras 1910. The field of view and angle of the lens can be adjusted to adjust the volume and accuracy. In this exemplary embodiment, the tracking accuracy of the internal cameras themselves is greatly reduced, but they cover a larger tracking volume and greatly improve the detection rate and overall awareness. It should be noted that the internal pairs are capable of tracking more closely and have a larger volume within the range. In some embodiments, the second set of stereo tracking cameras 1910 is configured to provide a second tracking volume having a second near-field three-dimensional volume pyramid that is 585 millimeters wide by 1020 millimeters high and a second far-field three-dimensional volume pyramid that is 2227 millimeters wide and 4589 millimeters high.

[0189] Figure 24 and Figure 25 shows the simulated triangulation accuracy of the first set of stereo tracking cameras and the second set of stereo tracking cameras. For each pair of the two stereo pairs, the simulated tracking accuracy is shown as a function of the position of the tracked object. The experimental results have proven to be very close to this predicted triangulation accuracy.

[0190] Figure 26 shows a dual-pass filter applied to the video frames output by the first set of stereo tracking cameras 1900 and the second set of stereo tracking cameras 1910. The dual-pass filter can enable the camera tracking system to perform narrow infrared, NIR, and visible light tracking. In some embodiments, the first set of stereo tracking cameras includes a dual-pass filter configured to pass a narrow wavelength band in visible light and another narrow wavelength band in near-infrared light. The camera tracking subsystem is configured to track the pose of retroreflective fiducial points of a reference array in the video stream that captures a narrow wavelength band in visible light and another narrow wavelength band in near-infrared light.

[0191] Referring Figure 26 , NIR illumination (e.g., 850 nanometers) is included in one of the stereo camera pairs for tracking under low ambient light conditions while also allowing visible light machine vision applications. As Figure 26 shown, this is achieved by passing light of both wavelengths.

[0192] Figure 27 shows a retroreflective tracking array having retroreflective spheres thereon. Figure 26 's dual-bandpass filter allows tracking of retroreflective objects without a clear visible light boundary, such as Figure 27The sphere shown in []. The pure NIR tracker only allows the NIR spectrum that matches the illumination to pass through to reduce light pollution, thereby effectively tracking black images with white NIR LEDs or reflective objects. Allowing the camera through the visible light spectrum is the preferred method for imaging not only the LED and reflective objects but also for tracking under bright illumination conditions. By reducing the camera's exposure, the imaging of retroreflective spheres without clear edges can be improved.

[0193] Figure 28 A flowchart of operations performed by a camera tracking subsystem is shown and is discussed in further detail in the following embodiments.

[0194] In various other embodiments, a camera tracking system for computer - assisted navigation during surgery includes a camera tracking wand. The camera tracking wand includes a first set of stereo tracking cameras having a first resolution, a first field of view, and spaced apart on the camera tracking wand at a first baseline distance. The camera tracking wand further includes a second set of stereo tracking cameras having a second resolution, a second field of view and spaced apart on the camera tracking wand at a second baseline distance less than the first baseline distance. The second set of stereo tracking cameras is positioned between the first set of stereo tracking cameras, and the resolution and / or field of view of the second set of stereo tracking cameras is different from the resolution and / or field of view of the first set of stereo tracking cameras. The camera tracking wand further includes a camera tracking subsystem configured to determine the pose 2800 of fiducial points of a reference array in video streams from the first set of stereo tracking cameras and the second set of stereo tracking cameras.

[0195] In some of the embodiments, the camera tracking subsystem is further configured to compensate 2802 for movement of the camera tracking wand indicated by motion data when determining the pose of fiducial points of the reference array.

[0196] In some embodiments, the first resolution and first field of view of the first set of stereo tracking cameras are configured to output frames in the video stream that image a reference array positioned during surgery, the reference array extending over at least 10 pixels by 10 pixels and no more than 30 pixels by 30 pixels. The second resolution and second field of view of the second set of stereo tracking cameras are configured to output frames in the video stream that image a reference array extending over at least 10 pixels by 10 pixels and no more than 30 pixels by 30 pixels.

[0197] Figure 29 A flowchart of operations performed by a processor of the camera tracking wand is shown and is discussed in further detail in the following embodiments.

[0198] In some embodiments, the camera tracking pole further includes: a set of near-infrared illuminators positioned around each of the cameras in the first set of stereo tracking cameras, and at least one processor operatively configured to turn on the illuminators when an ambient light level detected by at least one of the cameras in the first set of stereo tracking cameras is below an ambient light threshold, and in response to turning on the near-infrared illuminators, stop processing video from the first set of stereo tracking cameras using values from a visible light camera calibration file and begin processing video from the first set of stereo tracking cameras using values from a near-infrared light camera calibration file.

[0199] In some embodiments, the first set of stereo tracking cameras includes a dichroic filter configured to pass a narrow wavelength band in visible light and another narrow wavelength band in near-infrared light. The camera tracking subsystem is configured to track the pose of retroreflective fiducial points of a reference array in a video stream that captures the narrow wavelength band in visible light and the other narrow wavelength band in near-infrared light.

[0200] In some embodiments, the camera tracking pole further includes: a set of illuminators that primarily emit one color positioned around each of the cameras in the first set of stereo tracking cameras; and at least one processor operatively configured to turn on the illuminators when an ambient light level detected by at least one of the cameras in the first set of stereo tracking cameras is below an ambient light threshold, and in response to turning on the near-infrared illuminators, stop processing video from the first set of stereo tracking cameras using values from an ambient light camera calibration file and begin processing video from the first set of stereo tracking cameras using values from a color light camera calibration file.

[0201] Additional definitions and embodiments:

[0202] In the foregoing description of the various embodiments of the inventive concept, it should be understood that the terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the inventive concept. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the inventive concept pertains. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the present specification and the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0203] When an element is referred to as being "connected to" or "coupled to" or "responsive to" another element or a variation thereof, it can be directly connected to, coupled to, or responsive to the other element, or intervening elements may be present. In contrast, when an element is referred to as being "directly connected to", "directly coupled to", "directly responsive to" another element or a variation thereof, no intervening elements are present. Like reference numerals refer to like elements throughout. Additionally, as used herein, "coupling", "connecting", "responding", or variations thereof may include wireless coupling, connection, or response. Unless the context clearly dictates otherwise, as used herein, the singular forms "a", "an", and "the" are intended to include the plural forms. Well-known functions or constructions may not be described in detail for the sake of brevity and / or clarity. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0204] It should be understood that although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Thus, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments without departing from the teachings of the inventive concept. Throughout this specification, the same reference numerals or the same reference indicators denote the same or similar elements.

[0205] As used herein, the terms "comprise", "comprising", "includes", "include", "including", "have", "has", "having", or variations thereof are open-ended and include one or more stated features, integers, elements, steps, components, or functions, but do not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or combinations thereof. Additionally, as used herein, the common abbreviation "e.g.", derived from the Latin phrase "exempli gratia", may be used to introduce or specify one or more general examples of the previously mentioned items and is not intended to limit such items. The common abbreviation "i.e.", derived from the Latin phrase "id est", may be used to specify a particular item from a more general statement.

[0206] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices), and / or computer program products. It should be understood that the blocks shown in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions executed by one or more computer circuits. These computer program instructions can be provided to a processor circuit of a general purpose computer circuit, a special purpose computer circuit, and / or other programmable data processing circuit to produce a machine, such that the instructions executed by the processor of the computer and / or other programmable data processing device transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions / actions specified in the block diagrams and / or one or more flowchart blocks, and thereby create a means (functionality) and / or structure for implementing the functions / actions specified in the block diagrams and / or one or more flowchart blocks.

[0207] These tangible computer program instructions can also be stored in a computer-readable medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture that includes instructions implementing the functions / actions specified in the block diagrams and / or one or more flowchart blocks. Accordingly, embodiments of the inventive concept can be embodied in hardware and / or software (including firmware, resident software, microcode, etc.) that runs on a processor such as a digital signal processor, which may be collectively referred to as "circuitry", "module", or variations thereof.

[0208] It should also be noted that in some alternative implementations, the functions / actions noted in the blocks may not occur in the order noted in the flowchart. For example, depending on the functions / actions involved, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order. Additionally, the functionality of a given block of the flowchart and / or block diagram can be split into multiple blocks, and / or the functionality of two or more blocks of the flowchart and / or block diagram can be at least partially integrated. Finally, other blocks can be added / inserted between the blocks shown, and / or blocks / operations can be omitted, without departing from the scope of the inventive concept. Moreover, although some of the figures in the drawings include arrows on communication paths to indicate a primary direction of communication, it should be understood that communication can occur in a direction opposite to that depicted by the arrows.

[0209] Many variations and modifications can be made to the embodiments without materially departing from the principles of the inventive concept. All such variations and alterations are intended to be included within the scope of the inventive concept herein. Thus, the subject matter disclosed above should be regarded as illustrative and not restrictive, and the examples of the appended embodiments are intended to cover all such modifications, enhancements, and other embodiments that fall within the spirit and scope of the inventive concept. Accordingly, to the extent legally permitted to the maximum extent, the scope of the inventive concept will be determined by the broadest permissible interpretation of this disclosure, which includes the examples of the following embodiments and their equivalents, and should not be limited or restricted to the foregoing specific detailed description.

Claims

1. A camera tracking rod for a camera tracking system for computer - assisted navigation during surgery, the camera tracking rod comprises: A first set of stereo tracking cameras, the first set of stereo tracking cameras having a first resolution, a first field of view, and being spaced apart on the camera tracking rod by a first baseline distance; A second set of stereo tracking cameras, the second set of stereo tracking cameras having a second resolution, a second field of view, and being spaced apart on the camera tracking rod by a second baseline distance less than the first baseline distance, wherein the second set of stereo tracking cameras is positioned between the first set of stereo tracking cameras, and the resolution of the second set of stereo tracking cameras is different from the resolution of the first set of stereo tracking cameras and the field of view of the second set of stereo tracking cameras is different from the field of view of the first set of stereo tracking cameras; and A communication interface configured to provide video streams from the first set of stereo tracking cameras and the second set of stereo tracking cameras to a camera tracking subsystem, wherein: Each camera in the first set of stereo tracking cameras is attached to the surface of the camera tracking rod and is tilted inwardly towards each other at a first angle; Each camera in the second set of stereo tracking cameras is attached to the surface of the camera tracking rod and is tilted inwardly towards each other at a second angle different from the first angle; The first set of stereo tracking cameras has a narrower field of view than the second set of stereo tracking cameras; and The resolution of the first set of stereo tracking cameras is higher than that of the second set of stereo tracking cameras.

2. The camera tracking rod according to claim 1, further comprises: A high - rate inertial measurement unit, the high - rate inertial measurement unit including a three - axis gyroscope and a three - axis accelerometer, wherein the inertial measurement unit is configured to output motion data indicative of the measured motion of the camera tracking rod, and the communication interface is further configured to provide the motion data to the camera tracking subsystem.

3. The camera tracking rod according to claim 2, wherein, The inertial measurement unit is configured to include in the motion data a measurement of the angular velocity of the three - axis gyroscope and a measurement of the linear acceleration of the three - axis accelerometer.

4. The camera tracking rod according to claim 3, further comprising a processor configured to align the orientation of the video stream and the motion data measurements with a gravity reference.

5. The camera tracking rod according to claim 1, further comprising a microphone array including a plurality of microphones spaced apart along the camera tracking rod and connected to provide at least one audio stream to a computer platform, wherein, The computer platform is configured to perform at least one of the following: Record the at least one audio stream; Identify voice commands included in the at least one audio stream and selectively trigger defined operations associated with the identified voice commands; Measure the ambient noise level included in the at least one audio stream; and Triangulation involves determining the location of a sound source in at least two audio streams provided by the plurality of microphones described above.

6. The camera tracking pole according to claim 1, wherein, the first resolution and the first field of view of the first set of stereo tracking cameras are configured to output, in the video stream, frames that image a reference array placed during the surgery, the reference array extending over a pixel array that is at least 10 pixels by 10 pixels and no more than 30 pixels by 30 pixels.

7. The camera tracking pole according to claim 6, wherein, the second resolution and the second field of view of the second set of stereo tracking cameras are configured to output, in the video stream, frames that image the reference array extending over a pixel array that is at least 10 pixels by 10 pixels and no more than 30 pixels by 30 pixels.

8. The camera tracking pole according to claim 7, wherein: the first resolution is 7 megapixels, the first field of view is 50 degrees, and the first baseline distance is 0.5 meters; and the second resolution is 1.5 megapixels, the second field of view is 85 degrees, and the second baseline distance is 0.4 meters.

9. The camera tracking pole according to claim 1, wherein: the first set of stereo tracking cameras is configured to provide a first tracking volume, the first tracking volume having a first near - field three - dimensional volume pyramid that is 413 millimeters wide by 427 millimeters high and a first far - field three - dimensional volume pyramid that is 1619 millimeters wide by 1463 millimeters high; and the second set of stereo tracking cameras is configured to provide a second tracking volume, the second tracking volume having a second near - field three - dimensional volume pyramid that is 585 millimeters wide by 1020 millimeters high and a second far - field three - dimensional volume pyramid that is 2227 millimeters wide and 4589 millimeters high.

Images