Object tracking and synthetic image registration in visible light surgery via fiducial seeds

Abstract

To provide a camera tracking system for computer assisted navigation during surgery.SOLUTION: The camera tracking system includes a camera bar, first and second tracking cameras, and a third tracking camera. The first and second tracking cameras are attached to spaced locations on the camera bar. The third tracking camera is attached to a location on the camera bar that is between locations of the first and second tracking cameras and at a distance from a line extending through centers of the first and second tracking cameras.SELECTED DRAWING: Figure 23

Description

[Technical field]

[0001] The present disclosure relates to medical devices and systems, and more particularly to camera tracking systems used for computer-assisted navigation during surgery. [Background technology]

[0002] Computer-assisted navigation during surgery can provide the surgeon with a computerized visualization of the current pose of the surgical tool relative to the medical image of the patient's anatomy. A camera tracking system for computer-assisted navigation uses one or more stereo camera systems to track a set of fiducials attached to the surgical tool as the surgeon or other user positions it during surgery. The set of fiducials, also referred to as a dynamic reference array, allows the camera tracking system to determine the pose of the surgical tool relative to the anatomy and patient in the medical image for display to the surgeon. This allows the surgeon to navigate the surgical tool during the surgical procedure using real-time pose feedback.

[0003] Navigated surgical procedures using existing navigation systems are prone to events that cause intermittent pauses when tracked objects move outside the tracking area of ​​the camera system or are blocked from the camera view by intervening personnel and / or equipment, and there is a need to improve the tracking accuracy of navigation systems. Summary of the Invention

[0004] Some embodiments are directed to and provide a camera tracking system for computer-assisted navigation during surgery. The camera tracking system can include a camera bar, first and second tracking cameras, and a third tracking camera. The first and second tracking cameras can be mounted at spaced apart locations on the camera bar. The third tracking camera can be mounted at a location on the camera bar that is between the locations of the first and second tracking cameras and spaced apart a distance from a line that extends through the centers of the first and second tracking cameras.

[0005] Some other embodiments are directed to a surgical tool tracking array. The surgical tool tracking array can include a multi-planar reference array, a plurality of different optically encoded patterns, and a tool holder. The multi-planar reference array can have a plurality of planes. The plurality of different optically encoded patterns can each be on a different one of the planes. The tool holder can be configured to couple a portion of a surgical tool to the surgical tool tracking array.

[0006] Yet some other embodiments relate to a surgical system that includes a camera tracking system, a surgical robot, and at least one controller.

[0007] The camera tracking system may include a camera bar, first and second tracking cameras, a third tracking camera, and at least one processor. The first and second tracking cameras may be mounted at spaced apart locations on the camera bar. The third tracking camera may be mounted at a location on the camera bar that is between the locations of the first and second tracking cameras and spaced apart a distance from a line extending through the centers of the first and second tracking cameras. The at least one processor may be operable to determine a pose of a set of physical objects in three-dimensional space based on a triangulation of positions of objects imaged in the video stream to the locations of the first, second, and third tracking cameras that are spaced apart in three-dimensional space from the first, second, and third tracking cameras.

[0008] The surgical robot can include a robot base, a robot arm, and at least one motor. The robot arm can be connected to the robot base and can be configured to connect to an end effector that guides the movement of a surgical tool. The at least one motor can be operably connected to move the robot arm relative to the robot base.

[0009] At least one controller may be connected to the at least one motor and configured to determine a pose of the end effector relative to a planned pose of the end effector while guiding movement of the surgical tool to operate on the patient. Further, the at least one controller may be configured for generating steering information based on a comparison of the planned pose and the determined pose of the end effector, where the steering information indicates where the end effector needs to be moved to align with the planned pose such that the surgical tool is guided along the planned trajectory by the end effector towards the patient.

[0010] Other surgical systems, camera tracking systems, and surgical tool tracking arrays according to embodiments will become apparent to one of ordinary skill in the art upon review of the following figures and detailed description. All such surgical systems, camera tracking systems, and surgical tool tracking arrays are intended to be included within the scope of this detailed description, be within the scope of this disclosure, and be protected by the accompanying claims. Furthermore, it is intended that all embodiments disclosed herein may be implemented separately or combined in any manner and / or combination.

[0011] The accompanying drawings, which are included to provide a further understanding of the disclosure, and which are incorporated in and constitute a part of this application, illustrate certain non-limiting embodiments of the inventive concepts. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an embodiment of a surgical system according to some embodiments of the present disclosure. [Diagram 2] FIG. 2 illustrates surgical robot components of the surgical system of FIG. 1 according to some embodiments of the present disclosure. [Figure 3A] FIG. 3A illustrates camera tracking system components of the surgical system of FIG. 1 according to some embodiments of the present disclosure. [Figure 3B] FIG. 3B shows a front view of another camera tracking system component that may be used with the surgical system of FIG. 1 according to some embodiments of the present disclosure. [Figure 3C] FIG. 3C shows an isometric view of another camera tracking system component that may be used with the surgical system of FIG. 1 according to some embodiments of the present disclosure. [Figure 4] FIG. 4 illustrates an embodiment of an end effector connectable to a robotic arm and configured in accordance with some embodiments of the present disclosure. [Diagram 5] FIG. 5 illustrates a medical procedure in which a surgical robot and camera system are disposed around a patient. [Figure 6] FIG. 6 shows a block diagram view of the components of the surgical system of FIG. 5 used in a medical procedure. [Figure 7] FIG. 7 illustrates various display screens that may be displayed on the displays of FIGS. 5 and 6 when using the navigation features of the surgical system. [Figure 8] FIG. 8 shows a block diagram of some electrical components of a surgical robot according to some embodiments of the present disclosure. [Figure 9] FIG. 9 shows a block diagram of components of a surgical system including a camera tracking system and / or an imaging device connected to a computer platform that may be operatively connected to a surgical robot according to some embodiments of the present disclosure. [Figure 10]FIG. 10 illustrates an embodiment of a C-arm imaging device that can be used in combination with a surgical robot according to some embodiments of the present disclosure. [Figure 11] FIG. 11 illustrates an embodiment of an O-arm imaging device that can be used in combination with a surgical robot according to some embodiments of the present disclosure. [Figure 12] FIG. 12 shows a block diagram view of components of a surgical system including a pair of XR headsets and an auxiliary tracking bar that operates in accordance with some embodiments of the present disclosure. [Figure 13] FIG. 13 illustrates an XR headset configured in accordance with some embodiments of the present disclosure. [Figure 14] FIG. 14 illustrates electrical components of an XR headset that can be operatively connected to a computer platform, an imaging device, and / or a surgical robot according to some embodiments of the present disclosure. [Figure 15] FIG. 15 shows a block diagram illustrating the arrangement of optical components of an XR headset according to some embodiments of the present disclosure. [Figure 16] FIG. 16 shows an exemplary view through a display screen of an XR headset for providing navigational assistance for manipulating a surgical tool during a medical procedure, according to some embodiments of the present disclosure. [Figure 17] FIG. 17 illustrates an example configuration of an auxiliary tracking bar having two pairs of stereo cameras configured in accordance with some embodiments of the present disclosure. [Figure 18] FIG. 18 shows a block diagram view of components of a surgical system including a tracking camera in a pair of XR headsets and an auxiliary tracking bar that collectively operate in accordance with some embodiments of the present disclosure. [Figure 19] FIG. 19 shows an exemplary configuration of a surgical tool tracking array comprising a camera bar having three stereo tracking cameras configured in accordance with certain embodiments of the present disclosure. [Figure 20a]FIG. 20a shows single-point stereo camera triangulation using two tracking cameras. [Figure 20b] FIG. 20b illustrates single-point triangulation using three tracking cameras according to some embodiments of the present disclosure. [Figure 21] FIG. 21 shows an example of a surgical tool attached to a physical reference array with a fiducial disk, according to some embodiments of the present disclosure. [Figure 22] FIG. 22 illustrates an exemplary side view of a physical reference array attached to the surgical tool of FIG. 21 according to some embodiments of the present disclosure. [Figure 23] FIG. 23 shows an example of a surgical tool tracking array including a multi-planar reference array according to some embodiments of the present disclosure. [Figure 24] FIG. 24 illustrates an example of 2D image registration between undistorted image information and synthetic image information generated from a 3D model of a tool, according to some embodiments of the present disclosure. [Diagram 25] FIG. 25 illustrates an example of an end effector having a reference disk, according to some embodiments of the present disclosure. [Figure 26] FIG. 26 shows a flowchart of operations performed by a camera tracking system according to some embodiments of the present disclosure. [Figure 27] FIG. 27 shows another flowchart of operations performed by a camera tracking system according to some embodiments of the present disclosure. [Figure 28] FIG. 28 shows another flowchart of operations performed by a camera tracking system according to some embodiments of the present disclosure. [Figure 29] 1 illustrates another flowchart of operations performed by a camera tracking system according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The inventive concepts will now be described more fully below with reference to the accompanying drawings, in which examples of embodiments of the inventive concepts are shown. However, the inventive concepts may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the various inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Elements from one embodiment may be implicitly assumed to be present or used in another embodiment.

[0014] Various embodiments disclosed herein are directed to improving computer-assisted navigation during surgery. An augmented reality (XR) headset is operatively connected to a surgical system and configured to provide an interactive environment in which a surgeon, assistants, and / or other personnel can view and select from patient images, view and select from computer-generated surgical navigation information, and / or control surgical instruments in an operating room. As described below, the XR headset can be configured to augment a real-world scene with computer-generated XR images. The XR headset can be configured to provide an augmented reality (AR) display environment by displaying the computer-generated XR images on a see-through display screen through which light from the real-world scene can pass and be viewed by the user in combination. Alternatively, the XR headset can be configured to provide a virtual reality (VR) display environment by preventing or substantially preventing the user from directly viewing light from the real-world scene while the user is viewing the computer-generated AR images on the display screen. The XR headset can be configured to provide both AR and VR display environments. In one embodiment, both the AR and VR display environments are provided by lateral bands of effectively different opacity disposed between the see-through display screen and the real-world scene, such that the VR display environment is provided in the XR image aligned with the high opacity band, and the AR display environment is provided in the XR image aligned with the low opacity band. In another embodiment, both the AR and VR display environments are provided by computer-adjustable control of an opacity filter that variably suppresses how much light from the real-world scene passes through the see-through display screen to be combined with the XR image seen by the user. Thus, the XR headset can also be referred to as an AR headset or a VR headset.

[0015] FIG. 1 illustrates an embodiment of a surgical system 2 according to some embodiments of the present disclosure. Prior to the performance of an orthopedic or other surgical procedure, a three-dimensional ("3D") image scan of the patient's planned surgical area can be taken, for example, using the C-arm imaging device 104 of FIG. 10 or the O-arm imaging device 106 of FIG. 11, or from another medical imaging device such as a computed tomography (CT) image or an MRI. This scan can be taken preoperatively (e.g., most commonly several weeks before the procedure) or intraoperatively. However, any known 3D or 2D image scan can be used according to various embodiments of the surgical system 2. The image scan is transmitted to a computer platform in communication with the surgical system 2, such as computer platform 910 of a surgical system 900 (FIG. 9), which can include a camera tracking system component 6, a surgical robot 4 (e.g., robot 2 of FIG. 1), imaging devices (e.g., C-arm 104, O-arm 106, etc.), and an image database 950 for storing image scans of the patient. A surgeon reviewing the image scan(s) on a display device of the computer platform 910 (FIG. 9) generates a surgical plan that defines target poses for surgical tools to be used during a surgical procedure on the patient's anatomy. Exemplary surgical tools, also referred to as tools, may include, but are not limited to, drills, screwdrivers, saws, retractors, and implants, such as screws, spacers, interbody fusion devices, plates, rods, etc. In some embodiments, the surgical plan, which defines the target planes, is planned with the 3D image scan displayed on the display device.

[0016] As used herein, the term "pose" refers to a position and / or rotational angle of an object (e.g., a dynamic reference array, an end effector, a surgical tool, an anatomical structure, etc.) relative to another object and / or relative to a defined coordinate system. Thus, a pose may be defined based solely on a multi-dimensional position of an object relative to another object and / or relative to a defined coordinate system, based solely on a multi-dimensional rotational angle of an object relative to another object and / or relative to a defined coordinate system, or based on a combination of a multi-dimensional position and a multi-dimensional rotational angle. Thus, the term "pose" is used to refer to a position, a rotational angle, or a combination thereof.

[0017] 1 can assist a surgeon during a medical procedure, for example, by holding tools, aligning tools, using tools, guiding tools, and / or positioning tools for use. In some embodiments, the surgical system 2 includes a surgical robot 4 and a camera tracking system component 6. The ability to mechanically couple the surgical robot 4 and the camera tracking system component 6 allows the surgical system 2 to be operated and moved as a single unit, allowing the surgical system 2 to have a small footprint in area, allowing easier movement through narrow passageways and corners, as well as storage in a smaller area.

[0018] The surgical procedure can begin by moving the surgical system 2 from the medical storage to the medical procedure room. The surgical system 2 can be moved through doorways, halls, and elevators to reach the medical procedure room. Within the medical procedure room, the surgical system 2 can be physically separated into two separate and distinct systems, the surgical robot 4 and the camera tracking system component 6. The surgical robot 4 can be positioned adjacent to the patient in any suitable location to properly assist the medical personnel. The camera tracking system component 6 can be positioned at the base of the patient, on the patient's shoulder, or any other location suitable for tracking the current pose and pose movement of the surgical robot 4 and the tracking portion of the patient. The surgical robot 4 and the camera tracking system component 6 can be powered by an on-board power supply and / or plugged into an external wall outlet.

[0019] The surgical robot 4 can be used to assist the surgeon by holding and / or using tools during a medical procedure. To properly utilize and hold tools, the surgical robot 4 may rely on multiple motors, computers, and / or actuators to function properly. The robot body 8 shown in FIG. 1 can serve as a structure to which multiple motors, computers, and / or actuators can be secured within the surgical robot 4. The robot body 8 can also provide support for the robot telescoping support 16. The size of the robot body 8 can provide a sturdy platform to support attached components and can house, conceal, and protect multiple motors, computers, and / or actuators that can work the attached components.

[0020] The robot base 10 can act as a support for the bottom of the surgical robot 4. In some embodiments, the robot base 10 can support the robot body 8 and can mount the robot body 8 to a number of powered wheels 12. This mounting to the wheels allows the robot body 8 to move efficiently through space. The robot base 10 can span the length and width of the robot body 8. The robot base 10 can be from about 2 inches to about 10 inches tall. The robot base 10 can cover, protect, and support the powered wheels 12.

[0021] In some embodiments, at least one powered wheel 12 can be attached to the robot base 10 as shown in FIG. 1. The powered wheel 12 can be attached anywhere on the robot base 10. Each individual powered wheel 12 can rotate about a vertical axis in any direction. A motor can be disposed on, in, or adjacent to the powered wheel 12. This motor can move the surgical system 2 anywhere and stabilize and / or level the surgical system 2. A rod located in or adjacent to the powered wheel 12 can be pressed into a surface by the motor. The rod is not depicted but can be made of any metal suitable for lifting the surgical system 2. The rod can lift the powered wheel 10 that can lift the surgical system 2 to any height required to level or otherwise fix the orientation of the surgical system 2 in relation to the patient. The weight of the surgical system 2 supported by the rod of each wheel through a small contact area prevents the surgical system 2 from moving during the medical procedure. This firm positioning can prevent subjects and / or persons from accidentally displacing the surgical system 2.

[0022] Movement of the surgical system 2 can be facilitated using a robot rail 14. The robot rail 14 allows a human to move the surgical system 2 without grasping the robot body 8. As shown in FIG. 1 , the robot rail 14 can span the length of the robot body 8, can span a shorter length than the robot body 8, and / or can span a longer length than the robot body 8. The robot rail 14 can further provide protection for the robot body 8, preventing a subject and / or a worker from contacting, bumping into, or colliding with the robot body 8.

[0023] The robot body 8 can provide support for a horizontal articulated robot, hereafter referred to as "SCARA." The SCARA 24 can be beneficial for use within the surgical system 2 due to the repeatability and compactness of the robotic arm. The compactness of the SCARA can provide additional space within the medical procedure, thereby allowing medical professionals to perform the medical procedure without excessive clutter and area limitations. The SCARA 24 can be comprised of a robotic telescopic support 16, a robotic support arm 18, and / or a robotic arm 20. The robotic telescopic support 16 can be disposed along the robotic body 8. As shown in FIG. 1, the robotic telescopic support 16 can provide support for the SCARA 24 and the display 34. In some embodiments, the robotic telescopic support 16 can be telescopic in a vertical direction. The body of the robotic telescopic support 16 can be of any width and / or height configured to support the forces and weights applied thereto.

[0024] In some embodiments, the medical practitioner may move the Scalar 24 via commands submitted by the medical practitioner. The commands may originate from input received at the display 34, tablet, and / or XR headset (e.g., headset 920 of FIG. 9 ), as described in more detail below. The XR headset may eliminate the need for the medical practitioner to refer to any other display, such as the display 34 or a tablet, which allows for the Scalar 24 to be configured without the display 34 and / or tablet. The commands may be generated by a switch press and / or multiple switch presses, and / or may be generated based on hand gesture commands and / or voice commands sensed by the XR headset, as described in more detail below.

[0025] As shown in FIG. 5, the actuation assembly 60 can include a switch and / or multiple switches. The actuation assembly 60 can be operable to transmit motion commands to the SCARA 24 that enable the operator to manually manipulate the SCARA 24. Upon depression of a switch or multiple switches, the medical practitioner can move the SCARA 24 through applied hand motions. Alternatively or additionally, the operator can control the movement of the SCARA 24 through hand gesture commands and / or voice commands sensed by an XR headset, as described in more detail below. Additionally, when the SCARA 24 is not receiving a movement command, the SCARA 24 can be locked in place to prevent accidental movement by the practitioner and / or other subjects. By locking in place, the SCARA 24 provides a rigid platform from which the end effector 26 can guide a surgical tool during a medical procedure.

[0026] The robotic support arm 18 can be connected to the robotic telescoping support 16 by a variety of mechanisms. In some embodiments, as best seen in FIGS. 1 and 2, the robotic support arm 18 rotates in either direction relative to the robotic telescoping support 16. The robotic support arm 18 can rotate 360 ​​degrees about the robotic telescoping support 16. The robotic arm 20 can be connected to any suitable location on the robotic support arm 18 by a variety of mechanisms that allow it to rotate in either direction relative to the robotic support arm 18. In one embodiment, the robotic arm 20 can rotate 360 ​​degrees relative to the robotic support arm 18. This freedom of rotation allows the operator to position the robotic arm 20 according to the surgical plan.

[0027] 4 and 5 may be attached to the robotic arm 20 in any suitable location. The end effector 26 may be configured to attach to an end effector coupler 22 of the robotic arm 20 positioned by the surgical robot 4. The exemplary end effector 26 includes a tubular guide that guides movement of an inserted surgical tool relative to the anatomical structure where the surgical procedure is to be performed.

[0028] In some embodiments, a dynamic reference array 52 is attached to the end effector 26. The dynamic reference array, also referred to herein as a "DRA" or "reference array," may be a rigid body, a marker, or other markings that may be attached to or formed on one or more XR headsets worn by a practitioner in an operating room, an end effector, a surgical robot, a surgical tool in a navigated surgical procedure, or a patient's anatomy (e.g., bones). The computer platform 910 in combination with a camera tracking system component 6 or other 3D localization system is configured to track the pose (e.g., position and rotational orientation) of the DRA in real time. The DRA may include fiducials, such as the illustrated ball placement. This tracking of the 3D coordinates of the DRA allows the surgical system 2 to determine the pose of the DRA in any multi-dimensional space relative to the target anatomy of the patient 50 of FIG. 5.

[0029] As shown in FIG. 1, the light indicator 28 may be positioned on top of the scalar 24. The light indicator 28 may shine as any type of light that indicates the "state" that the surgical system 2 is currently operating in. In some embodiments, the light may be produced by an LED bulb that may form a ring around the light indicator 28. The light indicator 28 may be made of a fully transparent material that allows light to shine throughout the light indicator 28. The light indicator 28 may be attached to a lower display support 30. The lower display support 30 may allow an operator to conveniently move the display 34 to any suitable location, as shown in FIG. 2. The lower display support 30 may be attached to the light indicator 28 by any suitable mechanism. In some embodiments, the lower display support 30 may rotate about the light indicator 28 or may be rigidly attached to the light indicator 28. The upper display support 32 may be attached to the lower display support 30 by any suitable mechanism.

[0030] In some embodiments, a tablet can be used in conjunction with and / or without the display 34. The tablet can be disposed on the upper display support 32 in place of the display 34 and can be removable from the upper display support 32 during a medical procedure. Additionally, the tablet can communicate with the display 34. The tablet can be connected to the surgical robot 4 by any suitable wireless and / or wired connection. In some embodiments, the tablet can be capable of programming and / or controlling the surgical system 2 during a medical procedure. When controlling the surgical system 2 with a tablet, all input and output commands can be reproduced on the display 34. The use of the tablet allows the operator to operate the surgical robot 4 without having to move around the patient 50 and / or to the surgical robot 4.

[0031] As described below, in some embodiments, the surgeon and / or other practitioner may wear an XR headset that may be used in combination with the display 34 and / or tablet, or the XR head may eliminate the need to use the display 34 and / or tablet.

[0032] As shown in FIG. 3A and FIG. 5, the camera tracking system components 6 cooperate with the surgical robot 4 via a wired or wireless communication network. With reference to FIGS. 1, 3, and 5, the camera tracking system components 6 may include some components similar to the surgical robot 4. For example, the camera body 36 may provide the functionality found in the robot body 8. The robot body 8 may provide an auxiliary tracking bar to which the camera 46 is mounted. Structures within the robot body 8 may also support electronics, communication devices, and power sources used to power the camera tracking system components 6. The camera body 36 may be made of the same materials as the robot body 8. The camera tracking system components 6 may communicate directly with the XR headset, tablet, and / or display 34 by wireless and / or wired networks to enable the XR headset, tablet, and / or display 34 to control the functionality of the camera tracking system components 6.

[0033] The camera body 36 is supported by a camera base 38. The camera base 38 may function as the robot base 10. In the embodiment of FIG. 1, the camera base 38 may be wider than the robot base 10. The width of the camera base 38 may allow the camera tracking system components 6 to connect with the surgical robot 4. As shown in FIG. 1, the width of the camera base 38 may be wide enough to secure the outer robot base 10. When the camera tracking system components 6 and the surgical robot 4 are connected, the additional width of the camera base 38 may allow additional maneuverability of the surgical system 2 and support of the surgical system 2.

[0034] Similar to the robot base 10, multiple powered wheels 12 can be attached to the camera base 38. The powered wheels 12, as well as the action of the robot base 10 and the powered wheels 12, can allow the camera tracking system components 6 to be stabilized, leveled or set to a fixed orientation relative to the patient 50. This stabilization can prevent the camera tracking system components 6 from moving during the medical procedure and can allow the auxiliary tracking bar camera 46 to keep track of the DRA connected to the XR headset and / or the surgical robot 4 and / or keep track of one or more DRAs 52 connected to the anatomical structures 54 and / or tools 58 within the viewing range 56 as shown in FIGS. 3A and 5. This stability and maintenance of tracking enhances the ability of the surgical robot 4 to work effectively with the camera tracking system components 6. Additionally, the wider camera base 38 can provide additional support to the camera tracking system components 6. Specifically, as shown in Figures 3A and 5, a wide camera base 38 can prevent the camera tracking system components 6 from tipping over when the camera 46 is disposed on a patient.

[0035] The camera telescopic support 40 can support the camera 46 of the auxiliary tracking bar. In some embodiments, the telescopic support 40 can raise or lower the camera 46 vertically. A camera handle 48 can be attached to the camera telescopic support 40 at any suitable location and configured to allow an operator to move the camera tracking system components 6 to a planned position prior to a medical procedure. In some embodiments, the camera handle 48 is used to raise and lower the camera telescopic support 40. The camera handle 48 can effectuate the raising and lowering of the camera telescopic support 40 by depressing a button, switch, lever, and / or any combination thereof.

[0036] The lower camera support arm 42 may be attached to the camera telescopic support 40 at any suitable location, and in an embodiment, as shown in FIG. 1, the lower camera support arm 42 may rotate 360 ​​degrees around the telescopic support 40. This freedom of rotation allows an operator to position the camera 46 at any suitable location. The lower camera support arm 42 may be connected to the telescopic support 40 by any suitable mechanism. The lower camera support arm 42 may be used to provide support for the camera 46. The camera 46 may be attached to the lower camera support arm 42 by any suitable mechanism. The camera 46 may pivot in either direction at the attachment point between the camera 46 and the lower camera support arm 42. In an embodiment, a curved rail 44 may be disposed on the lower camera support arm 42.

[0037] The curved rail 44 may be disposed at any suitable location on the lower camera support arm 42. As shown in FIG. 3A, the curved rail 44 may be attached to the lower camera support arm 42 by any suitable mechanism. The curved rail 44 may be any suitable shape, which may be crescent, circular, oval, elliptical, and / or any combination thereof. The camera 46 may be movably disposed along the curved rail 44. The camera 46 may be attached to the curved rail 44 by, for example, rollers, brackets, braces, motors, and / or any combination thereof. Motors and rollers, not shown, may be used to move the camera 46 along the curved rail 44. As shown in FIG. 3A, during a medical procedure, if a subject blocks the camera 46 from viewing one or more DRAs being tracked, a motor may responsively move the camera 46 along the curved rail 44. This motor-driven movement allows the camera 46 to move to a new position where it is no longer blocked by the subject, without moving the camera tracking system components 6. While the camera 46 is prevented from viewing one or more tracked DRAs, the camera tracking system component 6 can send a stop signal to the surgical robot 4, the XR headset, the display 34, and / or the tablet. The stop signal can prevent the scara 24 from moving and / or alert the operator to put on the XR headset and / or look at the display 34 and / or the tablet until the camera 46 reacquires the tracked DRA 52. The scara 24 can be configured to respond to receiving the stop signal by ceasing further movement of the base and / or end effector coupler 22 until the camera tracking system can resume tracking the DRAs.

[0038] 3B and 3C show front and isometric views of another camera tracking system component 6' that may be used with the surgical system of FIG. 1 or may be used independently of a surgical robot. For example, the camera tracking system component 6' may be used to perform navigated surgery without the use of robotic guidance. One of the differences between the camera tracking system component 6' of FIG. 3B and 3C and the camera tracking system component 6 of FIG. 3A is that the camera tracking system component 6' of FIG. 3B and 3C includes a housing that carries a computer platform 910. The computer platform 910 may be configured to perform camera tracking operations to track the DRA, provide surgical navigation information to a display device, such as an XR headset and / or other display device, perform navigated surgery, and perform other computational operations disclosed herein. Thus, the computer platform 910 may include a navigation computer, such as one or more navigation computers of FIG. 14.

[0039] FIG. 6 shows a block diagram view of the components of the surgical system of FIG. 5 used in a medical procedure. With reference to FIG. 6, the tracking camera 46 of the auxiliary tracking bar has a navigation field of view 600 in which the poses (e.g., position and orientation) of the reference array 602 attached to the patient, the reference array 604 attached to the surgical instrument, and the robot arm 20 are tracked. The tracking camera 46 can be part of the camera tracking system components 6′ of FIG. 3B and FIG. 3C, including a computer platform 910 configured to perform the operations described below. The reference array enables tracking by reflecting light in a known pattern, which is decoded to determine the respective poses by the tracking subsystem of the surgical robot 4. If the line of sight between the patient reference array 602 and the tracking camera 46 on the auxiliary tracking bar is blocked (e.g., by a medical personnel, an instrument, etc.), further navigation of the surgical instrument may not be able to be performed, and a response notification can temporarily stop further movement of the robot arm 20 and the surgical robot 4, display a warning on the display 34, and / or issue an audible warning to the medical personnel. The display 34 is available to the surgeon 610 and assistant 612, but requires turning away from the patient and refocusing the eyes to different distances and positions to view it. The navigation software can be controlled by a technician 614 based on voice commands from the surgeon.

[0040] Figure 7 illustrates various display screens that may be displayed on the display 34 of Figures 5 and 6 by the surgical robot 4 when using the navigation features of the surgical system 2. The display screens may include, but are not limited to, radiographs of the patient overlaid with graphical representations of models of instruments positioned on the display screen relative to the anatomical structures based on the developed surgical plan and / or based on the pose of the tracked reference array, different stages of the surgical procedure, and various user selectable menus for controlling dimensional parameters (e.g., length, width, and / or diameter) of the virtually projected implant.

[0041] For navigated surgery, various processing components (e.g., computer platform 910) and associated software, described below, are provided that enable pre-operative planning of the surgical procedure, e.g., placement of implants, and electronic transfer of the plan to computer platform 910 to provide navigation information to one or more users during the planned surgical procedure.

[0042] For robotic navigation, various processing components (e.g., computer platform 910) and associated software described below are provided that enable pre-operative planning of the surgical procedure, e.g., implant placement, and electronic transfer of the plan to the surgical robot 4. The surgical robot 4 uses the plan to guide the robotic arm 20 and connected end effector 26 to produce target poses of the surgical tools relative to the patient's anatomy for the steps of the planned surgical procedure.

[0043] The various embodiments below are directed to using one or more XR headsets that can be worn by the surgeon 610, assistants 612, and / or other medical personnel to provide an improved user interface for receiving information from and / or providing control commands to the surgical robots, camera tracking system components 6 / 6′, and / or other medical equipment in the operating room.

[0044] FIG. 8 illustrates a block diagram of some electrical components of the surgical robot 4, according to some embodiments of the present disclosure. With reference to FIG. 8, a load cell (not shown) may be configured to track the force applied to the end effector coupler 22. In some embodiments, the load cell may be in communication with multiple motors 850, 851, 852, 853, and / or 854. Once the load cell senses the force, information regarding the degree of the applied force may be delivered from a switch array and / or multiple switch arrays to a controller 846. The controller 846 may take the force information from the load cell and process it with a switch algorithm. The switch algorithm is used by the controller 846 to control a motor driver 842. The motor driver 842 controls the operation of one or more of the motors 850, 851, 852, 853, and 854. The motor driver 842 may instruct a particular motor, for example, to produce the same degree of force through the motor as measured by the load cell. In some embodiments, the force produced may come from multiple motors, e.g., 850-854, as directed by the controller 846. Additionally, the motor driver 842 may receive input from the controller 846. The controller 846 may receive information from the load cells regarding the direction of the force sensed by the load cells. The controller 846 may process this information using a motion controller algorithm. The algorithm may be used to provide information to the particular motor driver 842. To replicate the direction of the force, the controller 846 may activate and / or deactivate the particular motor driver 842. The controller 846 may control one or more motors, e.g., one or more of 850-854, to induce motion of the end effector 26 in the direction of the force sensed by the load cells. This force-controlled motion allows the operator to move the scara 24 and the end effector 26 with ease and / or little resistance.Movement of the end effector 26 can be performed to position the end effector 26 in any suitable pose (i.e., location and angular orientation relative to defined three-dimensional (3D) orthogonal reference axes) for use by a medical practitioner.

[0045] 5, can be in the form of a bracelet that encases the end effector coupler 22. The activation assembly 60 can be located on any portion of the scara 24, any portion of the end effector coupler 22 can be worn by the medical practitioner (and communicate wirelessly), and / or any combination thereof. The activation assembly 60 can include a primary button and a secondary button.

[0046] Depressing the primary button may allow the operator to move the scara 24 and end effector coupler 22. According to one embodiment, once set in position, the scara 24 and end effector coupler 22 may not move until the operator programs the surgical robot 4 to move the scara 24 and end effector coupler 22, or are moved using the primary button. In some examples, it may be necessary to depress at least two non-adjacent primary activation switches before the scara 24 and end effector coupler 22 will respond to operator commands. Depressing at least two primary activation switches may prevent accidental movement of the scara 24 and end effector coupler 22 during a medical procedure.

[0047] When activated by the primary button, the load cells can measure the magnitude and / or direction of force exerted by an operator, i.e., medical personnel, on the end effector coupler 22. This information can be transferred to one or more motors, e.g., one or more of 850-854, in the SCARA 24, which can be used to move the SCARA 24 and the end effector coupler 22. The information regarding the magnitude and direction of the force measured by the load cells can cause one or more motors, e.g., one or more of 850-854, to move the SCARA 24 and the end effector coupler 22 in the same direction as sensed by the load cells. This force-controlled movement can enable the operator to move the SCARA 24 and the end effector coupler 22 comfortably and without great effort, since the motors move the SCARA 24 and the end effector coupler 22 at the same time that the operator moves the SCARA 24 and the end effector coupler 22.

[0048] In some examples, the secondary button can be used by the operator as a "selection" device. During a medical procedure, the surgical robot 4 can inform the medical personnel of a particular condition via the XR headset 920, the display 34, and / or the light indicator 28. The XR headset 920 is configured to display an image on a see-through display screen to form an augmented reality image that is superimposed on real-world objects visible through the see-through display screen, respectively. The medical personnel can be prompted by the surgical robot 4 to select a function, mode, and / or an assessment of the condition of the surgical system 2. Pressing the secondary button once can activate a particular function, mode, and / or an acknowledgment information that is conveyed to the medical personnel via the XR headset 920, the display 34, and / or the light indicator 28. Additionally, pressing the secondary button multiple times in quick succession can activate additional functions, modes, and / or select information that is conveyed to the medical personnel via the XR headset 920, the display 34, and / or the light indicator 28.

[0049] 8, the electrical components of the surgical robot 4 include a platform subsystem 802, a computer subsystem 820, a motion control subsystem 840, and a tracking subsystem 830. The platform subsystem 802 includes a battery 806, a power distribution module 804, a connector panel 808, and a charging station 810. The computer subsystem 820 includes a computer 822, a display 824, and a speaker 826. The motion control subsystem 840 includes a driver circuit 842, motors 850, 851, 852, 853, 854, stabilizers 855, 856, 857, 858, an end effector connector 844, and a controller 846. The tracking subsystem 830 includes a position sensor 832 and a camera converter 834. The surgical robot 4 can also include a removable foot pedal 880, and a removable tablet computer 890.

[0050] The input power is provided to the surgical robot 4 via a power source that can be provided to the power distribution module 804. The power distribution module 804 is configured to receive the input power and generate various power supply voltages that are provided to other modules, components, and subsystems of the surgical robot 4. The power distribution module 804 can be configured to provide various voltage supplies to a connector panel 808, for example, to power the motors 850-854 and the end effector coupler 844, to other components such as the computer 822, the display 824, the speaker 826, the driver 842, and to the camera converter 834 and other components for the surgical robot 4. The power distribution module 804 can also be connected to a battery 806 that serves as a temporary power source when the power distribution module 804 does not receive power from the input power. In other cases, the power distribution module 804 can function to charge the battery 806.

[0051] The connector panel 808 may function to connect different devices and components to the surgical robot 4 and / or associated components and modules. The connector panel 808 may include one or more ports that receive lines or connections from various components. For example, the connector panel 808 may include a ground terminal port that may ground the surgical robot 4 to other equipment, a port for connecting a foot pedal 880, a port for connecting to a tracking subsystem 830 that may include a position sensor 832, a camera converter 834, and a DRA tracking camera 870. The connector panel 808 may also include other ports that allow USB, Ethernet, HDMI communications to other components, such as a computer 822. According to some embodiments, the connector panel 808 may include wired and / or wireless interfaces for operatively connecting one or more XR headsets 920 to the tracking subsystem 830 and / or the computer subsystem 820.

[0052] The control panel 816 may provide various buttons or indicators to control the operation of the surgical robot 4 and / or produce information from the surgical robot 4 for the operator to view. For example, the control panel 816 may include buttons to power the surgical robot 4 on and off, to raise and lower the vertical support 16, and to raise and lower the stabilizers 855-858, which may be designed to engage the casters 12 and lock the surgical robot 4 from physically moving. Other buttons may stop the surgical robot 4 in an emergency, remove all motor power and apply mechanical brakes to prevent all movement. The control panel 816 may also have indicators to inform the operator of certain system conditions, such as a line power indicator or the charge status of the battery 806. According to some embodiments, one or more XR headsets 920 may communicate, for example, via the connector panel 808, to control the operation of the surgical robot 4 and / or receive and display information generated by the surgical robot 4 for view by the person wearing the XR headset 920.

[0053] The computer 822 of the computer subsystem 820 includes an operating system and software for operating the assigned functions of the surgical robot 4. The computer 822 can receive and process information from other components (e.g., the tracking subsystem 830, the platform subsystem 802, and / or the motion control subsystem 840) to display information to the operator. The computer subsystem 820 can also provide output to the operator through a speaker 826. The speaker can be part of the surgical robot, part of the XR headset 920, or internal to another component of the surgical system 2. The display 824 can correspond to the display 34 shown in FIGS. 1 and 2.

[0054] Tracking subsystem 830 can include a position sensor 832 and a camera converter 834. Tracking subsystem 830 may correspond to camera tracking system component 6 of FIG. 3. DRA tracking camera 870 works with position sensor 832 to determine the pose of DRA 52. This tracking can be done in a manner consistent with this disclosure, including the use of infrared or visible light techniques to track the location of active or passive elements of DRA 52, such as LEDs or reflective markers (also referred to as markers), respectively.

[0055] The functional operations of the tracking subsystem 830 and the computer subsystem 820 may be included in a computer platform 910, which may be carried by the camera tracking system component 6' of Figures 3A and 3B. The tracking subsystem 830 may be configured to determine a pose, e.g., location and angular orientation, of the tracked DRA. The computer platform 910 may also include a navigation controller configured to use the determined pose to provide navigation information to a user that guides their movement of the tracked tools relative to the image of the patient and / or the tracked anatomical structures with which the position is registered during the planned surgical procedure. The computer platform 910 may display information on a display of Figures 3B and 3C and / or in one or more XR headsets 920. When used with a surgical robot, the computer platform 910 may be configured to communicate with the computer subsystem 820 and other subsystems of Figure 8 to control the movement of the end effector 26. For example, as described below, the computer platform 910 can generate graphical representations of a patient's anatomy, surgical tools, a user's hands, etc., with displayed size, shape, color, and / or pose controlled based on the determined pose of one or more tracked DRAs, and the displayed graphical representations can be dynamically modified to track changes in the determined pose over time.

[0056] The motion control subsystem 840 can be configured to physically move the vertical column 16, the upper arm 18, the lower arm 20, or rotate the end effector coupler 22. The physical movement can be accomplished through the use of one or more motors 850-854. For example, the motor 850 can be configured to vertically raise or lower the vertical column 16. The motor 851 can be configured to move the upper arm 18 laterally about a point of engagement with the vertical column 16, as shown in FIG. 2. The motor 852 can be configured to move the lower arm 20 laterally about a point of engagement with the upper arm 18, as shown in FIG. 2. The motors 853 and 854 can be configured to move the end effector coupler 22 to provide translational movement and rotation along three-dimensional axes. 9 can provide control inputs to the controller 846 that guide the movement of the end effector coupler 22 to position the connected passive end effector at a planned pose (i.e., location and angular orientation relative to defined 3D orthogonal reference axes) relative to the anatomical structure to be operated on during a planned surgical procedure. The motion control subsystem 840 can be configured to measure the position of the end effector coupler 22 and / or the end effector 26 using integrated position sensors (e.g., encoders).

[0057] 9 shows a block diagram of components of a surgical system including camera tracking system components 6 (FIG. 3A) or 6′ (FIGS. 3B, 3C) and / or imaging devices (e.g., C-arm 104, O-arm 106, etc.) connected to a computer platform 910, which may be operatively connected to a surgical robot 4 according to some embodiments of the present disclosure. Alternatively, at least some of the operations disclosed herein as being performed by the computer platform 910 may additionally or alternatively be performed by components of the surgical system.

[0058] 9, a computer platform 910 includes a display 912, at least one processor circuit 914 (also referred to as a processor for brevity), at least one memory circuit 916 (also referred to as a memory for brevity) containing computer readable program code 918, and at least one network interface 902 (also referred to as a network interface for brevity). The display 912 may be part of an XR headset 920 according to some embodiments of the present disclosure. The network interface 902 may be configured to connect to the C-arm imaging device 104 of FIG. 10, the O-arm imaging device 106 of FIG. 11, another medical imaging device, an image database 950 containing medical images of a patient, components of a surgical robot 4, and / or other electronic devices.

[0059] When used with a surgical robot 4, the display 912 may correspond to the display 34 of Figure 2, and / or the tablet 890 of Figure 8, and / or an XR headset 920 operatively connected to the surgical robot 4, the network interface 902 may correspond to the platform network interface 812 of Figure 8, and the processor 914 may correspond to the computer 822 of Figure 8. The network interface 902 of the XR headset 920 may be configured to communicate over a wired network, e.g., thin-wire Ethernet, and / or over a wireless RF transmit and receive link in accordance with one or more wireless communication protocols, e.g., WLAN, 3GPP 4G and / or 5G (New Radio) cellular communication standards, etc.

[0060] The processor 914 may include one or more data processing circuits, such as general purpose and / or special purpose processors, e.g., a microprocessor and / or a digital signal processor, etc. The processor 914 is configured to execute computer readable program code 918 in the memory 916 to perform operations that may include some or all of the operations described herein as being performed for surgical planning, navigated surgery, and / or robotic surgery.

[0061] The computer platform 910 can be configured to provide surgical planning functionality. The processor 914 can operate to display images of anatomical structures, e.g., vertebrae, received from one of the imaging devices 104 and 106 and / or from the image database 950 through the network interface 920 on the display device 912 and / or on the XR headset 920. The processor 914 receives the operator's definition of which anatomical structures shown in one or more images are to undergo a surgical procedure, e.g., screw placement, such as by the operator's touch to select a location on the display 912 for the planned procedure or by using a mouse-based cursor to define the location of the proposed procedure. When the images are displayed on the XR headset 920, the XR headset can be configured to sense gesture-based commands formed by the wearer and / or sense voice-based commands issued by the wearer, which can be used to control selection between menu items and / or how objects are displayed on the XR headset 920, as described in further detail below.

[0062] The computer platform 910 may be configured to allow measurements of anatomical structures that may be particularly useful for knee surgery, such as measurements of various angles to determine hip center, angular center, natural landmarks (e.g., transepicondylar line, Whitesides line, posterior condylar line), etc. Some measurements may be automatic, while some other measurements may involve human input or assistance. The computer platform 910 may be configured to allow an operator to input the correct implant selection for the patient, including size and alignment selection. The computer platform 910 may be configured to perform automatic or semi-automatic (with human input) segmentation (image processing) of CT images or other medical images. The patient's surgical plan may be stored in a cloud-based server, which may correspond to the database 950, for retrieval by the surgical robot 4.

[0063] For example, during an orthopaedic surgical procedure, the surgeon can select which cut to make (e.g., posterior femur, proximal tibia, etc.) using a computer screen (e.g., touch screen) or augmented reality (XR) interaction (e.g., hand gesture-based commands and / or voice-based commands), for example, via an XR headset 920. The computer platform 910 can generate navigation information that provides visual guidance to the surgeon to perform the surgical procedure. When used with the surgical robot 4, the computer platform 910 can provide guidance that can cause the surgical robot 4 to automatically move the end effector 26 to a target pose such that the surgical tool is aligned at the target location to perform the surgical procedure on the anatomical structure.

[0064] In some embodiments, the surgical system 900 can use two DRAs to track the position of the patient's anatomical structures, such as one connected to the patient's tibia and one connected to the patient's femur. The system 900 can use standard navigational instruments for alignment and confirmation (e.g., pointers similar to those used in the Globus Excelsius GPS system for spinal surgery).

[0065] A particularly challenging task in navigated surgery is how to plan the location of implants in the spine, knee, and other anatomical structures where surgeons have difficulty performing the task on a computer screen that is a 2D representation of the 3D anatomical structures. The system 900 can address this problem by displaying computer-generated three-dimensional (3D) representations of the anatomical structures and candidate implant devices using an XR headset 920. The computer-generated representations are scaled and posed relative to one another on the display screen under the guidance of the computer platform 910 and can be manipulated by the surgeon while viewed through the XR headset 920. The surgeon can manipulate the displayed computer-generated representations of the anatomical structures, implants, surgical tools, etc., using, for example, hand gesture-based and / or voice-based commands sensed by the XR headset 920.

[0066] For example, the surgeon can view a displayed virtual handle on the virtual implant and can manipulate (e.g., grab and move) the virtual handle to move the virtual implant to a desired pose and adjust the placement of the planned implant relative to the graphical representation of the anatomy. Thereafter, during the surgical procedure, the computer platform 910 can display navigation information through the XR headset 920 that facilitates the surgeon's ability to more accurately follow the surgical plan for inserting the implant and / or perform another surgical procedure on the anatomy. If the surgical procedure involves resection of bone, the progress of the bone resection, e.g., the depth of the cut, can be displayed in real time via the XR headset 920. Other features that can be displayed via the XR headset 920 can include, but are not limited to, gap or ligament balance along the range of joint motion, implant contact lines along the range of joint motion, ligament tension and / or relaxation by color or other graphical rendering, etc.

[0067] In some embodiments, the computer platform 910 can enable planning the use of standard surgical tools and / or implants, e.g., posterior stabilized and cruciate retained implants, cemented and uncemented implants, e.g., total or partial knee replacements, and / or hip replacements, and / or correction systems for trauma-related surgeries.

[0068] An automated imaging system can be used in conjunction with the computer platform 910 to obtain pre-operative, intra-operative, post-operative, and / or real-time image data of an anatomical structure. An exemplary automated imaging system is illustrated in FIGS. 10 and 11. In some embodiments, the automated imaging system is a C-arm 104 (FIG. 10) imaging device or an O-arm® 106 (FIG. 11). (O-arm® is copyrighted by Medtronic Navigation, Inc., having a place of business in Louisville, Colorado, USA). It may be desirable to image x-rays of a patient from several different positions without the need for frequent manual repositioning of the patient, as may be required with x-ray systems. X-ray diagnostic equipment of the C-arm 104 may solve the problem of frequent manual repositioning and may be well known in the medical field of surgery and other interventional procedures. As shown in FIG. 10, the C-arm includes an elongated C-shaped member terminating in opposing distal ends 112 of the "C" shape. The C-shaped member is attached to an X-ray source 114 and an image receiver 116. The space within the arm's C-arm 104 provides room for a physician to attend to the patient with substantial freedom from interference from the X-ray support structure.

[0069] The C-arm is mounted to allow rotational movement of the arm in two degrees of freedom (i.e., about two orthogonal axes in a spherical motion). The C-arm is slidably mounted to the X-ray support structure, which allows for orbital rotational movement about the C-arm's center of curvature, which may allow for selective orientation of the X-ray source 114 and image receiver 116 vertically and / or horizontally. The C-arm may also be rotatable laterally (i.e., in a direction perpendicular to the orbital direction allowing for selectively adjustable positioning of the X-ray source 114 and image receiver 116 relative to both the width and length of the patient). The spherical rotation aspect of the C-arm device allows the physician to image the patient's X-rays at the optimal angle determined for the particular anatomical situation being imaged.

[0070] The O-arm® 106 shown in FIG. 11 includes a gantry housing 124 that can enclose an image capture portion, not illustrated. The image capture portion includes an X-ray source and / or X-ray emitting portion and an X-ray receiving and / or image receiving portion, which can be disposed approximately 180 degrees apart from one another and mounted on a rotor (not shown) relative to the orbit of the image capture portion. The image capture portion can be operable to rotate 360 ​​degrees during image acquisition. The image capture portion can rotate about a center point and / or axis, allowing patient image data to be acquired from multiple directions or in multiple planes.

[0071] The O-Arm® 106 with gantry housing 124 has a central opening for positioning around the object to be imaged and a radiation source rotatable around the interior of the gantry housing 124, which can be adapted to project radiation from a number of different projection angles. The detector system is adapted to detect radiation for each projection angle to obtain images of the object from a number of projection planes in a pseudo-simultaneous manner. The gantry can be mounted in a cantilevered manner to the O-Arm® support structure of a support structure such as a wheeled mobile cart with wheels. The positioning unit translates and / or tilts the gantry to a predetermined position and orientation, preferably under the control of a computerized motion control system. The gantry can include a source and a detector disposed opposite each other on the gantry. The source and detector can be fixed to a motorized rotor that can rotate the source and detector around the interior of the gantry in coordination with each other. The source can be pulsed in multiple positions and orientations over a partial and / or full 360 degree rotation for multi-planar imaging of a target object located inside the gantry. The gantry may further include a rail and bearing system for guiding the rotor as it rotates, which system may carry sources and detectors. Both and / or either the O-arm® 106 and C-arm 104 may be used as automated imaging systems to scan the patient and send information to the surgical system 2.

[0072] Images captured by the imaging system can be displayed on the XR headset 920 and / or another display device of the computer platform 910, the surgical robot 4, and / or another component of the surgical system 900. The XR headset 920 may be connected, e.g., via the computer platform 910, to one or more of the imaging devices 104 and / or 106 and / or the image database 950 to display images therefrom. A user may provide control inputs, e.g., gesture-based commands and / or voice-based commands, through the XR headset 920 to control the operation of one or more of the imaging devices 104 and / or 106 and / or the image database 950.

[0073] FIG. 12 shows a block diagram view of components of a surgical system including a pair of XR headsets 920 (head mounted displays HMD1 and HMD2) that correspond to the XR headset 920 shown in FIG. 13 and that may operate in accordance with some embodiments of the present disclosure.

[0074] Referring to the exemplary scenario of FIG. 12, both the assistant 612 and the surgeon 610 are each wearing an XR headset 920. It is optional for the assistant 612 to wear the XR headset 920. The XR headset 920 is configured to provide an interactive environment, as described further below, through which the wearer can view and interact with information related to the surgical procedure. This interactive XR-based environment can eliminate the need for the technician 614 shown in FIG. 6 to be in the operating room and can eliminate the need for the use of the display 34 shown in FIG. 6. Each XR headset 920 can include one or more cameras configured to provide an additional source of tracking of DRAs or other reference arrays attached to surgical tools, patient anatomy, end effector 26, and / or other equipment. In the example of FIG. 12, the XR headset 920 has a field of view (FOV) 1202 for tracking the DRA and other objects, the XR headset 920 has an FOV 1212 that partially overlaps with the FOV 1202 for tracking the DRA and other objects, and the tracking camera 46 has another FOV 600 that partially overlaps with the FOVs 1202 and 1212 for tracking the DRA and other objects.

[0075] If one or more cameras are blocked from viewing a tracked object, e.g., a DRA attached to a surgical tool, but the DRA is in the field of view of one or more other cameras, the tracking subsystem 830 and / or navigation controller 828 can continue to track the object seamlessly without losing sight of navigation. Furthermore, if the DRA is partially obscured from one camera's viewpoint, but the entire DRA is seen through multiple camera sources, the camera tracking inputs can be aggregated to continue navigation of the DRA. One of the XR headsets and / or tracking cameras 46 can display and track the DRA on another one of the XR headsets to enable the computer platform 910 (FIGS. 9 and 14), the tracking subsystem 830, and / or another computing component to determine the pose of the DRA relative to one or more defined coordinate systems, e.g., the XR headset 920, the tracking camera 46, and / or another coordinate system defined relative to the patient, the table, and / or the room.

[0076] The XR headset 920 may be operatively connected to view video, photographs, and / or other received information and / or to provide commands to control various equipment in the operating room, including, but not limited to, neuromonitoring, microscopes, video cameras, and anesthesia systems. Data from various equipment, e.g., display of patient vitals or microscope feeds, may be processed and displayed within the headset.

[0077] Exemplary XR Headset Components and Integration into Navigated Surgery, Surgical Robots, and Other Devices FIG. 13 illustrates an XR headset 920 configured in accordance with some embodiments of the disclosure. The XR headset includes a headband 1306 configured to secure the XR headset to a wearer's head, an electronics housing 1304 supported by the headband 1306, and a display screen 1302 extending laterally downward from the electronics housing 1304. The display screen 1302 may be a see-through LCD display device or a semi-reflective lens that reflects an image projected by the display device toward the wearer's eyes. A set of DRA fiducials 1310, e.g., dots, are painted or attached at known spacings located on one or both sides of the headset. The headset's DRA allows a tracking camera of an auxiliary tracking bar to track the pose of the headset 920 and / or allows another XR headset to track the pose of the headset 920.

[0078] The display screen 1302 acts as a see-through display screen, also called a combiner, that reflects light from a display panel of the display device toward the user's eyes. The display panel may be located between the electronics housing and the user's head and angled to project virtual content toward the display screen 1302 for reflection toward the user's eyes. The display screen 1302 is semi-transparent and semi-reflective, allowing the user to see the virtual content in reflection superimposed on the user's view of the real-world scene. The display screen 1302 may have regions of varying opacity, such as the illustrated upper horizontal band having a higher opacity than the lower horizontal band. The opacity of the display screen 1302 may be electronically controlled to adjust how much light from the real-world scene passes through to the user's eyes. A high opacity configuration of the display screen 1302 results in a high contrast virtual image superimposed on a dim view of the real-world scene. A low opacity configuration of the display screen 1302 may result in a dimmer virtual image superimposed on a sharper view of the real-world scene. The opacity can be controlled by applying an opaque material to the surface of the display screen 1302 .

[0079] According to some embodiments, the surgical system includes an XR headset 920 and an XR headset controller, e.g., controller 1430 of FIG. 14. The XR headset 920 is configured to be worn by a user during a surgical procedure and has a see-through display screen 1302 configured to display an XR image and to allow at least a portion of a real-world scene to pass through it for viewing by the user. The XR headset 920 also includes an opacity filter positioned between at least one of the user's eyes and the real-world scene when the user views the see-through display screen 1302. The opacity filter is configured to provide opacity to light from the real-world scene. The XR headset controller is configured to communicate with a navigation controller, e.g., controllers 828A, 828B, and / or 828C of FIG. 14, to receive navigation information from the navigation controller that provides guidance to the user during a surgical procedure on the anatomical structure, and is further configured to generate an XR image based on the navigation information for display on the see-through display screen 1302.

[0080] The opacity of the display screen 1302 may be configured as a gradient with more continuously varying opacity moving downward from the top of the display screen 1302. The darkest point of the gradient may be at the top of the display screen 1302, with the display screen 1302 becoming less and less opaque further down until the opacity becomes transparent or nonexistent. In a further exemplary embodiment, the gradient may change from about 90% opacity to completely transparent at approximately mid-eye height of the display screen 1302. If the headset is properly calibrated and positioned, mid-eye height corresponds to the point where a user would look straight out, and the end of the gradient may be located at the "horizon" line of the eye. The dark portion of the gradient allows for crisp and clear visuals of the virtual content and helps block out the distracting brightness of the overhead operating room lights.

[0081] The use of opacity filters in this manner allows the XR headset 920 to provide virtual reality (VR) functionality by substantially or completely blocking light from the real-world scene along the top of the display screen 1302, and AR functionality along the center or bottom of the display screen 1302. This allows the user to make the AR semi-transparent when desired, allowing clear optics of the patient's anatomy during the procedure. Configuring the display screen 1302 as a gradient instead of a more constant opacity band allows the wearer to experience a more natural transition from a more VR-type view to a more AR-type view without experiencing abrupt changes in real-world scene brightness and depth of field that could otherwise strain the eyes, such as during a more rapid shift between an upward and downward view.

[0082] The display panel and viewing screen 1302 can be configured to provide a wide field of view, see-through XR display system. In one exemplary configuration, they produce an 80° diagonal field of view (FOV) with a 55° vertical range for a user to view virtual content. Other diagonal FOV angles and vertical coverage angles can be produced through different sized display panels, different curvature lenses, and / or different distance and angular orientations between the display panel and the curved viewing screen 1302.

[0083] FIG. 14 illustrates electrical components of an XR headset 920 that can be operatively connected to a computer platform 910, to one or more imaging devices, such as the C-arm imaging device 104, the O-arm imaging device 106, and / or to an image database 950, and / or to a surgical robot 800 according to various embodiments of the present disclosure.

[0084] The XR headset 920 provides an improved human interface for performing navigated surgical procedures. The XR headset 920 may be configured to provide functionality including, for example, but not limited to, any one or more of hand gesture-based and / or voice-based command identification, display of XR graphical objects on a display device 1450, via the computer platform 910. The display device 1450 may be a video projector, flat panel display, etc. that projects the displayed XR graphical objects onto a display screen 1302. A user may view the XR graphical objects as overlays fixed to certain real-world objects viewed via the display screen 1302 (FIG. 13). The XR headset 920 may additionally or alternatively be configured to display video feeds from one or more cameras mounted on the XR headset 920 and other cameras on the display screen 1450.

[0085] The electrical components of the XR headset 920 can include multiple cameras 1440, microphones 1442, gesture sensors 1444, a pose sensor (e.g., an inertial measurement unit (IMU)) 1446, a display module 1448 including a display device 1450, and a wireless / wired communication interface 1452. As described below, the camera 1440 of the XR headset can be a visible light capture camera, a near infrared capture camera, or a combination of both.

[0086] The camera 1440 can be configured to operate as a gesture sensor 1444 by capturing for identification of a user's hand gesture performed within the field of view of the camera 1440. Alternatively, the gesture sensor 1444 can be a proximity sensor and / or a touch sensor that senses a hand gesture performed in proximity to the gesture sensor 1444 and / or senses physical contact, such as tapping on the sensor or housing 1304. The pose sensor 1446, e.g., an IMU, can include a multi-axis accelerometer, tilt sensor, and / or another sensor that can sense rotation and / or acceleration of the XR headset 920 along one or more defined coordinate axes. Some or all of these electrical components can be contained within the component housing 1304 or can be contained within a separate housing configured to be worn elsewhere, such as on the hip or shoulder.

[0087] As described above, the surgical system 2 includes the camera tracking system components 6 / 6′ and a tracking subsystem 830 that may be part of the computer platform 910. The surgical system may include an imaging device (e.g., the C-arm 104, the O-arm 106, and / or an image database 950) and / or a surgical robot 4. The tracking subsystem 830 is configured to determine a pose of a DRA attached to an anatomical structure, an end effector, a surgical tool, etc. The navigation controller 828 is configured to determine a target pose of the surgical tool relative to the anatomical structure based on a surgical plan derived from a surgical planning function performed by the computer platform 910 of FIG. 9 that defines where the surgical procedure will be performed using the surgical tool at the anatomical structure and based on the pose of the anatomical structure determined by the tracking subsystem 830. The navigation controller 828 may be further configured to generate steering information based on the surgical tool target pose, the anatomical structure pose, and the surgical tool and / or end effector pose, where the steering information indicates where to move the surgical tool and / or the end effector of the surgical robot to perform the surgical plan. Various cameras 1440 of the XR headset 920 may be connected to the camera tracking system components 6 / 6' to track the pose of the DRA, the user's hands, etc.

[0088] The electrical components of the XR headset 920 may be operatively connected to the electrical components of the computer platform 910 via a wired / wireless interface 1452. The electrical components of the XR headset 920 may be operatively connected via the computer platform 910 or directly connected to, for example, various imaging devices, such as the C-arm imaging device 104, the I / O arm imaging device 106, an image database 950, and / or other medical equipment via the wired / wireless interface 1452.

[0089] The surgical system 2 further includes at least one XR headset controller 1430 (also referred to as "XR headset controller" for brevity), which may reside on the XR headset 920, the computer platform 910, and / or another system component connected by a wired cable and / or wireless communication link. Various functionality is provided by software executed by the XR headset controller 1430. The XR headset controller 1430 is configured to receive navigation information from the navigation controller 828 that provides guidance to a user during a surgical procedure on an anatomical structure, and is configured to generate an XR image based on the navigation information for display on the display device 1450 for projection on the see-through display screen 1302.

[0090] The arrangement of the display device 1450 relative to the display screen (also called a "see-through display screen") 1302 is configured to display XR images such that the XR images appear to be in the real world when viewed through the display screen 1302 by a user wearing the XR headset 920. The display screen 1302 can be positioned by a headband 1306 in front of the user's eyes.

[0091] The XR headset controller 1430 may be in a housing configured to be worn on a user's head or other part of the user's body while viewing the display screen 1302, or may be located remotely from a user viewing the display screen 1302 while communicatively connected to the display screen 1302. The XR headset controller 1430 may be configured to operably process signaling from the camera 1440, microphone 142, and / or pose sensor 1446, and is connected to display XR images on a display device 1450 for a user viewing at the display screen 1302. Thus, it should be understood that the XR headset controller 1430, illustrated as a circuit block within the XR headset 920, is operably connected to other illustrated components of the XR headset 920, but is not necessarily present within a common housing (e.g., electronics housing 1304 of FIG. 13) or transportable by the user. For example, the XR headset controller 1430 may reside within the platform 910, which in turn may reside within the housing of the computer tracking system component 6' shown in Figures 3B and 3C.

[0092] Example optical layout of XR headset components FIG. 15 shows a block diagram illustrating the arrangement of optical components of the XR headset 920 according to some embodiments of the disclosure. Referring to FIG. 15, the display device 1450 is configured to display an XR image generated by the XR headset controller 1430, the light from which is projected as an XR image 1500 toward the display screen 1302. The display screen 1302 is configured to combine the light of the XR image 1500 with the light from the real-world scene 1502 into a synthetic augmented view 1504 directed toward the user's eye 1510. The display screen 1302 configured in this way operates as a see-through display screen. The XR headset 920 can include any number of tracking cameras 1440. The cameras 1440 can be visible light capture cameras, near infrared capture cameras, or a combination of both.

[0093] Exemplary user view via XR headset The operation of the XR headset allows both 2D images and 3D models to be displayed on the display screen 1302. Preferably, the 2D images can be displayed in a more opaque band (upper band) of the display screen 1302, and more preferably, the 3D models can be displayed in a more transparent band (lower band) of the display screen 1302, otherwise known as the environment area. Below the lower band, where the display screen 1302 ends, the wearer gets an unobstructed view of the operating room. Note that the location where the XR content is displayed on the display screen 1302 can be fluid. Depending on the position of the headset relative to the content, it is possible that the location where the 3D content is displayed can move to the opaque band and the location where the 2D content is displayed can be placed in the transparent band and can be stable with respect to the real world. Furthermore, the entire display screen 1302 can be darkened under electronic control to convert the headset to virtual reality in surgical planning or fully transparent during a medical procedure. As described above, the XR headset 920 and associated operations can be performed in conjunction with robotically assisted procedures as well as supporting navigated procedures.

[0094] FIG. 16 shows an exemplary view through a display screen 1302 of an XR headset 920 for providing navigational assistance to a user manipulating a surgical tool 1602 during a medical procedure, according to some embodiments of the present disclosure. Referring to FIG. 16, when the surgical tool 1602 is brought into the vicinity of the tracked anatomical structure such that the dynamic reference arrays 1630 and 1632 connected to the surgical tool 1602 are within the field of view of the cameras 1440 (FIG. 15) and / or 46 (FIG. 6), the graphical representation 1600 of the tool may be displayed in 2D and / or 3D images in relation to the graphical representation 1610 of the anatomical structure. The user may use the viewed graphical representation to adjust the trajectory 1620 of the surgical tool 1602, which may be shown as extending from the graphical representation 2000 of the tool through the graphical representation 1610 of the anatomical structure. The XR headset 920 may also display text information and other objects 1640. A dashed line 1650 extending across the displayed display screen represents an exemplary division between upper and lower bands of different opacity levels.

[0095] Other types of XR images (virtual content) that can be displayed on the display screen 1302 can include, but are not limited to, any one or more of the following: 1) 2D axial, sagittal, and / or coronal views of the patient's anatomy, 2) Overlay of the planned tracked tool versus the currently tracked tool and surgical implant location; 3) A gallery of preoperative images; 4) Video feeds from microscopes and other similar systems, or remote video conferencing; 5) Options and Config Settings and Buttons 6) A floating 3D model of the patient's anatomy with surgical planning information; 7) Real-time tracking of surgical instruments against floating patient anatomy; 8) enhanced superimposition of the patient's anatomy with command and guidance; and 9) Expanded overlay of surgical instruments.

[0096] Exemplary Camera Configurations for Tracking System Components 17 shows an example configuration of the auxiliary tracking bar 46 with two pairs of stereo tracking cameras configured in accordance with some embodiments of the present disclosure. The auxiliary tracking bar 46 is part of the camera tracking system components of FIGS. 3A, 3B, and 3C. According to one embodiment, the stereo tracking cameras include a stereo pair of spaced apart visible light capture cameras and another stereo pair of spaced apart near-infrared capture cameras. Alternatively, only one stereo pair of visible light capture cameras or only one stereo pair of near-infrared capture cameras may be used in the auxiliary tracking bar 46. Any number of near-infrared and / or visible light cameras may be used.

[0097] Pose measurement chain As described above, navigated surgery may involve computer vision tracking and determining the pose (e.g., position and orientation in a six degree of freedom coordinate system) of a surgical instrument, such as by determining the pose of a mounted DRA that includes a spaced fiducial, such as a disk or sphere, positioned in a known manner relative to a camera tracking system. The computer vision uses spaced tracking cameras, such as stereo cameras, configured to capture near infrared and / or visible light. In this scenario, there are three parameters that compete together for optimization: (1) accuracy, (2) robustness, and (3) user ergonomics during the surgical procedure.

[0098] A computational operation can combine (chain) the measured poses in a manner that can improve the optimization of one or more of the above three parameters by incorporating additional tracking cameras mounted on one or more XR headsets. As shown in FIG. 17, in accordance with some embodiments of the present disclosure, a stereo pair of visible light tracking cameras and another stereo pair of near-infrared tracking cameras can be attached to the auxiliary tracking bar of a camera tracking system component. A computational algorithm is disclosed that analyzes the poses of fully observed or partially observed (e.g., when less than all criteria of the DRA are displayed by a pair of stereo cameras) DRAs and combines the observed poses or partial poses in a manner that can improve accuracy, robustness, and / or ergonomics during navigated surgery.

[0099] As described above, an XR headset can be configured to augment a real-world scene with a computer-generated XR image. An XR headset can be configured to provide an XR display environment by displaying the computer-generated XR image on a see-through display screen through which light from the real-world scene can pass for a combined view by the user. Alternatively, an XR headset can be configured to provide a VR display environment by preventing or substantially preventing light from the real-world scene from being directly viewed by the user along the display path of the displayed XR image. An XR headset can be configured to provide both an AR and a VR display environment. In one embodiment, both the AR and VR display environments are provided by lateral bands of effectively different opacity disposed between the see-through display screen and the real-world scene, such that the VR display environment is provided in the XR image aligned with the high opacity band, and the AR display environment is provided in the XR image aligned with the low opacity band. In another embodiment, both the AR and VR viewing environments are provided by computer adjustable control of an opacity filter that variably suppresses how much light from the real-world scene passes through the see-through viewing screen for combination with the XR image the user sees. Thus, an XR headset can also be referred to as an AR headset or a VR headset.

[0100] As also described above, the XR headset can include near-infrared and / or visible light tracking cameras configured to track DRA fiducials connected to surgical instruments, patient anatomy, other XR headsets, and / or robotic end effectors. Using near-infrared and / or visible light tracking in the XR headset provides additional tracking volume coverage beyond what a single auxiliary tracking bar camera can provide. Adding a near-infrared tracking camera to an existing auxiliary tracking bar can track the position of the headset more robustly than visible light, but with less accuracy. Mechanically calibrating the visible light tracking coordinate system and the near-infrared tracking coordinate system allows the coordinate systems to be sufficiently aligned to perform a 3D DRA fiducial triangulation operation using stereo matching, and jointly determine the pose of the DRA fiducial between the visible light tracking coordinate system and the infrared tracking coordinate system. Using both visible light and near-infrared tracking coordinate systems may enable one or more of the following: (a) identifying tools that would not be identified using only one coordinate system; (b) improving pose tracking accuracy; (c) allowing a greater range of motion without losing track of the surgical instruments, patient anatomy, and / or robotic end effectors; and (d) naturally tracking the XR headset in the same coordinate system as the navigated surgical instruments.

[0101] 18 shows a block diagram view of the components of a surgical system including the tracking cameras of the XR headset 920 (head mounted displays HMD1 and HMD2) and the tracking cameras of the camera tracking bar of the camera tracking system component 6′ that houses the computer platform 910. The computer platform 910 can include a tracking subsystem 830, a navigation controller 828, and an XR headset controller 1430, as previously shown in FIG.

[0102] Referring to the surgical system of Figure 18, the surgeon and assistant are both wearing XR headsets HMD1 920 and HMD2 920, respectively, where each includes a tracking camera that can be configured as shown in Figure 13. It is optional for the assistant to wear the XR headset HMD2 920.

[0103] The combination of the XR headsets HMD1 920 and HMD2 920 with the tracking cameras 46 on the auxiliary tracking bar in conjunction with the computer platform 910 can more robustly track exemplary objects of the patient reference array (R), the robotic end effector (E), and the surgical tool (T) or instrument. The overlapping views from different perspectives provided by the XR headsets HMD1 920 and HMD2 920 and the tracking cameras 46 on the auxiliary tracking bar are shown in FIG.

[0104] Each labeled item in Figure 18 represents a unique coordinate system. The coordinate system labels are explained as follows: A = the visible light coordinate system of the second headset HMD2 920; N3 = Near Infrared (NIR) coordinate system of the second headset HMD2 920; S = visible light coordinate system of the primary headset HMD1 920; N2 = NIR coordinate system of the primary headset HMD1 920, N = NIR coordinate system of the auxiliary navigation bar 46; V = the visible light coordinate system of the secondary navigation bar 46, R = NIR coordinate system of the patient reference array 602; T = NIR coordinate system of the tracked tool 604; E = NIR coordinate system of the tracked robot end effector on the robot arm 20, and W = Inertially navigated world coordinate system with a stable gravity vector.

[0105] The spatial relationships of some of these labeled objects (and by extension, coordinate systems) can be measured and calibrated during the manufacturing process, when the equipment is installed in the operating room, and / or before a surgical procedure is performed. In the disclosed system, the following coordinate systems are calibrated:

[0106]

number

[0107] where the term "T" is defined as a six degree of freedom (6DOF) uniform transformation between the two coordinate systems shown. Thus, for example,

[0108]

number

[0109] The term is a 6 degree of freedom uniform transformation between the visible light coordinate system of the primary headset HMD1 920 and the NIR coordinate system of the primary headset HMD1 920.

[0110] In one embodiment, XR headsets HMD1 920 and HMD2 920 have passive visible light fiducials (coordinate systems S and A) painted or otherwise attached to them, such as the reference array fiducial 1310 shown in Figure 13. The tracking cameras are spatially calibrated to these passive fiducials (coordinate systems N2 and N3).

[0111] As explained above, the cameras on the XR headsets HMD1 920 and HMD2 920, as well as the tracking camera 46 on the auxiliary tracking bar, have partially overlapping fields of view. If one or more of the cameras of the XR headset HMD1 920 is blocked from viewing a DRA attached to a tracked object, such as a tracked tool (T), but the DRA is visible through the cameras of the other XR headset HMD2 920 and / or the tracking camera 46 on the auxiliary tracking bar, the computer platform 910 can continue to track the DRA seamlessly without losing navigation. In addition to this, if the DRA is partially occluded from the perspective of the cameras of the XR headset HMD1 920, but the entire DRA is visible through the cameras of the other XR headset HMD2 920 and / or the tracking camera 46 on the auxiliary tracking bar, the tracking inputs of the cameras can be merged to continue navigation of the DRA.

[0112] More specifically, the various coordinate systems can be chained together by independently observing the various camera systems provided by the XR headsets HMD1 920 and HMD2 920 and the tracking camera 46 on the auxiliary tracking bar. For example, each of the XR headsets HMD1 920 and HMD2 920 may require a virtual extension of the robot end effector (E). One XR headset HMD1 920 (N2) and the tracking camera 46 on the auxiliary tracking bar (N) can see (E), but perhaps not the other XR headset HMD2 920 (N3). The position of (E) relative to (N3) can still be calculated through one of several different operating methods. An embodiment of the operation performs pose chaining from the patient reference (R). If the patient reference (R) is seen by (N3) and either (N) or (N2), the pose of (E) relative to (N3) can be directly solved by one of the following two equations:

[0113]

number

[0114] The key to this pose chaining is that the relationships between the frames at the end of each chain are inferred (circled and moved downwards). Chains can be of any length, and are enabled by having multiple stereo camera systems (e.g., N, N2, N3).

[0115] The camera tracking system may be configured to receive tracking information related to the tracked object from a first tracking camera (e.g., N3) and a second tracking camera (e.g., N2) during the surgical procedure. The camera tracking system may generate a first pose transformation between a first object (e.g., R) coordinate system and a first tracking camera (e.g., N3) coordinate system based on the first object tracking information from the first tracking camera (e.g., N3) indicating a pose of the first object (e.g., R).

[0116]

number

[0117] The camera tracking system can determine a second pose transformation between the first subject (e.g., R) coordinate system and the first tracking camera (e.g., N2) coordinate system based on the first subject tracking information from the second tracking camera (e.g., N2) indicating the pose of the first subject (e.g., R).

[0118]

number

[0119] The camera tracking system can determine a third pose transformation between the second object (e.g., E) coordinate system and the first tracking camera (e.g., N2) coordinate system based on second object tracking information from a second tracking camera (e.g., N2) indicating a pose of the second object (e.g., E).

[0120]

number

[0121] The camera tracking system can determine a fourth pose transformation between the second object (e.g., E) coordinate system and the first tracking camera (e.g., N3) coordinate system based on a combination of the first, second, and third pose transformations.

[0122]

number

[0123] It is possible to determine the following.

[0124] In some further embodiments, the camera system can further determine a pose of the second object (e.g., E) and the first tracking camera system (e.g., N3) coordinate system based on processing the tracking information via a fourth pose transformation.

[0125] Because the fields of view of the various camera systems overlap, the camera tracking system can determine the pose of the second subject (e.g., E) relative to the first tracking camera (e.g., N3) when the first camera is blocked from viewing the second subject (e.g., E). For example, in some embodiments, the camera tracking system can calculate a fourth pose transformation between the second subject (e.g., E) coordinate system and the first tracking camera (e.g., N3) coordinate system without using any tracking information from the first tracking camera (e.g., N3) that indicates the pose of the second subject (e.g., E).

[0126]

number

[0127] The device is further configured to determine:

[0128] The camera tracking system can achieve higher tracking accuracy by merging synchronized images from multiple camera systems. For example, the camera tracking system can determine the pose of the second subject (e.g., E) relative to the first tracking camera (e.g., N3) by merging synchronized images of the second subject (e.g., E) from multiple viewpoints (first and second tracking cameras), with weighting that can be determined based on the accuracy specifications of each camera. More specifically, the camera tracking system can determine a fourth pose transformation between the second subject (e.g., E) coordinate system and the first tracking camera (e.g., N3) coordinate system based on the second subject tracking information from the first tracking camera (e.g., N3) indicating the pose of the second subject (e.g., E) and further based on the result of the combination of the first, second, and third pose transformations.

[0129]

number

[0130] The device may be further configured to determine:

[0131] The surgical system can be configured to display, on a see-through display screen of the XR headset, an XR image having a pose determined based on the fourth pose transformation. The camera tracking system can be further configured to generate an XR image as a graphical representation of a second subject (e.g., E) posed on the see-through display screen based on processing the first subject tracking information from the first and second tracking cameras and the second subject tracking information from the second tracking camera via the fourth pose transformation.

[0132] As described above, the camera tracking system may include a navigation controller 828 communicatively connected to a first tracking camera (e.g., N3) and a second tracking camera (e.g., N2) to receive tracking information and configured to perform determination of the first, second, third, and fourth pose transformations.

[0133] Registration of surgical tool characteristics to a reference array identified by a camera tracking system During a surgical procedure, the camera tracking system can simultaneously track the pose of a surgical tool held or supported within the field of view of a set of tracked cameras, and can resume tracking the surgical tool when the surgical tool moves from outside to inside the field of view, e.g., after being reacquired. Many surgical tools require software configuration to be properly tracked. In order for the camera tracking system to track the pose of reference arrays attached to or attached to the surgical tool, the camera tracking system should be informed of which of the tracked reference arrays the characteristics of the surgical tool are registered with. This allows the navigation controller 828 (FIG. 14) to operate with knowledge of the particular characteristics of the surgical tool. For example, registration of the characteristics of the surgical tool, such as the distance from an identified reference array to the tip of the surgical tool, allows the navigation controller 828 to navigate the movement of the surgeon's tool tip during a surgical procedure. Similarly, registration of the curvature direction of the surgical tool relative to the identified reference array enables the navigation controller 828 to display an accurate graphical representation of the surgical tool through the XR headset 920 and accurately position it relative to the tracked anatomical structures during the surgical procedure.

[0134] In the registration process, also called a pairing process, a user holds a surgical tool with a reference array in the field of view of a set of tracking cameras to identify the reference array, and then the user defines the characteristics of that surgical tool according to some embodiments. The registration process is repeated for each combination of reference array and surgical tool characteristics that are tracked to the surgical procedure, and can be further repeated when the reference array is removed from one type of surgical tool and attached to another type of surgical tool. It may be important for a surgeon or other medical personnel wearing an XR headset to be able to perform the registration process of a pair of surgical tools in a time-efficient manner, helping the surgeon avoid errors in registering the characteristics of the surgical tool with the reference array, and reducing interruptions to the surgeon's focus before and during the surgical procedure.

[0135] Some further embodiments of the present disclosure are directed to using an XR headset during the registration process to register the identified reference array to characteristics of the surgical tool and display a representation of those characteristics via the XR headset so that the user can verify the accuracy of the registration. The use of an XR headset during the registration process results in a more intuitive, time-efficient and reliable process for surgeons and other medical personnel (users) to register the surgical tool to the camera tracking system prior to and / or during the surgical procedure.

[0136] The registration process may be initiated automatically in response to a reference array being brought into the field of view of a set of tracking cameras and the camera tracking system determining that an identified reference array is not registered as being paired with a characteristic of the surgical tool.

[0137] In an embodiment, the XR headset includes a set of tracking cameras that may be included in the electronics housing 1304 of the XR headset 920, for example, as shown in FIG. 13. The camera tracking system (e.g., system 6 of FIG. 18) identifies a reference array (e.g., physical reference array 1800 of FIGS. 21-22) to be tracked by the set of tracking cameras, such as by identifying the reference array in a video stream from the tracking cameras. The reference array may be uniquely identified based on the different orientations between the sets of fiducials that form the respective reference array. The camera tracking system determines whether the identified reference array is paired and registered with a characteristic of one of a plurality of surgical tools defined in a surgical tool database. Based on the reference array determined to be not registered and receiving the user's input, the camera tracking system registers the reference array as paired with a characteristic of a surgical tool selected from among the plurality of surgical tools based on the user's input. The camera tracking system then produces a representation of the characteristic on a display device of the XR headset 920 for display to the user.

[0138] Thus, a user can initiate registration by raising a surgical tool within the field of view of a set of tracking cameras attached to the XR headset 920. When an unregistered reference array comes into the field of view of the tracking cameras, the camera tracking system can cause a visual prompt to be displayed via the XR headset 920, which prompts the user to perform registration of the reference array to the associated surgical tool characteristic. The camera tracking system receives video from the tracking cameras, identifies the reference array based on the spacing and relativity, and determines that the identified reference array is not already registered as being paired with any defined characteristic of a surgical tool defined in the surgical tool database. The camera tracking system receives user input identifying the characteristic of the surgical tool and registers the reference array as being paired with the characteristic of the surgical tool.

[0139] Examples of features of surgical tools may include, but are not limited to, drills, screwdrivers, saws, retractors, and structural and / or operational features of implants such as screws, spacers, interbody fusion devices, plates, rods, etc.

[0140] Example Configuration of "Three Camera" Tracking for Tracking System Components If five or more three-dimensional (3D) points of an object can be identified, only one camera is needed to solve for the six degrees of freedom (6DOF) position and orientation of the object relative to the camera. If the two cameras are properly calibrated together, the 6DOF pose can be solved to use only three 3D points via stereo triangulation. According to various embodiments of the present disclosure, a third camera is introduced, positioned between but spaced apart from a line that runs through the pair of stereo cameras. This third camera improves accuracy by producing a third, off-axis perspective of the object, edge, or point tracked in 2D or 3D.

[0141] When using two or more cameras, the pose of an object, such as a surgical tool or a robotic end effector, is solved by triangulating the tracked features against a database of feature locations. Thus, depending on the accuracy: How accurately the features are located in the 2D image.

[0142] b. How the tracked object features are spread in 3D space.

[0143] c. What is the prevalence and number of tracking cameras?

[0144] d. How well the camera is calibrated.

[0145] Adding a third camera reduces the error in (a.) due to simple averaging of noise, while directly affecting the error due to (c.). The third camera can also serve as a validation of the accuracy of the calibration of the other two cameras. For example, if the stereo pair is not calibrated in orientation or yaw, the tracked object will appear triangulated but out of depth. This calibration error can be directly observed by adding a third camera. Finally, by placing the third camera slightly above the other two cameras and optionally angling it, if all cameras are visible light cameras, the third camera obtains a new perspective on the 3D geometry of the object that was not previously observable when using NIR cameras.

[0146] Figure 20a shows single-point stereo camera triangulation using two tracking cameras (2010 and 2012), and Figure 20b shows single-point triangulation using three tracking cameras (2020, 2022, and 2024) according to some embodiments of the present disclosure.

[0147] 19 shows an example configuration of a camera bar 1900 having three stereo tracking cameras (e.g., an auxiliary tracking bar of the camera tracking system component 6 / 6′ of FIGS. 6 and 18 or part of the XR headset 920) configured in accordance with some embodiments of the disclosure. The camera bar 1900 may be part of a camera tracking system, which may be within the camera tracking system component 6 / 6′, may be at least partially within the navigation controller 828, or may be within the XR headset 920. In this embodiment, the three tracking cameras may be visible light cameras and / or the tracking cameras may be configured for sensing near-infrared (NIR) light.

[0148] The first and second tracking cameras (1910 and 1920) may be mounted at spaced apart locations on the camera bar 1900. The centers of the first and second tracking cameras are spaced apart a distance D1. The third tracking camera 1930 may be mounted at a position on the camera bar 1900 that is between the positions of the first and second tracking cameras (1910 and 1920) and spaced a distance D2 from a line 1940 that extends through the centers of the first and second tracking cameras (1910 and 1920).

[0149] The distance D2 that the third tracking camera 1930 is spaced from the line 1940 may be between one-quarter and one-half the distance D1 between the centers of the first and second tracking cameras (1910 and 1920). Additionally, the third tracking camera 1930 may be the same distance away from the first tracking camera 1910 as the third tracking camera 1930 is from the second tracking camera 1920. All three tracking cameras may be pointing downward and angled so that the views of the tracking cameras converge towards a single point that is within the distance range that the tracking cameras are configured to track objects at, such as the farthest and nearest predicted distances between objects being tracked in the surgical environment and the tracking camera locations.

[0150] In this embodiment, the use of a visible light camera and high contrast tracking targets, as can be used in Figures 21-25, results in an alternative to NIR solutions, (a) disc-based targets with geometrically encoded identification and (b) edge-dominant targets with QR or barcode style identification.

[0151] 21-22 show a surgical tool 1950 attached to a physical reference array 1800 that includes fiducial disks arranged in two parallel planes, each containing the location of at least one passive fiducial disk. The array of passive fiducial disks can be tracked simultaneously by multiple cameras at different angles. The placement of the passive fiducial disks allows them to be seen by the tracking cameras even on steep trajectories.

[0152] FIG. 21 illustrates various angles of view of a surgical tool 1950 attached to an exemplary physical reference array 1800 with five reference disks 1820 and 1830a-d. In FIG. 21, the reference disk 1820 is in a first plane that is parallel to and spaced apart from the plane that contains the reference disks 1830a-d. In this example, the reference disks 1820, 1830a, 1830d may be referred to as outer disks, and their positions may remain the same across multiple physical reference arrays. The reference disks 1830b, 1830c are inner disks, and their positions along the respective arms of the physical reference array may vary between different physical reference arrays, and the differences may be detected to uniquely identify the physical reference array. Each physical reference array includes a coupling element 1810 for attachment to a surgical tool 1950 (e.g., a surgical tool). The tracking system can detect the unique pattern of the reference discs 1820, 1830a-d and recognize the particular physical reference array and therefore determine the position of the surgical tool 1950. Because the identifier of the object is encoded in the 3D geometric distance between the reference discs, changing the relative position of the two inner discs 1830b and 1830c to the rest of the array creates a different pattern and therefore changes the unique identifier of the object.

[0153] 21, it can be seen that the fiducial disk may be detectable from a rear view of the physical reference array 1800. This allows the surgical tool 1950 to be detected, identified, and tracked while the physical reference array 1800 is not facing or angled directly toward the camera. Thus, the surgeon may be more free to use an instrument (e.g., the surgical tool 1950) without having to adjust or use the instrument so that the physical reference array 1800 can face the tracking camera.

[0154] In some embodiments, discs must be at least 4mm center to center on adjacent discs in order to be detected as different discs and encoded as unique patterns within the tracking software. These unique patterns allow multiple instruments to be tracked at once.

[0155] Comparing Figures 21-22, the use of five disks can provide redundancy. For example, in Figure 22, reference disk 1830b is obscured by 1820, but the distance between reference disk 1830c and reference disk 1830d can be used to identify the physical reference array. Only three disks are needed for pause recovery, but four disks are needed to uniquely identify the target. This allows any one disk to be obscured or obscured without losing any functionality.

[0156] These reference disks can be inserted, painted, or laser etched into the array. Because the array may not require sensitive electronics or retroreflective beads, the array can be easily autoclaved.

[0157] In additional or alternative embodiments, a different number of reference disks can be used. In some examples, a physical reference array may use only four reference disks, doubling the number of unique patterns available. A single internal reference disk can be used instead of two internal reference disks (e.g., reference disks 1830b-c), and the position of the single internal reference disk can be adjusted to any position along either arm of the physical reference array.

[0158] In some embodiments, each of the disks is consistent in shape and size, and the positions of the outer three disks remain constant between patterns. In additional or alternative embodiments, the size, shape, and position of the disk markers may be adjusted, so long as the physical reference array includes at least two parallel planes, each of which includes at least one reference disk.

[0159] In some embodiments, the disks are either retroreflective or colored for visible light tracking. The disks may be replaceable, for example, using a snap-in / push-out method or thread features from the back or array face. The disks may be designed for visible light tracking and may be machine washable. Since contrast may be important for visible light tracking, the disks may be replaced once contrast is lost through use or cleaning.

[0160] In some embodiments, parallel planes allow tracking from multiple angles (e.g., top-down and side) depending on the surgical field setup, which can reduce the risk of occlusion and lost navigation. In additional or alternative embodiments, the reference disks used in the tracking arrays can be designed for visible light and machine washable for use on multiple occasions. In additional or alternative embodiments, a physical reference array can allow optimized accuracy while still allowing the surgical site to be seen from the surgeon's perspective.

[0161] In some embodiments, the physical reference array can use any shape fiducial. For example, the physical reference array can include two or more parallel planes, each including at least one spherical fiducial. In additional or alternative embodiments, some fiducials used in the physical reference array can be pre-determined to allow greater redundancy or greater unique patterns to act as unique identifiers.

[0162] 22 shows a side view of the physical reference array 1800 attached to a surgical tool 1950. In this view, the reference disks are not visible because they are on edge. A first plane 1850 containing the reference disks 1820 is parallel to a second plane 1860 containing the reference disks 1830a-d, and the first and second planes are spaced apart in a direction perpendicular to the planes.

[0163] FIG. 26 illustrates an example of a process performed by a camera tracking system including a camera bar 1900 and tracking cameras (1910 / 1920 / 1930), according to some embodiments. The camera tracking system including the camera bar 1900 may further include at least one processor operative to perform the process of FIG. 26. The at least one processor may determine a pose of a set of physical objects in three-dimensional space based on a triangulation of positions of objects imaged in the video stream from the first, second, and third tracking cameras (e.g., 1910 / 1920 / 1930) to positions of the first, second, and third tracking cameras spaced apart in three-dimensional space, at block 2600. The set of physical objects may be a physical reference array (e.g., physical reference array 1800) of spaced apart fiducials (e.g., fiducial disks 1820 and 1830a-d). Exemplary operations for determining a pose are described above.

[0164] At least one processor of the camera tracking system may further operate to select an identifier corresponding to a physical reference array from among a set of registered identifiers corresponding to a set of registered reference arrays defined in a database, in block 2602. The selection in block 2602 may be based on correlating a similarity of a pattern of distances between fiducials of the physical reference array imaged in the video stream to a pattern of distances between fiducials of one of the registered reference arrays defined in the database, and selecting the identifier corresponding to one of the registered reference arrays.

[0165] The databases referenced here and below may be stored in the camera tracking system, or the camera tracking system may communicate with a database (e.g., database 950) over one or more networks (e.g., a database stored on a distributed cloud resource) so that the camera tracking system can receive information of the registered reference arrays necessary to perform these steps of the process.

[0166] In block 2604, at least one processor may operate to retrieve from the database registered spatial relationships between the fiducials of the physical reference array (e.g., 1800) based on the identifier selected (in block 2602), and determine the pose of the physical reference array based on triangulation of the registered spatial relationships and positions of the fiducials of the physical reference array (e.g., 1820 and 1830a-d) imaged in the video stream to the positions of the first, second, and third tracking cameras spaced in three-dimensional space from the first, second, and third tracking cameras.

[0167] FIG. 27 shows additional or alternative examples of processes performed by a camera tracking system including a camera bar 1900 and tracking cameras (1910 / 1920 / 1930), according to some embodiments.

[0168] A camera tracking system including camera bar 1900 may further include at least one processor operative to execute the process of Figure 27. The at least one processor may determine, in block 2700, a pose of a set of physical objects in three-dimensional space based on a triangulation of positions of objects imaged in the video stream from the first, second, and third tracking cameras (e.g., 1910 / 1920 / 1930) to positions of the first, second, and third tracking cameras spaced apart in three-dimensional space. The set of physical objects may be a physical reference array (e.g., physical reference array 1800) of spaced apart fiducials (e.g., fiducial disks 1820 and 1830a-d).

[0169] Block 2700 may be performed in the same or similar manner as block 2600 above.

[0170] In block 2702, the at least one processor may be further operable to determine a first pose of a physical reference array of spaced apart fiducials in three-dimensional space based on triangulation positions of fiducials imaged in the video stream from the first and second tracking cameras to positions of the first and second tracking cameras spaced apart in three-dimensional space.

[0171] In block 2704, the at least one processor may be further operable to determine a second pose of the physical reference array of spaced apart fiducials in three-dimensional space based on triangulation positions of the fiducials imaged in the video stream from the first and third tracking cameras to positions of the first and third tracking cameras spaced apart in three-dimensional space.

[0172] At block 2706, the at least one processor may be further operable to determine an accuracy of calibration of the at least first and second tracking cameras based on a comparison of the first and second poses.

[0173] During manufacturing, shipping, or implementation, tracked cameras may be inadvertently moved or adjusted. The accuracy of the calibration may indicate the level of precision or accuracy of the tracked cameras, i.e., how precise is the orientation, yaw positioning, and / or spacing between the tracked cameras in 3D space.

[0174] At block 2708, the at least one processor may be further operable to initiate a process of calibrating the orientation and / or yaw positioning of the first and second tracking cameras in response to the accuracy of the calibration not meeting a defined rule. The defined rule may correspond to when the accuracy of the calibration is less than a defined threshold, which may indicate excessive movement has occurred in one or more cameras.

[0175] In one embodiment, the camera tracking system can make adjustments or indicate needed adjustments to correct / improve the accuracy of the calibration of the first and second tracking cameras.

[0176] FIG. 28 shows additional or alternative examples of processes performed by a camera tracking system including a camera bar 1900 and tracking cameras (1910 / 1920 / 1930), according to some embodiments.

[0177] A camera tracking system including the camera bar 1900 may further include at least one processor operative to execute the process of Figure 28. The at least one processor may determine, in block 2800, a pose of a set of physical objects in three-dimensional space based on a triangulation of positions of objects imaged in the video stream from the first, second, and third tracking cameras (e.g., 1910 / 1920 / 1930) to positions of the first, second, and third tracking cameras spaced apart in three-dimensional space. The set of physical objects may be a physical reference array (e.g., physical reference array 1800) of spaced apart fiducials (e.g., fiducial disks 1820 and 1830a-d).

[0178] Block 2800 may be performed in the same or similar manner as block 2600 above.

[0179] In block 2802, the at least one processor may further operate to detect potential discs (e.g., reference discs 1820 and 1830a-d) within frames of the video streams from the video streams from the first, second, and third tracking cameras.

[0180] For example, in the example of Figures 21-22, the process of block 2802 may be accomplished by taking a frame of the video stream, increasing the contrast of the frame, and having the computer tracking system search for fiducials such as reference disks 1820 and 1830a-d. By increasing the contrast of the frame, the computer tracking system may be able to more easily detect the black reference disks 1820 and 1830a-d on the physical reference array 1800. Optionally, the computer tracking system may additionally or alternatively invert the contrast in the frame such that the computer tracking system looks for white disks corresponding to reference disks 1820 and 1830a-d.

[0181] At block 2804, the at least one processor may further operate to estimate a location of a center of gravity of the disc within a frame of the video stream.

[0182] At block 2806, the at least one processor may further operate to verify whether the positions of the centroids of the disks form a pattern that corresponds to the pattern of the disks of the reference array registered in the database.

[0183] In one embodiment, the computer tracking system compares the pattern formed by the positions of the centroids of the disks to the patterns of a reference array registered in a database. If the patterns of the registered reference array match the patterns formed by the positions of the centroids of the disks, the estimated centroid positions and the patterns formed by those estimated positions are verified.

[0184] Optionally, if at least one processor cannot be verified at block 2806, the process may return to block 2800. However, if at least one processor can be verified at block 2806, the process may continue to block 2808.

[0185] At block 2808, the at least one processor may further operate to determine a gradient along an edge of the disc to generate a gradient ray extending toward the center of gravity of the disc. As described above, the determination at block 2808 may be responsive to successful validation of the disc in block 2806.

[0186] In one embodiment, the gradient rays are rays that are generated orthogonal to the tangent at spaced locations around the edge of the disk. Once multiple gradient rays are generated for the disk, the gradient rays may be determined to converge to a point, which may be determined to be the center of gravity of the disk.

[0187] At block 2810, the at least one processor may further operate to adjust the gradient rays to reduce distortion based on defined intrinsic parameters of the lenses of the first, second, and third tracking cameras and output distortion-compensated gradient rays.

[0188] The defined intrinsic parameters of the tracked camera's lenses may include the focal length of the lens, the optical center (principal point) of the tracked camera, the lens distortion produced by each lens, a skew factor defining the angle between the x pixel axis and the y pixel axis, and / or pixel coordinates.

[0189] At block 2812, the at least one processor may be further operable to perform a double cone ellipse fitting on the distortion-compensated gradient rays to determine locations of the distortion-compensated centroids of the disks. The locations of the compensated centroids of each disk may correspond to where the distortion-compensated gradient rays of the disks converge.

[0190] The at least one processor may further operate to determine a pose of the physical reference array of spaced fiducials in three-dimensional space in block 2814. This determination in block 2814 may be based on triangulation of the location of the distortion-compensated center of gravity of the disk of the video stream from the first, second, and third tracking cameras to the positions of the first, second, and third tracking cameras spaced apart in three-dimensional space.

[0191] FIG. 23 shows an example of a surgical tool tracking array including a multi-planar reference array 2310. The surgical tool tracking array may include a tool holder that couples a portion of the surgical tool 2300 to the surgical tool tracking array. The multi-planar reference array 2310 may have multiple planar surfaces 2310a-e. The planar surfaces may include a front square plane (e.g., 2310a) that may be configured to face away from the surgical tool 2300 when coupled to the tool holder. The planar surfaces may also include multiple side square planes, each extending from a different side edge of the front square plane and angled toward the surgical tool when coupled to the tool holder. For example, in FIG. 23, the multi-planar reference array 2310 includes four side square planes 2310b-e, each extending from a different one of the four side edges of the front square plane 2310a and angled toward the surgical tool when coupled to the tool holder.

[0192] The planes 2310a-e may each include the same or different optically encoded patterns. The optically encoded patterns may include barcode patterns that optically encode an identifier for the plane. The barcode patterns have the ability to encode a large number of potential identities by supporting up to a 6x6 array of black and white regions, each representing a bit of information. The resulting 2^36 bits of information yields 6.8719E10 potential identification codes. This also allows for 2 bits of error correction, and when divided by 5 planes per multi-plane reference array, thousands of unique identification codes are still possible. The inclusion of multiple planes also allows for recognition and identification of objects from larger off-axis viewpoints than a physical reference array (e.g., 1800), making the array smaller and less obtrusive.

[0193] Although five planes are shown in this example, any number of planes may be used and may be configured in any arrangement. Additionally, the planes may be other shapes besides squares and have optically encoded patterns within the shape. For example, the planes may be rectangular, triangular, oval, circular, etc.

[0194] FIG. 24 shows an example of 2D image registration between undistorted image information and synthetic image information generated from a 3D model 2420 of a surgical tool. Once the object is reliably identified, a 3D model of the entire object (i.e., tool, robot end effector, or other type of item) can be loaded. The 3D model 2420 can be projected onto the 2D image plane 2400 of all tracked cameras at the location determined by the reference pose recovery. Edges, corners, and other salient model features are then compared to the undistorted live camera image data, and the pose of the model is iterated until all registration errors between the two objects (synthetic and real-world) are minimized. Assuming there are no defects in the tool or object, this optimized model pose represents an improved, more functionally relevant pose. If the tool or object is bent, improperly assembled, or otherwise defective, the process will not converge and a safety warning will be triggered. This process allows for confirmation that the 3D model 2420 is properly projected onto the 2D image. This is an iterative method that adjusts the 3D model to a more accurate position on the 2D image, resulting in the ability to detect deformations of the tool or object.

[0195] FIG. 25 shows an example of an end effector 2500 with fiducial disks 2502a-j. The use of fiducial disks 2502a-j on the end effector allows for reliable and robust identification and tracking up to 80 degrees off-axis in any of the perspectives shown in FIG. 25 (side view, front view, and top view). Additionally, rings 2504a-c can be inserted, painted, or laser etched into the shaft of the end effector for optional precision improvement or orientation verification of the end effector. Rings 2504a-c can be used to create a barcode or other machine-readable optical code that provides identification of the end effector 2500 or additionally or alternatively provides identification of the shaft on which rings 2504a-c are located. Additionally, a camera tracking system can generate tangential or perpendicular light beams of rings 2504a-c that can be used to determine the pose of the end effector. For example, by generating rays perpendicular to rings 2504a-c, a computer tracking system may be able to determine information about the pose of the end effector, such as the pose at which the shaft of the end effector is pointing.

[0196] In one embodiment, the end effector 2500 is fabricated from metal that has gone through a black anodization process and has a white reference disc thereon.

[0197] In another embodiment, the end effector 2500 is white metal with a black reference disk on it, except that the camera tracking system inverts the contrast so that the camera tracking system detects a black end effector with a white reference disk.

[0198] 29 illustrates an example of a process performed by a camera tracking system including a camera bar 1900 and tracking cameras (1910 / 1920 / 1930) according to some embodiments. The camera tracking system including the camera bar 1900 may further include at least one processor operative to perform the process of FIG.

[0199] In block 2900, at least one processor of the camera tracking system may be attached to the surgical tool (e.g., 2300) and operable to decode optically coded patterns, each in a different one of a plurality of different planes (e.g., 2310a-e) of a multi-planar reference array (e.g., 2310) imaged in the video stream from the first, second, and third tracking cameras (e.g., 1910 / 1920 / 1930). Additionally, the at least one processor may determine one or more identifiers for a corresponding one or more of the plurality of planes of the multi-planar reference array imaged in the video stream. The determination of block 2900 may be based on decoding one or more optically coded patterns of the plurality of planes of the multi-planar reference array imaged in the video stream.

[0200] In examples where the optically encoded pattern is a two-dimensional barcode pattern, the at least one processor may further operate to decode the two-dimensional barcode pattern.

[0201] In block 2902, the at least one processor may be further operable to retrieve, from the database, a registered spatial relationship between corners and / or edges of one or more optically coded patterns of the multiple faces of the multi-faceted reference array imaged in the video stream based on the one or more identifiers determined in block 2900. Additionally, the at least one processor may determine a pose of the multi-faceted reference array based on the registered spatial relationship and triangulation of positions of corners and / or edges of one or more optically coded patterns of the multiple faces of the multi-faceted reference array imaged in the video stream relative to positions of the first, second, and third tracking cameras spaced apart in three-dimensional space.

[0202] At block 2904, the at least one processor may be further operable to retrieve from the database a registered composite three-dimensional model of the surgical tool registered as attached to the multi-planar reference array.

[0203] In block 2906, the at least one processor may be further operable to determine a relational projection of the registered synthetic three-dimensional model relative to a physical surgical tool imaged in a two-dimensional plane of frames in the video streams from the first, second, and third tracking cameras based on the determined pose of the multi-planar reference array determined in block 2902.

[0204] In block 2908, the at least one processor may be further operable to identify an offset between a position of a feature of the registered synthetic three-dimensional model and a position of a feature of the physical surgical tool imaged in the two-dimensional plane of a frame in the video stream from the first, second, and third tracking cameras based on the relational projection determined in block 2906.

[0205] At block 2910, the at least one processor may further operate to determine a pose of the physical surgical tool based on the identified offset and the pose of the multi-planar reference array.

[0206] At block 2912, the at least one processor may be further operable to identify when the physical surgical tool has excessive deformation based on the identified offsets that satisfy the tool deformation law. If excessive deformation is identified, the at least one processor may trigger a safety warning.

[0207] Although various embodiments have been described in the context of using a set of tracking cameras on an XR headset, these and other embodiments can be used with any form of tracking cameras, such as an auxiliary tracking bar and / or a set of tracking cameras on a separate XR headset. Thus, in some embodiments, the set of tracking cameras is separate and spaced apart from the XR headset while the camera tracking system receives tracking information from the set of tracking cameras. The various operations disclosed herein for pose chaining can be used to track the pose of the reference array and / or the user's hands.

[0208] Further provisions and embodiments: In the above description of various embodiments of the inventive concept, it should be understood that the terms used herein are only intended to describe specific embodiments and are not intended to limit the inventive concept.Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the inventive concept belongs.It will be further understood that terms as defined in commonly used dictionaries should be interpreted as having a meaning that is not inconsistent with their meaning in the context of this specification and the related technical field, and not in the idealized or overly formal meaning as expressly defined herein.

[0209] When an element is referred to as being "connected," "coupled," "responsive," or variations thereof to another element, the element may be directly connected, coupled, or responsive to the other element, or intervening elements may be present. In contrast, when an element is referred to as being "directly connected," "directly coupled," "directly responsive," or variations thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, as used herein, "coupled," "connected," "responsive," or variations thereof may include wirelessly coupled, connected, or responsive. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly dictates otherwise. Well-known functions or constructions may not be described in detail for brevity and / or clarity. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0210] In this specification, terms such as first, second, third, etc. may be used to describe various elements / operations, but it is understood that these elements / operations should not be limited by these terms. These terms are used only to distinguish one element / operation from another. Thus, a first element / operation in some embodiments can be referred to as a second element / operation in other embodiments without departing from the teachings of the inventive concept. The same reference numbers or the same reference characters refer to the same or similar elements throughout the specification.

[0211] As used herein, the terms "comprise", "comprising", "comprises", "include", "including", "includes", "have", "has", "having", or variations thereof, are open ended and include one or more stated features, integers, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integers, elements, steps, components, functions, or groups thereof. Furthermore, as used herein, the common abbreviation "eg", derived from the Latin "exempli gratia", may be used to introduce or designate a general example or examples of the aforementioned items and is not intended to be limiting of such items. The common abbreviation "ie", derived from the Latin "id est", may be used to designate a particular item from a more general list.

[0212] Exemplary embodiments are described herein with reference to block diagrams and / or flowchart illustrations illustrating computer-implemented methods, apparatus (systems and / or devices), and / or computer program products. It will be understood that blocks of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions executed by one or more computer circuits. These computer program instructions can be provided to a general-purpose computer circuit, a special-purpose computer circuit, and / or a processor circuit of other programmable data processing circuitry to present a machine, whereby the instructions executed via the processor of the computer and / or other programmable data processing device transform and control transistors, values ​​stored in memory locations, and other hardware components in such circuitry to implement the function / action specified in the block diagram and / or flowchart block(s), thereby creating means (functions) and / or structures for implementing the function / action specified in the block diagram and / or flowchart block(s).

[0213] These computer program instructions may also be stored on a tangible computer readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner to produce an article of manufacture, the instructions stored on the computer readable medium including instructions that implement the functions / acts specified in the block diagram and / or flowchart block(s).Thus, embodiments of the inventive concepts may be embodied in hardware and / or software (including firmware, resident software, microcode, etc.) operating on a processor, such as a digital signal processor, which may be collectively referred to as "circuitry," "modules," or variations thereof.

[0214] It should also be noted that in some alternative implementations, the functions / acts described in the blocks may occur in a different order than the order described in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functions / acts involved. Furthermore, the functionality of a given block in the flowcharts and / or block diagrams may be separated into multiple blocks, and / or the functionality of two or more blocks in the flowcharts and / or block diagrams may be at least partially integrated. Finally, other blocks may be added / inserted between the illustrated blocks, and / or blocks or acts may be omitted, without departing from the scope of the inventive concept. Furthermore, while some of the figures include arrows in communication paths to indicate a primary direction of communication, it should be understood that communication may occur in a direction opposite to that of the depicted arrows.

[0215] Many changes and modifications can be made to the embodiments without substantially departing from the principles of the inventive concept. All such changes and modifications are intended to be included herein within the scope of the inventive concept. Therefore, the subject matter disclosed above should be considered as illustrative rather than restrictive, and the example embodiments added are intended to cover all such modifications, enhancements, and other embodiments that are within the spirit and scope of the inventive concept. Therefore, to the maximum extent permitted by law, the scope of the inventive concept should be determined by the broadest permissible interpretation of this disclosure, including the following example embodiments and their equivalents, and should not be limited or restricted by the foregoing detailed description.

Claims

1. 1. A camera tracking system for computer-assisted navigation during surgery, comprising: Camera bar, first and second tracking cameras mounted at spaced apart locations on said camera bar; a third tracking camera mounted at a position on the camera bar between the positions of the first and second tracking cameras and spaced a distance from a line extending through the centers of the first and second tracking cameras; and at least one processor operable to determine a pose of the set of physical objects in three dimensional space based on triangulation of positions of the set of physical objects in frames of video streams from the first, second, and third tracking cameras relative to positions of the first, second, and third tracking cameras spaced apart in three dimensional space; The set of physical objects is a physical reference array of spaced fiducials, and the at least one processor is adapted to: the physical reference array is further operable to compare a pattern of distances between the fiducials of the physical reference array in a frame of the video stream with a pattern of distances between the fiducials in each of a set of reference arrays registered in a database, to select from the set of reference arrays registered in the database a reference array having a pattern of distances between the fiducials similar to the pattern of distances between the fiducials of the physical reference array, and to determine a pose of the physical reference array based on the selected reference array. Camera tracking system.

2. The camera tracking system of claim 1 , wherein the third tracking camera is at the same distance from the first and second tracking cameras.

3. 2. The camera tracking system of claim 1, wherein the distance that the third tracking camera is spaced from the line is between one-quarter and one-half the distance between the centers of the first and second tracking cameras.

4. The at least one processor 2. The camera tracking system of claim 1, further comprising: based on the selected reference array, retrieve from the database a relationship in the three-dimensional space between the fiducials of the physical reference array registered in the database; and determine the pose of the physical reference array based on the relationship and a triangulation position of the fiducials of the physical reference array in frames of the video stream from the first, second, and third tracked cameras relative to the positions of the first, second, and third tracked cameras spaced apart in three-dimensional space.

5. The at least one processor further comprises: determining a first pose of the physical reference array of spaced apart fiducials in the three-dimensional space based on triangulated positions of the fiducials in frames of the video streams from the first and second tracking cameras relative to positions of the first and second tracking cameras spaced apart in the three-dimensional space; determining a second pose of the physical reference array of spaced apart fiducials in the three-dimensional space based on triangulated positions of the fiducials in frames of the video streams from the first and third tracking cameras relative to positions of the first and third tracking cameras spaced apart in the three-dimensional space; and The camera tracking system of claim 1 , further operative to determine an accuracy of calibration of at least the first and second tracked cameras based on a comparison of the first and second poses.

6. The processor further comprises:

6. The camera tracking system of claim 5, operative to initiate a process of calibrating orientation and / or yaw positioning of the first and second tracked cameras in response to the accuracy of the calibration not satisfying defined rules.

7. The set of physical objects is a physical reference array of spaced apart reference disks, and the at least one processor: detecting potential discrepancies in frames of the video streams from the video streams from the first, second, and third tracking cameras; estimating a position of a center of gravity of the disc for the frames of the video stream; verifying whether the positions of the centroids of the disks form a pattern which corresponds to the pattern of disks of a reference array registered in a database; responsive to successful verification of the disc, determining a gradient along an edge of the disc to generate a gradient ray extending towards a centre of gravity of the disc; adjusting the gradient rays to reduce distortion based on defined intrinsic parameters of lenses of the first, second, and third tracking cameras, and outputting distortion-compensated gradient rays; performing a double cone ellipse fitting on the distortion compensated gradient rays to determine locations of distortion compensated centroids of the disks, the location of the compensated centroid for each disk corresponding to where the distortion compensated gradient rays of the disks converge; and 2. The camera tracking system of claim 1, further operative to determine a pose of the physical reference array of spaced fiducials in the three-dimensional space based on a triangulation of positions of the distortion-compensated center of gravity of the disk in the video streams from the first, second, and third tracking cameras relative to positions of the first, second, and third tracking cameras spaced apart in the three-dimensional space.

8. 1. A camera tracking system for computer-assisted navigation during surgery, comprising: Camera bar, first and second tracking cameras mounted at spaced apart locations on said camera bar; a third tracking camera mounted at a position on said camera bar between the positions of said first and second tracking cameras and spaced a distance from a line extending through the centers of said first and second tracking cameras; a multi-planar reference array mountable to the surgical tool, the multi-planar reference array including a plurality of planes, each plane having an optically encoded pattern; A camera tracking system comprising at least one processor operative to decode the pattern of at least one of a plurality of the planes of the multi-plane reference array in frames of video streams from the first, second, and third tracked cameras, and to determine, based on a result of decoding the pattern, which of the plurality of planes the decoded pattern is a pattern of.

9. The at least one processor further comprises:

10. The camera tracking system of claim 8, further operable to obtain from a database a relationship between which of a plurality of said planes the pattern is in and positions between corners and / or edges of the pattern, and determine a pose of the multi-plane reference array based on the relationship and a triangulation of positions between the corners and / or edges of at least one pattern of a plurality of planes of the multi-plane reference array in frames of the video streams from the first, second and third tracked cameras spaced apart in three-dimensional space.

10. 9. The camera tracking system of claim 8, wherein the optically encoded pattern is a two-dimensional barcode pattern, and the at least one processor is further operative to decode the two-dimensional barcode pattern.

11. the at least one processor being part of a navigation controller; determining a pose of a surgical tool attached to the set of physical objects relative to a planned pose of the surgical tool when performing a surgical procedure on a patient; generating steering information based on a comparison of the determined pose of the surgical tool to the planned pose of the surgical tool, the steering information indicating where the surgical tool needs to be moved and angularly oriented to align with the planned pose when performing the surgical procedure; and The camera tracking system of claim 1 , operative to provide the steering information to a display device.

12. a surgical robot and at least one controller, The surgical robot includes: Robot base, a robotic arm connected to the robot base, the robotic arm configured to connect to an end effector that guides movement of a surgical tool; and at least one motor operably connected to move the robot arm relative to the robot base; the at least one controller is coupled to the at least one motor; determining a pose of the end effector relative to a planned pose of the end effector while guiding movement of the surgical tool to operate on a patient; and configured to generate steering information based on a comparison of the planned pose and the determined pose of the end effector; 2. The camera tracking system of claim 1, wherein the steering information indicates where the end effector needs to be moved to align with the planned pose such that the surgical tool is guided by the end effector along a planned trajectory towards the patient.

Images