Surgical navigation system and method for multi-modal tracking and object detection

Abstract

A surgical navigation system is provided that includes a tracker, a localizer, and a controller. The tracker is coupled to a surgical object and includes tracking elements arranged in a tracker geometry. The localizer is configured to track the tracker and detect parameters of the surgical object. The controller is in communication with the localizer and configured to utilize the localizer to detect a pose of the tracker geometry. The controller is further configured to receive a first parameter of the surgical object, create a temporary descriptor that describes an appearance of the surgical object, receive a second parameter of the surgical object, and update the temporary descriptor to create an updated temporary descriptor that describes the appearance of the surgical object. Finally, the controller is configured to track the surgical object based on a combination of the updated temporary descriptor and the pose of the tracker geometry.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and all benefits to U.S. Provisional Patent Application No. 63 / 622,670, filed January 19, 2024, the entire contents of which are hereby expressly incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to systems and methods for identifying and tracking components of a surgical system. Background Technology

[0004] Surgical navigation systems assist users in tracking / positioning objects in the operating room. For example, a navigation system assists surgeons in placing surgical instruments relative to the patient's anatomy. Typically, instruments and anatomical structures are tracked together in a manner that their relative movement is displayed on a monitor. Navigation systems usually include a tracking device attached to the object being tracked. A locator works in conjunction with the tracking device to determine the position of the tracking device and ultimately the position and / or orientation of the object. The navigation system then monitors the movement of the object via the tracking device.

[0005] Many navigation systems rely on an unobstructed line of sight between the tracking device and the locator's sensors. These systems also depend on the tracking device being positioned within the locator's field of view. Therefore, efforts have been made to reduce the likelihood of obstructing the line of sight between the tracking element and the sensors, and to maintain the tracking element within the locator's field of view. However, such navigation systems cannot prevent line-of-sight obstruction during surgical procedures due to the object moving out of the locator's line of sight, or to prevent the tracking device from moving out of the field of view.

[0006] Errors can occur when vision is obstructed or when the tracking device is out of sight. Typically, in such cases, navigation is interrupted, and an error message is communicated to the user until the tracking device is detected again or the navigation system is reset. This can lead to delays in surgical procedures. For example, once these errors occur, manipulators that rely on navigation data to autonomously position cutting tools relative to patient tissue must cease operation. This can significantly increase surgical procedure time, especially if difficulties arise during the restoration of vision. This contradicts the need in modern surgical practice to reduce surgical time in order to minimize the risk of infection and the risks associated with prolonged anesthesia.

[0007] Many navigation systems also rely on the specific geometry of the tracking devices to distinguish between a first tracking device and a second tracking device, and thus differentiate an object coupled to the first tracking device from another object coupled to the second tracking device. While effective, this approach depends on each tracking device having a unique geometry. If two visually indistinguishable tracking devices are used with these systems, it may be impossible to determine which tracking device is the first / second tracking device, or which object is coupled to the first / second tracking device. Furthermore, system complexity and cost increase due to the unique tracker geometry for each tracker.

[0008] Therefore, there is a need in the art for navigation systems and methods that overcome tracking interruptions between the tracking device and the locator, as well as problems arising from tracking two different objects using trackers with “identical” geometry. Summary of the Invention

[0009] According to a first aspect, a surgical navigation system is provided. The surgical navigation system includes: a tracker coupled to a surgical object, a locator, and a controller communicating with the locator. The tracker includes tracking elements arranged in tracker geometry. The locator is configured to track the tracker and detect parameters of the surgical object. The controller is configured to: detect the pose of the tracker geometry using the locator; receive a first parameter of the surgical object detected by the locator; create a temporary descriptor describing the appearance of the surgical object based on the first parameter; receive a second parameter of the surgical object detected by the locator; update the temporary descriptor based on the second parameter to create an updated temporary descriptor describing the appearance of the surgical object; and track the surgical object based on a combination of the updated temporary descriptor and the pose of the tracker geometry.

[0010] According to a second aspect, a surgical navigation system is provided for tracking a surgical object during a surgical procedure. The surgical navigation system includes: a tracker coupled to the surgical object and including tracker geometry; a locator configured to detect the tracker and detect the presence of the surgical object and parameters of the surgical object; and a controller communicating with the locator. The controller is configured to: receive the parameters of the surgical object; create a temporary descriptor describing the appearance of the surgical object based on the parameters of the surgical object; detect movement of the tracker and movement of the temporary descriptor using the locator; determine whether the movement of the tracker is sufficiently correlated with the movement of the temporary descriptor, and if correlated, create a tracking entity including the tracker geometry and the temporary descriptor; and track the movement of the surgical object based on the movement of the tracking entity.

[0011] According to a third aspect, a surgical navigation system is provided for tracking a surgical object during a surgical procedure. The surgical navigation system includes: a locator configured to detect the presence of the surgical object and detect parameters of the surgical object; and a controller communicating with the locator. The controller is configured to: receive the parameters of the surgical object from the locator at a first time; create a temporary descriptor describing the appearance of the surgical object based on the parameters received from the locator at the first time; receive the parameters of the surgical object from the locator at a second time; determine whether the parameters received at the first time are different from the parameters received at the second time, and if they are different, create an updated temporary descriptor describing the appearance of the surgical object based on the parameters received at both the first and second times; and track the pose of the surgical object based on the updated temporary descriptor.

[0012] According to a fourth aspect, a surgical navigation system is provided. The surgical navigation system includes: a first tracker coupled to a first surgical object and including tracking elements arranged in the geometry of the first tracker; a second tracker coupled to a second surgical object and including tracking elements arranged in the geometry of the first tracker; a locator configured to track the first tracker and the second tracker, and to detect parameters associated with the first surgical object and the second surgical object; and a controller communicating with the locator. The controller is configured to: detect the pose of the first tracker and a first parameter associated with the first surgical object using the locator; create a first tracking entity based on the pose of the first tracker and the first parameter; detect the pose of the second tracker and a second parameter associated with the second surgical object using the locator; create a second tracking entity based on the pose of the second tracker and the second parameter; and track the poses of the first surgical object and the second surgical object based on the movement of the first tracking entity and the second tracking entity, respectively.

[0013] According to a fifth aspect, a surgical navigation system is provided for tracking a surgical object during a surgical procedure. The surgical navigation system includes: a locator configured to detect the presence of the surgical object and detect parameters of the surgical object; a database containing descriptors of the surgical object; and a controller communicating with the locator. The controller is configured to: receive the parameters of the surgical object from the locator; compare the parameters of the surgical object with the descriptors stored in the database; and determine, based on the comparison, that no descriptor describing the appearance of the surgical object exists in the database. In response, the controller is configured to: create a temporary descriptor describing the surgical object based on the detected parameters of the surgical object; and track the pose of the surgical object based on the temporary descriptor.

[0014] According to a sixth aspect, a surgical navigation system is provided for tracking a surgical object during a surgical procedure. The surgical navigation system includes: a tracker coupled to the surgical object, a locator including a first sensor and a second sensor, and a controller communicating with the locator. The locator is configured to: detect the attitude of the tracker in a first tracking mode using the first sensor; and detect parameters of the surgical object in a second tracking mode using the second sensor. The controller is configured to: receive the attitude of the tracker in the first imaging mode; receive the parameters of the surgical object in the second imaging mode; create a temporary descriptor describing the appearance of the surgical object based on the detected parameters of the surgical object; and track the surgical object based on the attitude of the tracker and the temporary descriptor.

[0015] According to a seventh aspect, a surgical navigation system is provided for tracking a surgical object during a surgical procedure. The surgical navigation system includes: a tracker coupled to the surgical object; a locator including a NIR sensor and a visible light sensor; and a controller communicating with the locator. The locator is configured to: detect the attitude of the tracker in an NIR space using the NIR sensor; and detect parameters of the surgical object in a visible light space using the visible light sensor. The controller is configured to: receive the attitude of the tracker in the NIR space; receive the parameters of the surgical object in the visible light space; create a temporary descriptor describing the appearance of the surgical object based on the detected parameters; determine that the attitude of the tracker is no longer known in the NIR space; and, in response, track the surgical object based on the parameters of the surgical object in the visible light space.

[0016] According to an eighth aspect, a surgical navigation system is provided for tracking a surgical object during a surgical procedure. The surgical navigation system includes: a tracker coupled to the surgical object, a locator including a first sensor and a second sensor, and a controller communicating with the locator. The locator is configured to: detect the tracker's pose in a first tracking mode using the first sensor; and detect parameters of the surgical object in a second tracking mode using the second sensor. The controller is configured to: receive the tracker's pose detected in the first imaging mode; receive the parameters of the surgical object detected in the second imaging mode; create a temporary descriptor describing the appearance of the surgical object based on the detected parameters of the surgical object; create a tracking entity describing the association between the tracker and the surgical object based on the tracker's pose and the temporary descriptor; and track the surgical object based on the pose of the tracking entity. The controller is further configured to: disable the tracking entity in response to determining that the tracker is occluded from the perspective of the locator; re-enable the tracking entity in response to determining that the tracker is no longer occluded from the perspective of the locator and that the tracker is still attached to the surgical object; and resume tracking of the surgical object based on the pose of the tracking entity.

[0017] Any aspect of the foregoing may be combined, in whole or in part, with any other aspect. Whether combined, in whole or in part, any aspect of the foregoing may be further combined, in whole or in part, with any of the following implementations.

[0018] In some implementations, the system may be configured to learn more information about the appearance of the surgical object over time. In such implementations, the first parameter may be detected by the locator at a first time, and the second parameter may be detected by the locator at a second time. In some cases, the first time is before the second time. Furthermore, the controller may be configured to: detect a third parameter of the surgical object; update the updated temporary descriptor based on the third parameter; and track the pose of the surgical object based on a combination of the updated temporary descriptor and the pose of the tracker geometry. In some cases, the second parameter may be implemented as a motion parameter, and the controller may be configured to: detect pose changes in the tracker geometry to create the updated temporary descriptor based on determining that the motion parameter is sufficiently consistent with the pose changes of the tracker geometry. In some cases, the controller may be configured to replace the first parameter with the second parameter in response to determining that the second parameter is inconsistent with the first parameter.

[0019] In some implementations, at least one of the parameters may be a physical parameter of the surgical object. The physical parameter may be the object's geometry or shape, contour, color, envelope, surface roughness, surface markings, or color or shading. When multiple parameters are determined, the first parameter may be a first color of the surgical object, and the second parameter may be a second color of the surgical object. In some cases, the second parameter is a color change of the surgical object. In some implementations, at least one of the parameters may be a motion parameter of the surgical object. The motion parameter may be the surgical object's velocity or rate, acceleration, rotation, or displacement.

[0020] In some implementations, the controller may be configured to create a tracking entity based on the (updated) temporary descriptor and the tracker geometry. The temporary descriptor and / or the updated temporary descriptor may include either physical or motion parameters. In some cases, the controller may be configured to track the surgical object by tracking the tracking entity based on a combination of the pose of the updated temporary descriptor and the tracker geometry.

[0021] In some implementations, the controller may be configured to detect environmental conditions and normalize the detected parameters based on those conditions. The environmental conditions may be the illumination of the surgical object, the illumination of the tracking geometry, and / or the overall illumination of the space within the locator's line of sight.

[0022] In some implementations, the controller can be configured to operate in multiple states. In such implementations, the controller can be configured to: operate in a first state, in which the controller associates the tracker geometry with the surgical object after a triggering event; and operate in a second state, in which the controller automatically associates the tracker geometry with the surgical object. The triggering event may be from user input or the surgical object being in a predefined pose.

[0023] In some implementations, the locator may be able to detect multiple light patterns. In such implementations, the locator may include a first sensor configured to detect visible light and a second sensor configured to detect infrared or near-infrared light. The pose of the tracker geometry may be detected by the second sensor, and / or the first parameter may be detected by the first sensor. In these implementations, the controller may be configured to detect the pose of the tracker geometry in a first coordinate system and detect the first parameter in a second coordinate system. The controller may be configured to register at least one of the first and second coordinate systems to a third coordinate system, and / or the controller may be configured to register one of the first and second coordinate systems to the other of the first and second coordinate systems.

[0024] In some implementations, the controller may be configured to determine that the pose of the tracker geometry cannot be detected by the locator. In such implementations, the controller may be configured to track the pose of the surgical object solely based on the updated temporary descriptor in response to determining that the pose of the tracker geometry cannot be detected by the locator. In some cases, the controller may be configured to determine that the pose of the tracker geometry can be detected by the locator after determining that the pose of the tracker geometry cannot be detected by the locator. The controller may be configured to track the pose of the surgical object based on at least one of the updated temporary descriptor and the pose of the tracker geometry in response to determining that the pose of the tracker geometry can be detected by the locator. Furthermore, the controller may be configured to register or re-register the tracker to the surgical object by comparing new parameters of the surgical object with the updated temporary descriptor.

[0025] In some implementations, the controller may utilize a machine learning module. For example, the controller may provide the first parameter and the second parameter to the machine learning module, and the machine learning module may output computed parameters of the surgical object based on the first parameter and the second parameter. In some cases, the controller may be configured to update the updated temporary descriptor based on the computed parameters to create a learned temporary descriptor describing the association between the surgical object and the detected tracker geometry. The system may track the pose of the surgical object based on a combination of the learned temporary descriptor and the pose of the tracker geometry. The controller may be configured to: receive a third parameter of the surgical object; compare the computed parameters with the third parameter; and update the learned temporary descriptor based on the comparison to create an updated learned temporary descriptor describing the association between the surgical object and the detected tracker geometry. The system may track the pose of the surgical object based on a combination of the updated learned temporary descriptor and the pose of the tracker geometry.

[0026] In some implementations, the system may be configured to track multiple surgical objects by tracking multiple trackers. In such implementations, the trackers may be implemented as a first tracker, and the surgical objects may be implemented as a first surgical object. In some cases, a second tracker may be coupled to a second surgical object and include tracking elements arranged with the same tracker geometry as the first tracker. To distinguish the first tracker from the second tracker (and vice versa), the locator may be configured to track the first tracker and the second tracker, and detect parameters associated with the first surgical object and the second surgical object. Furthermore, the controller may be configured to: detect the pose of the first tracker and a first parameter associated with the first surgical object using the locator; create a first tracking entity based on the pose of the first tracker and the first parameter; detect the pose of the second tracker and a second parameter associated with the second surgical object using the locator; create a second tracking entity based on the pose of the second tracker and the second parameter; and track the poses of the first surgical object and the second surgical object based on the movement of the first tracking entity and the second tracking entity, respectively.

[0027] In some implementations, the system may include a database containing descriptors of surgical objects, and confirm that the surgical object detected by the locator does not match a previously detected surgical object. In such implementations, the controller may be configured to: compare the first parameter of the surgical object with the descriptors stored in the database; determine, based on the comparison, that a descriptor describing the appearance of the surgical object does not exist in the database; and, in response, create a temporary descriptor describing the surgical object based on the detected parameters of the surgical object. In some cases, the updated temporary descriptor may be implemented as a database entry containing at least the first parameter and the second parameter.

[0028] In some implementations, the system may include a display communicating with the controller, the display being configured to depict the surgical object relative to a surgical target. In some cases, the surgical object may be represented by computer graphics, and the computer graphics may be depicted relative to the surgical target. The computer graphics may be based on one of the temporary descriptor and the parameters. The computer graphics may include a unique identifier identifying the surgical object.

[0029] In some implementations, the system can be configured to determine information about a surgical object and / or tracker that has been occluded and has left the locator. For example, the controller can be configured to determine that the tracker is still associated with the surgical object by comparing new parameters of the surgical object with the temporary descriptor. In this example, the parameters of the surgical object can be implemented as initial parameters, and the controller can be configured to compare the new parameters with the temporary descriptor by comparing the new parameters with the initial parameters.

[0030] The operations described with reference to the controller (such as any combination of aspects and implementations described herein) may be implemented as a computer program product or instructions stored on a non-transitory computer-readable medium. For the computer program product, it may be configured to be executed by the controller to cause the controller to perform the operations. For the non-transitory computer-readable medium, it may be connected to the controller, and the instructions stored on the non-transitory computer-readable medium may be configured to cause the controller to perform the operations. Attached Figure Description

[0031] The advantages of the invention will become readily apparent as the invention is better understood by taking into account the following detailed description in conjunction with the accompanying drawings.

[0032] Figure 1 This is a perspective view of one implementation of a surgical system that includes a robotic manipulator and a navigation system including a locator.

[0033] Figure 2A It is based on one implementation method. Figure 1 The surgical system and the perspective view of the tracking entity created by the navigation system.

[0034] Figure 2B It is based on an implementation method, such as Figure 2A The tracking entity represents Figure 1 A perspective view of the surgical system.

[0035] Figure 3 yes Figure 1 A block diagram of one implementation of a software suite that can be used in a navigation system.

[0036] Figure 4 This is a flowchart of a method for tracking objects in the operating room, based on one implementation.

[0037] Figure 5 It is a flowchart of a tracking manager process for managing tracking entities, based on one implementation method.

[0038] Figure 6 It is a schematic diagram of a tracking entity created by a navigation system, based on one implementation method.

[0039] Figure 7 It is a schematic diagram of various tracking entities created by a navigation system, based on one implementation method.

[0040] Figure 8 This is a flowchart of a navigation manager process for tracking surgical objects, based on one implementation method.

[0041] Figure 9 This is a flowchart of a method for tracking objects in the operating room, based on one implementation. Detailed Implementation

[0042] I. Example System Overview

[0043] Referring to the accompanying drawings, which are shown throughout several views and where similar numbers indicate similar or corresponding parts, a surgical navigation system (hereinafter referred to as the "System") and a method for operating the surgical navigation system are illustrated throughout the drawings.

[0044] refer to Figure 1An example configuration of an operating room or surgical suite for performing medical procedures on a patient is illustrated. The illustrated configuration includes a surgical navigation system 100, a surgical robot 200, surgical instruments 250, and an implant IM to be placed in the patient's body. The surgical navigation system 100 is configured to track the movement of various objects in the operating room. Such objects include, for example, the patient, the surgical robot 200 and / or the surgical instruments 250, and other objects. The surgical navigation system 100 tracks these objects to display their relative positions and orientations to the surgeon, and in some cases to control or constrain the movement of the surgical instruments 250 relative to a virtual cutting boundary associated with the patient.

[0045] The surgical navigation system 100 may include a computer cart assembly 102 housing a navigation controller 104. A navigation interface operatively communicates with the navigation controller 104. The navigation interface includes a first display 106 adapted to be located outside a sterile area, and a second display 107 adapted to be located within a sterile area. The displays 106 and 107 are adjustablely mounted to the computer cart assembly 102. First and second input devices (not shown), such as a keyboard and mouse, may be used to input information into the navigation controller 104 or otherwise select / control certain aspects of the navigation controller 104. Other input devices are contemplated, including a touchscreen 108, gesture control, or voice activation. The displays may be implemented as head-mounted displays configured for extended or augmented reality and adapted to display any graphics or images described herein in a manner superimposed or overlaid on a real-world view or video.

[0046] In addition, the navigation system 100 includes a locator 110 that communicates with the navigation controller 104. In the illustrated implementation, the locator 110 is a multimodal locator and includes an optical (video) camera unit 112 and an infrared and / or near-infrared (NIR) sensor unit 114. The optical camera unit 112 includes one or more sensors 118 adapted to sense light in the visible spectrum and can be configured as a video camera, machine vision, or computer vision system. The visible light sensor 118 is configured to detect color and / or generate data that can be used to create a depth map. The infrared sensor unit 114 may include one or more sensors 119 adapted to sense light in the infrared or near-infrared spectrum. The locator 110 may include an illuminator configured to radiate infrared light into a surgical field such that the infrared sensor 119 can detect reflected or backscattered radiation. A housing 116 accommodates the optical camera unit 112 and the infrared sensor unit 114.

[0047] Positioner 110 communicates with navigation controller 104. In some implementations, camera controller 120 facilitates communication between sensors 118, 119 and navigation controller 104 via a wired or wireless connection (not shown). In other implementations, sensors 118, 119 may communicate directly with navigation controller 104. Processing of signals from visible light sensor 118 and IR sensor 119 can be performed at navigation controller 104 to process both navigation information and machine vision information. An example of navigation system 100 is described in U.S. Patent No. 9,008,757 entitled “Navigation System Including Optical and Non-Optical Sensors,” which is hereby incorporated by reference.

[0048] The navigation controller 104 may be a personal computer or a laptop computer. The navigation controller 104 may have a display 106, a central processing unit (CPU) and / or other processor, memory (not shown), and storage devices (not shown). The navigation controller 104 may be loaded with software that converts signals received from the camera unit 112 and the infrared sensor unit 114 into data representing the position and orientation of the tracked object. Additionally, the software converts signals received from the camera unit 112 into object-identifiable data, such as object identification via the optical camera unit 112. Position and orientation signals and / or data are transmitted to the navigation controller 104 for the purpose of tracking the object. Alternatively, all computer processing components and functionality may be integrated into a single processing unit or distributed among or between multiple processing units. Furthermore, although described herein as being performed at a specific computer or controller, those skilled in the art will understand that any processing task may be performed at or by other computers or controllers. The computer cart assembly 102, display 106, and camera unit 112 may be similar to those described in U.S. Patent No. 7,725,162 entitled “Surgery System”, issued May 25, 2010, to Malackowski et al., which is hereby incorporated by reference.

[0049] The surgical navigation system 100 can be used to track the posture of multiple tracking devices 150 (referred to herein as trackers 150). In the illustrated implementation, one tracker 150 is coupled to a first anatomical position of the patient, another tracker 150 is coupled to a second anatomical position of the patient, another tracker 150 is coupled to a surgical instrument 250, another tracker 150 is coupled to a surgical robot 200 (or an instrument 250 coupled thereto), and other trackers 150 are contemplated.

[0050] Tracker 150 can be an active or passive tracker. An active tracker requires a power source and has an array of reference markers (also called tracking elements or markers) that actively generate and emit radiation at wavelengths detectable by visible light sensor 118. The reference markers of an active tracker can be light-emitting diodes (LEDs), including, for example, infrared LEDs. The active reference marker array can be "always on" or operable to be selectively triggered (i.e., emit radiation) based on and in response to commands from surgical navigation system 100. In such selectively triggered active trackers, the tracker can communicate with navigation controller 104 of surgical navigation system 100 via a wired or wireless connection. In other examples, an active tracker may include active electromagnetic elements, active radio frequency elements, etc.

[0051] In an alternative implementation, tracker 150 may include a passive tracker. An active tracker may be battery-powered using an internal battery, or may have wires receiving power from navigation controller 104, which may receive external power. Passive tracker arrays typically do not require a power supply. Passive trackers may include barcodes, QR codes, or any computer-detectable pattern. Passive trackers may include passive reflective markers, radiopaque markers, passive electromagnetic components, passive radio frequency components, etc.

[0052] Furthermore, each tracker 150 may include a tracker geometry. For example, if the tracker 150 includes an array of optical elements, these optical elements may be arranged relative to each other to form a tracker geometry. In some implementations, each tracker 150 has a common tracker geometry. In other implementations, each tracker 150 has a unique tracker geometry. In a further implementation, multiple trackers 150 may each have a common tracker geometry, while other trackers 150 may have unique tracker geometries that differ from the common tracker geometry.

[0053] In some examples, navigation system 100 and / or locator 110 are radio frequency (RF) based. For example, navigation system 100 may include an RF transceiver coupled to navigation controller 104. Here, tracker 150 may include an RF transmitter or transponder, which may be passive or actively activated. The RF transceiver transmits an RF tracking signal, and the RF transmitter responds with an RF signal, such that the tracking status is transmitted to (or interpreted by) navigation controller 104. The RF signal may have any suitable frequency. The RF transceiver may be positioned at any suitable location to effectively track objects using the RF signal. Furthermore, examples of RF-based navigation systems may have structural configurations different from those illustrated throughout the figures for navigation system 100.

[0054] In some examples, navigation system 100 and / or locator 110 are based on electromagnetic (EM). For example, navigation system 100 may include an EM transceiver coupled to navigation controller 104. Here, tracker 150 may include EM components attached thereto (e.g., various types of magnetic trackers, electromagnetic trackers, inductive trackers, etc.), which may be passive or actively energized. The EM transceiver generates an EM field, and the EM components respond with EM signals, such that the tracking status is transmitted to (or interpreted by) navigation controller 104. Navigation controller 104 may analyze the received EM signals to correlate with the relevant status. Also here, examples of EM-based navigation systems may have structural configurations different from those illustrated throughout the figures for navigation system 100.

[0055] In some examples, navigation system 100 and / or locator 110 may be based on one or more other types of tracking systems. For example, an ultrasound-based tracking system coupled to navigation controller 104 may be provided to acquire ultrasound images of markers defining trackable features 150, such that tracking status is transmitted to (or interpreted by) navigation controller 104 based on the ultrasound images. As a further example, a fluorescence fluoroscopy-based imaging system (e.g., a C-arm) coupled to navigation controller 104 may be provided to acquire X-ray images of radiopaque markers defining trackable features, such that tracking status is transmitted to (or interpreted by) navigation controller 104 based on the X-ray images.

[0056] Furthermore, in some examples, a machine vision tracking system (e.g., one or more charge-coupled devices) coupled to the navigation controller 104 may be provided to acquire 2D and / or 3D machine vision images of structural features defining the trackable feature 150, such that the tracking status is transmitted to (or interpreted by) the navigation controller 104 based on the machine vision images. Ultrasonic, X-ray, and / or machine vision images may be 2D, 3D, or a combination thereof, and may be processed by the navigation controller 104 in near real-time to determine the tracking status of the tracker 150.

[0057] Without departing from the scope of this disclosure, various types of tracking and / or imaging systems may define the locator 110 and / or form part of the navigation system 100. Furthermore, the navigation system 100 and / or the locator 110 may have other suitable components or structures not specifically listed herein, and the various techniques, methods, and / or components described herein with respect to the optical-based navigation system 100 shown throughout the accompanying drawings may be implemented or provided for any other example of the navigation system 100 described herein. For example, the navigation system 100 may utilize inertial tracking only and / or combinations of different tracking techniques, sensors, etc. Other configurations are contemplated.

[0058] As described above, the navigation system 100 can be used to track the surgical robot 200. In some implementations, the surgical robot 200 includes a base 202 and a manipulator 204 comprising multiple links and joints. The base 202 may be fixed to a point in the operating room, such as an operating table. Alternatively, the base 202 may be easily movable, allowing the surgical robot to be repositioned within the operating room. In one example, the surgical robot 200 may have a configuration such as the robot manipulator described in U.S. Patent No. 10,327,849 entitled "Robotic System and Method for Backdriving the Same," the contents of which are hereby incorporated by reference in their entirety.

[0059] The surgical robot 200 may house a manipulator controller 208 or other type of control unit. The manipulator controller 208 may include one or more computers, or any other suitable form of controller that directs the movement of the manipulator 204 and / or the base 202. The manipulator controller 208 may have a central processing unit (CPU) and / or other processors, memory, and storage devices. The processor may include one or more processors for controlling the operation of the manipulator 204. The processor may be any type of microprocessor, multiprocessor, and / or multicore processing system. The manipulator controller 208 may additionally or alternatively include one or more microcontrollers, field-programmable gate arrays, system-on-a-chip, discrete circuitry, and / or other suitable hardware, software, or firmware capable of performing the functions described herein. The term "processor" is not intended to limit any implementation to a single processor. The surgical robot 200 may also include a user interface (UI) with one or more displays and / or input devices (e.g., push buttons, keyboard, mouse, microphone (voice-activated), gesture control devices, touchscreen, etc.).

[0060] Surgical robot 200 can be used to control surgical instrument 250. For this purpose, surgical robot 200 may include an end effector 210 configured to couple surgical instrument 250 to manipulator 204. In some implementations, manipulator 204 and instrument 250 may be arranged similarly to that shown in U.S. Patent No. 9,566,121, filed March 15, 2014, entitled “End Effector of a Surgical Robotic Manipulator,” which is hereby incorporated by reference. In other implementations, surgical instrument 250 is attached to manipulator 204, as shown in U.S. Patent No. 9,119,655, published September 1, 2015, entitled “Surgical Manipulator Capable of Controlling a Surgical Instrument in Multiple Modes,” the disclosure of which is hereby incorporated by reference.

[0061] II. Example Tracing Method

[0062] As described above, each of the respective trackers 150 can be active and / or passive. Typically, the locator 110 distinguishes the trackers 150 (and therefore the objects to which they are attached) from one another based on the specific characteristics of each tracker 150. For example, the locator 110 may distinguish one tracker from another based on the shape of the tracker 150 and / or the arrangement of its tracking elements (i.e., unique tracker geometry). However, in the present system 100, each tracker 150 may have the same shape and / or the same arrangement of its tracking elements. Therefore, the locator 110 may not be able to identify / track the objects to which the tracker 150 is attached based solely on the details of the tracker 150.

[0063] Instead of relying on tracker details to identify / track objects in the operating room, navigation system 100 can utilize locator 110 to determine parameters of objects coupled to tracker 150 to distinguish objects from one another, allowing system 100 to independently identify and track each object. Object parameters may include physical parameters such as geometry or shape, contour, envelope, surface roughness, surface markings, or the object's color or shading. Parameters may also include motion parameters of the object, such as the surgical object's velocity or rate, acceleration, rotation, and / or displacement.

[0064] Now for reference Figure 2A and Figure 2B Together with each of systems 100, 200, device 250, and implant IM, it is shown Figure 1 The operating room. Additionally... Figure 2A and Figure 2B This includes various bounding boxes surrounding each of elements 200, 250, and IM (excluding navigation system 100). These bounding boxes are abstract visualizations intended to depict how navigation system 100 views each element 200, 250, and IM. More specifically, each bounding box represents a tracking entity 300 based on the object's parameters and the tracker 150 to which it is attached. Tracking entity 300 describes the association between the object and tracker 150. More specifically, tracking entity 300 includes contextual information associated with tracker 150, enabling tracker 150 to be distinguished from other trackers 150. For example, in the case where tracker 150 is coupled to an object, the contextual information includes a description of the object as detected by visible light sensor 118. For objects that do not include tracker 150, such as implanted IMs, tracking entity 300 is based solely on the object's parameters.

[0065] Tracking entity 300 is created by navigation system 100 and allows system 100 to track objects visible to locator 110. Unlike some other navigation systems and methods, current system 100 does not rely on pre-stored data to identify and track objects in the operating room. Instead, system 100 relies on tracking entity 300 to track objects associated with that tracking entity 300. Tracking entity 300 can be created when an object moves into the field of view of locator 110. For example, if surgical instrument 250 moves into the field of view of locator 110 during a procedure, navigation system 100 will create tracking entity 300 for instrument 250 intraoperatively. This allows navigation system 100 to identify and track any object that can be recognized by locator 110, not just objects that system 100 knows based on pre-stored data.

[0066] refer to Figure 3An example of a software suite 310 operable by navigation controller 104 is shown. The illustrated software suite 310 is employed by navigation controller 104 to create tracking entities 300 and use said entities 300 to track objects in an operating room. Software suite 310 may include an object detector 312, a motion detector 316, a feature extractor 314, a tracking database 318, a tracking manager 320, a recovery module 322, and a navigation manager 324. Although software suite 310 is illustrated and described as part of navigation controller 104, it is further contemplated that any element of software suite 310 may be incorporated into other computing components. Software suite 310 or a portion thereof may also reside in a remote computing component. For example, a portion of software suite 310 may reside on navigation controller 104, while the remainder of software suite 310 may reside in a cloud computing environment. Software suite 310 may also include and / or communicate with an online learning module 326. Similar to software suite 310, online learning module 326 may reside on navigation controller 104 (or other components of system 100). Alternatively, module 326 may exist in a remote computing component.

[0067] Each element of the software suite 310 can be used to create and track one or more tracking entities 300. The object detector 312 applies an object detection algorithm to the image captured by the locator 110 to detect objects present in the operating room (i.e., present in the image). In one implementation, the object detector 312 determines a cross-union metric and compares this metric with a confidence threshold to determine whether a feature present in the image captured by the locator 110 is an object for the purpose of creating the tracking entity 300. In this implementation, any feature in the image with a confidence score higher than the confidence threshold is considered an object by the object detector 312.

[0068] Both motion detector 316 and feature extractor 314 are used to determine parameters of the object detected in object detector 312. Motion detector 316 can use any suitable method to track the movement of the object in the operating room. For example, motion detector 316 can utilize Kalman filtering to estimate and / or predict the motion associated with the object. On the other hand, feature extractor 314 can use any suitable method to extract features of the object in the operating room from the image captured by locator 110.

[0069] The navigation controller 104 writes entries to the tracking database 318 to create tracking entities 300 based on objects detected by the object detector 312. Alternatively, the controller 104 may determine that a tracking entity already exists for the object and invoke and / or modify the entry in the database 318 associated with the existing tracking entity 300. Furthermore, the navigation controller 104 may invoke the tracking manager 320 and the recovery module 322 to control the tracking entity 300. The tracking manager 320 may output, modify, deactivate, and / or delete the tracking entity 300 based on objects detected in images captured by the locator 110 and associated parameters. The recovery module 322 may reactivate a previously created tracking entity 300 based on the object and associated parameters.

[0070] During navigation of surgical objects, such as surgical robots 200, surgical instruments 250, and / or other objects present in the operating room, navigation controller 104 may invoke navigation manager 324. As described in more detail below, navigation manager 324 is typically configured to control system 100 how to track surgical objects using visible light sensor 118 and IR sensor 119 of locator 110.

[0071] The tracking database 318 has been described as storing tracking entities 300. In some implementations, tracking entities 300 include temporary descriptors describing the appearance of objects according to locator 110. In this implementation, each tracking entity 300 contains a corresponding temporary descriptor and a corresponding tracker 150. If an object does not have a tracker 150 coupled to it, the tracking entity 300 associated with that object does not include tracker 150. The temporary descriptor typically includes all parameters of the object as detected by locator 110. In one example, the temporary descriptor includes a two-dimensional / three-dimensional model of the object created based on the shape of the object captured by locator 110. In this example, if tracker 150 is coupled to the object and the parameters of the object match the parameters of the temporary descriptor, system 100 knows that the object exists in the image data. Furthermore, the temporary descriptor can be updated when system 100 determines other parameters of the object.

[0072] In some implementations, the tracking database 318 is structured such that each tracking entity 300 is stored as an entry in the database 318. Here, temporary descriptors and trackers are each stored as sub-entries of the entry representing the tracking entity 300, and parameters are stored as sub-entries of the temporary descriptor sub-entries. For example, the tracking database 318 may contain a series of rows and columns. In this example, a row may represent a tracking entity 300. The first column included in the row may contain a unique ID, the second column may contain a representation of the tracker 150 (e.g., tracker geometry), and the third column may contain a temporary descriptor of the object (and therefore parameters).

[0073] As described above, each temporary descriptor includes at least one parameter of the object it is associated with and represents a description of each object as understood by navigation system 100. Navigation system 100 uses temporary descriptors to track objects and determine which object each of the trackers 150 is coupled to. Although the tracking entity 300 is shown in the figure as including parameters that only describe the shape of the object, this is merely for the sake of simplicity and clarity. The tracking entity / temporary descriptor may instead include other parameters of the object that can be detected by locator 110.

[0074] For example, when the object is a surgical instrument 250, the temporary descriptor may include parameters such as the outer contour / shape of the instrument 250, the color of the instrument 250, surface markings on the housing of the instrument 250, and the position of the instrument 250. These parameters are continuously updated as the locator 110 detects new and / or changed parameters. In the current example, the navigation system 100 may initially create a temporary descriptor using only one of the aforementioned parameters, such as the outer contour / shape of the instrument 250. Subsequently, the navigation system 100 may detect the color of the instrument 250 and surface markings on its housing, and add these parameters to the temporary descriptor to create an updated temporary descriptor. After this, the system 100 may detect the position of the instrument 250 as, for example, in the center of the operating room, such as... Figures 1 to 2B As shown, the location is added as another parameter to the (updated) temporary descriptor. If the surgeon moves between the locator 110 and the instrument 250 such that the instrument 250 is briefly obscured and out of the locator 110's field of vision, the system 100 can re-identify the instrument 250 by comparing the parameters of the instrument 250 with the parameters of the (updated) temporary descriptor. If the instrument 250 does not change and / or move during its brief obscuration out of the locator 110's field of vision, the system 100 can determine that all / sufficient parameters of the instrument 250 match the temporary descriptor previously associated with the instrument 250, and re-associate the (updated) temporary descriptor with the instrument 250.

[0075] A temporary descriptor for the surgical instrument 250 or any other object detectable by the positioner 110 may also include motion parameters of the instrument 250. Continuing with the example above, and referring to... Figures 1 to 2BThe surgeon may move instrument 250 behind the patient, making it invisible to locator 110 as instrument 250 moves downwards. In this case, the temporary descriptor may include the last known position of instrument 250 and its downward velocity. If the surgeon moves instrument 250 back upwards and into the field of view of locator 110, locator 110 can detect the new position and / or velocity of instrument 250, and the system can determine that instrument 250 is the same instrument that locator 110 previously could not see, and re-associate the temporary descriptor with instrument 250. This is achieved by comparing the new position / velocity detected when instrument 250 is moved back into the field of view of locator 110 with the most recent position parameter of the temporary descriptor.

[0076] The temporary descriptor of an object may include parameters detected by locator 110 at different times, such as different motion parameters as described in the example above. In the example above, the shape of instrument 250 may have been detected at a first time, and surface markings may have been detected at a second time after the first time. If system 100 determines that the object has changed over time, system 100 may even replace some of the parameters included in the object's temporary descriptor. For example, locator 110 may have detected that instrument 250 is primarily white at a first time. However, at a second time, locator 110 may have detected that the instrument is now half red and half white (e.g., due to blood covering part of instrument 250). System 100 may determine that these two parameters are inconsistent with each other and replace most of the white parameter in the associated temporary descriptor with the half-red, half-white parameter detected at the second time.

[0077] The online learning module 326 may include a machine learning algorithm configured to be trained on image data collected by the navigation system 100. As a result of such training, the online learning module 326 may be able to calculate additional parameters of an object based on parameters of a previously detected object. These additional parameters generated by the online learning module 326 are referred to herein as computed parameters. In one example, the locator 110 may detect a first parameter, which includes the shape of the device 250 from a first viewpoint. The first parameter may be input to the online learning module 326, and the machine learning algorithm may generate computed parameters corresponding to the shape of the device 250 from a second viewpoint. In another example, the locator 110 may detect the first and second parameters as the shape of the device 250 from a first viewpoint and the color of the side of the device 250 facing the locator 110. These parameters may be input to the machine learning algorithm, and the algorithm may generate a first computed parameter as the shape of the device 250 from a second viewpoint, and a second computed parameter as the color of the opposite side of the device 250 (not facing the locator 110). These calculated parameters can be added to a (possibly previously updated) temporary descriptor of instrument 250 to create a learning-type temporary descriptor. Like other parameters, system 100 can update / change the calculated parameters if a new parameter inconsistent with the calculated parameters is detected.

[0078] refer to Figure 4 The present invention describes an example method 400 for tracking objects in an operating room. Method 400 may be executed entirely by navigation controller 104, partially by navigation controller 104 and partially by other components of system 100, entirely by components of system 100 other than navigation controller 104, partially by navigation controller 104 and partially by a remote computing unit, or entirely by a remote computing unit. For clarity and simplicity, method 400 is described as being typically executed by system 100.

[0079] Method 400 begins at 404: Image data is received from locator 110. The image data includes optical image data from optical camera unit 112 and IR image data from infrared sensor unit 114, respectively. The image data represents the operating room as captured by locator 110. At 408, the image data is normalized. More specifically, the image data may be modified to account for varying environmental conditions in the operating room, such as changes in room brightness. As described in more detail below, image normalization may be necessary to ensure that the parameters of the detected object are not affected by environmental conditions.

[0080] At 412, image data is input to object detector 312, and object detector 312 attempts to detect any / all objects present in the image data. Assuming an object is detected in the image data, the method proceeds to 416. At 416, feature extractor 314 and motion detector 316 are applied to the image data. Based on the outputs from feature extractor 314 and motion detector 316, system 100 determines parameters associated with the detected objects in the image data.

[0081] At 418, system 100 compares the parameters determined at 416 with any existing temporary descriptors. If no existing temporary descriptor exists (e.g., this is the first object introduced into system 100), the step at 418 can be skipped instead. The comparison at 418 may include a comparison of the parameters of the detected object with the parameters included in existing temporary descriptors associated with each tracking entity 300 stored in the tracking database 318. At 420, system 100 uses the comparison performed at 418 to determine whether the detected object in the image data matches any existing tracking entity 300. If the parameters of the detected object match a temporary descriptor of one of the existing tracking entities 300, the method proceeds to 424A.

[0082] At 424A, an existing tracking entity is recovered, for example, via recovery module 322. That is, if the parameters of the object do not match the parameters of any existing temporary descriptor, the method proceeds to 424B. At 424B, system 100 creates a new tracking entity 300 including a new temporary descriptor based on the detected parameters of the object and associates the new tracking entity 300 with the object. If tracker 150 is coupled to the object, the new tracking entity 300 also contains the relationship between the object / temporary descriptor and tracker 150. This step may include detecting the pose of tracker 150, which allows system 100 to associate the pose of tracker 150 with the pose of the temporary descriptor. And the pose of tracking entity 300 is determined by determining the pose of tracker 150. In some implementations, the tracking database is cleared at the end of each use of system 100. In these implementations, the determination at 420 is skipped during the first iteration of method 400 and method 400 instead proceeds from 416 to 424B.

[0083] Depending on the configuration of navigation system 100, the steps at 424A and 424B can be performed in different ways. In one implementation, if the parameters of the detected object do not match the parameters of any existing tracking entity, system 100 automatically attempts to create a new tracking entity for the detected object. In other implementations, system 100 waits for a trigger condition before attempting to create a new tracking entity. In one example, the trigger condition is input from the user. In this example, system 100 may determine that the parameters of the object do not match any existing tracking entity, and therefore, a new tracking entity should be created. However, instead of automatically creating a new tracking entity, system 100 waits for user input before creating a new tracking entity for the object. Alternatively, the trigger condition could be placing the object at a specific location relative to locator 110. In this example, system 100 will not attempt to create a new tracking entity unless the object is at said location. More specifically, this location could be a table within the field of view of locator 110. Here, system 100 can detect objects and their parameters at 412 and 416, but will only create a new tracking entity when the object is perceived by locator 110 as being on the table.

[0084] Following 424A or 424B, method 400 continues by calling tracking manager 320 at 428, followed by navigation manager 324 at 432. Navigation system 100 utilizes tracking manager 320 to manage active and inactive tracking entities 300. Navigation system 100 utilizes navigation manager 324 to manage how the locator tracks surgical objects.

[0085] refer to Figure 5 The diagram illustrates an example flow executed by the trace manager 320 during method 400. In this example flow, the trace manager 320 is invoked at 428 during method 400 and begins by outputting trace entity 300 at 428A. The output trace entity 300 is either the trace entity recovered at 424A or the trace entity 300 created at 424B. In either case, the trace entity is output along with a unique identifier, allowing the user (and system 100) to distinguish each trace entity 300 from other trace entities 300 based on the unique identifier assigned to each entity 300. The flow then proceeds to 428B.

[0086] At 428B, the tracking manager 320 determines whether the object still exists in the image data. If not, the process proceeds to 428C, at which point the tracking entity 300 associated with the object is deactivated and the process invoked at 428 ends. If the object still exists in the image data, the process continues to 428D.

[0087] At 428D, the tracking manager 320 determines whether any parameters of the object have changed over time. For example, the tracking manager 320 may determine that the color of at least a portion of the object has changed since the creation of the tracking entity 300 associated with the object. If not, the process ends. If the parameters have changed, the process moves to 428E. At 428E, the temporary descriptor associated with the tracking entity 300 is updated based on the parameter changes determined at 428D. This allows the navigation system 100 to continue tracking the object even if it changes in the operating room. For example, if the object is a surgical instrument 250, the instrument 250 may be partially covered in blood during surgery. When the instrument 250 changes color due to blood or other contaminants, the system 100 may need to update the color parameters of the tracking entity 300 associated with the instrument 250 in order to be able to identify the instrument. After updating the tracking entity 300, the process ends.

[0088] Method 400 is depicted in a linear fashion, where each step / process occurs one after another, and is typically described as if only one object is detected and tracked at a time. However, method 400 does not need to be executed as illustrated. For example, system 100 may determine at 412 that multiple objects are present in the image data. Therefore, system 100 may determine parameters for each of the objects, and then attempt to match each of the objects with the existing tracked entity 300, before proceeding to 428. In another example, the tracking manager continues to run along with the rest of method 400. In other words, method 400 may proceed to 428 as illustrated, and then continue to loop through the tracking manager process, while also looping through the steps shown at 404 to 424B. This allows system 100 to control the tracked entity 300 using the tracking manager 320, while simultaneously accepting the characterization and tracking of new objects that emerge in the image data.

[0089] refer to Figure 6 and Figure 7An example of a tracking entity 300 created by navigation system 100 is shown. In the illustrated implementation, the object is a surgical instrument 250 held by the user. The surgical instrument 250 also includes one of the trackers 150 coupled thereto. As described above, the locator 110 includes an optical camera unit 112 and an IR sensor unit 114, such that the locator 110 can detect the object in an optical coordinate system and detect the tracker 150 in an IR coordinate system (and / or the optical coordinate system). The navigation controller 104 has access to data relating to these coordinate systems, such that the controller 104 can calibrate the optical coordinate system to the IR coordinate system. The navigation controller 104 can also calibrate both coordinate systems to a global coordinate system. According to method 400 described above, the locator 110 detects the orientation of the tracker 150 in the IR coordinate system (using any suitable method) and detects the parameters of the instrument 250 in the optical coordinate system. In the illustrated implementation, this parameter is simply the shape of the instrument 250, and Figure 7 This includes various tracking entities created based on the shapes of the corresponding tracker 150 and device 250.

[0090] Once the system 100 determines the orientation of the tracker 150 and the parameters of the instrument 250, it associates the tracker 150 with the instrument 250 and those parameters. This association is stored as a tracking entity 300 in the tracking database 318. After creating the tracking entity 300 for the surgical instrument 250, the orientation of the instrument 250 is tracked based on the orientation of the tracking entity 300, and the orientation of the tracking entity 300 is tracked based on the orientation of the tracker 150.

[0091] Figure 6 and Figure 7 The tracking entity 300 shown can be used by a user to track the posture of the instrument 250 relative to the patient and / or other objects present in the operating room. For example, an X-ray image of the patient can be displayed on one of the displays 106, 107, and the tracking entity 300 can be overlaid as a computer graphic on top of the X-ray image. The computer graphic can be based on parameters determined by system 100, and the graphic may include a unique identifier assigned to the tracking entity 300. In this example, the position of the tracking entity 300 on displays 106, 107 is based on the position of the surgical instrument 250 relative to the patient, as captured by locator 110.

[0092] refer to Figure 8This illustrates an example flow executed by navigation manager 324 during method 400. In this example flow, navigation manager 324 is invoked at 432 during method 400 and begins receiving tracking entity 300 (or multiple tracking entities) at 432A. Navigation manager 324 is typically configured to control system 100 to track surgical objects using the visible light sensor 118 and IR sensor 119 of locator 110. More specifically, since locator 110 is capable of tracking the orientation of tracker 150 in an IR coordinate system using IR sensor 119 and tracking parameters of the surgical object in an optical coordinate system using visible light sensor 118, navigation manager 324 controls locator 110 and / or navigation controller 104 to track the object using combinations of different tracking modes (e.g., optical and infrared). Figure 8 The illustrated flow is one that the navigation manager 324 can utilize to implement this multimodal tracking.

[0093] After receiving the tracked entity 300 at 432A, the process continues to 432B. At 432B, the navigation manager 324 instructs the navigation controller 104 to track the surgical object based on the pose of the tracked entity 300 output by the tracking manager 320. For example, the navigation controller 104 may track the object based on the pose of the tracker 150. Alternatively, the controller 104 may track the object based on the pose of a temporary descriptor (e.g., the pose of shape parameters included in the temporary descriptor). Furthermore, the controller 104 may track the object based on a combination of the pose of the tracker 150 and the pose of the temporary descriptor. Further still, the controller 104 may track the object based on a combination of the pose of the tracker 150 and a known relationship between the tracker 150 and the temporary descriptor. The process then proceeds to 432C.

[0094] At 432C, navigation manager 324 determines whether tracker 150 associated with tracked entity 300 has been occluded and out of the field of view of locator 110's IR sensor 119. If not, the process returns to 432B and system 100 continues tracking the object as usual. However, if navigation manager 324 determines that tracker 150 of tracked entity 300 has been occluded and out of the field of view, the process moves to 432D. At 432D, navigation manager 324 begins to track the object solely based on a temporary descriptor associated with tracked entity 300, as detected by locator 110's visible light sensor 118. In other words, system 100 switches from tracking the object using data from both the optical and infrared coordinate systems to tracking the object solely based on data from the optical coordinate system.

[0095] After system 100 begins tracking the object based on a temporary descriptor (i.e., based on parameters detected by visible light sensor 118), the process moves to 432E. At 432E, navigation manager 324 determines whether tracker 150 can be detected again by IR sensor 119. If not, the process moves back to 432D and system 100 continues to track the object via parameters detected by visible light sensor 118, but not via the pose of tracker 150, as tracker 150 is occluded and out of the field of view of IR sensor 119. However, if tracker 150 is detected at 432E, the process returns to 432B, and system 100 switches from tracking the object using only data from the optical coordinate system to tracking the object using data from both the optical and infrared coordinate systems. This may include, for example, using recovery module 322 to recover the tracked entity.

[0096] Therefore, even if tracker 150 is occluded and leaves the field of view of locator 110, navigation manager 324 allows navigation system 100 to continue tracking objects in the operating room. In some implementations, after tracker 150 is occluded, navigation controller 104 can confirm that the surgical object corresponds to one of the active tracking entities 300, which is no longer associated with tracker 150 due to being occluded and leaving the field of view. For example, navigation controller 104 can determine new parameters of the surgical object between 432C and 432D and compare the new parameters with a temporary descriptor for each active tracking entity 300. If the new parameters match at least one of the parameters included in the temporary descriptor of one of the active tracking entities 300, navigation controller 104 can determine that the object corresponds to that tracking entity 300.

[0097] refer to Figure 9 This describes an example method 500 for tracking objects in an operating room. Method 500 is... Figure 4 The simplified version of method 400 shown, and Figure 9 Provided for describing such Figure 4 , Figure 5 and Figure 8 The following is a simplified illustration of method 400. In this simplified version, it is assumed that system 100 has determined that database 318 does not contain any tracking entity 300 (e.g., at system 100 startup), or that the parameters of the object do not match any tracking entity 300 stored in database 318. Therefore, referring back... Figure 4 In the method 400 shown, system 100 has performed at least some of steps 404 to 420 and determined that the object is unidentified and / or untracked. Although the order of some steps is... Figure 9 Method 500 looks similar to Figure 4 , Figure 5 and Figure 8 Method 400 is different, but the steps of methods 400 and 500 can be performed in any suitable order.

[0098] At point 504, navigation controller 104 determines that locator 110 has detected an unidentified / untracked object. This is actually when controller 104 determines that the object does not match any existing tracked entity 300. Figure 4 The operation occurs at 420. Then, method 500 continues to 508. At 508, the controller detects the pose of the tracker 150 coupled to the object, and method 500 continues to 512.

[0099] At 512, navigation controller 104 determines a first parameter of the object, such as by receiving the first parameter from locator 110. After determining the first parameter, method 500 moves to 516, where controller 104 creates a temporary descriptor based on the first parameter. At this point, controller 104 may instruct system 100 to track the object based on the temporary descriptor and / or the pose of tracker 150. Otherwise, method 500 proceeds to 520 and determines a second parameter of the object. After determining the second parameter, method 500 moves to 524.

[0100] At 524, controller 104 updates temporary descriptor 300 based on the second parameter determined at 520 and creates an updated temporary descriptor. Subsequently, at 526, controller 104 instructs the system to track the object based on the updated temporary descriptor and the pose of tracker 150. If the object does not have a tracker 150 coupled to it, any use of the pose of tracker 150 in method 500 can be ignored / skipped. In either case, the step at 528 is substantially similar to the step at 432B of method 400.

[0101] The systems and methods described herein can be implemented, in part or in whole, as instructions stored on a non-transitory computer-readable medium. As used herein, the term computer-readable medium does not encompass transient electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); therefore, the term computer-readable medium can be considered tangible and non-transitory. Non-limiting examples of non-transitory computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or masked read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0102] The systems and methods described herein can be implemented, in whole or in part, as computer program products. Computer program products may include processor-executable instructions stored on at least one non-transitory computer-readable medium. Computer program products may also include or depend on stored data. Computer program products may encompass a basic input / output system (BIOS) for interacting with the hardware of a dedicated computer, device drivers for interacting with specific devices of a dedicated computer, one or more operating systems, user applications, background services, background applications, etc. Computer program products may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated from source code by a compiler; (iv) source code for execution by an interpreter; (v) source code for compilation and execution by a just-in-time (JIT) compiler, etc. As an example only, the source code can be written using the syntax of languages ​​including: C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, JavaScript®, HTML5 (Hypertext Markup Language 5th Edition), Ada, ASP (Dynamic Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SIMULINK, and Python®.

[0103] Several implementations have been discussed in the foregoing description. However, the implementations discussed herein are not intended to be exhaustive or to limit the invention to any particular form. The terminology used is intended to be descriptive rather than restrictive in nature. In view of the above teachings, many modifications and variations are possible, and the invention may be practiced in ways other than those specifically described.

[0104] Many features and advantages of the invention are apparent from the detailed description, and therefore the appended claims are intended to cover all such features and advantages of the invention that fall within the true spirit and scope of the invention. Furthermore, since numerous modifications and variations will readily occur to those skilled in the art, it is not intended to limit the invention to the exact constructions and operations illustrated and described, and therefore all suitable modifications and equivalents falling within the scope of the invention may be employed.

Claims

1. A surgical navigation system, the surgical navigation system comprising: A tracker coupled to a surgical object and including tracking elements arranged in tracker geometry; A locator configured to track the tracker and detect parameters of the surgical subject; as well as The controller communicates with the locator and is configured to: The locator is used to detect the orientation of the tracker's geometry. Receive a first parameter of the surgical object detected by the locator. A temporary descriptor describing the appearance of the surgical object is created based on the first parameter. Receive a second parameter of the surgical object detected by the locator. The temporary descriptor is updated based on the second parameter to create an updated temporary descriptor describing the appearance of the surgical object. The surgical object is tracked based on a combination of the updated temporary descriptor and the pose of the tracker geometry.

2. The surgical navigation system as described in claim 1, wherein: The first parameter is detected by the locator at a first time, and the second parameter is detected by the locator at a second time.

3. The surgical navigation system of claim 2, wherein the first time is prior to the second time.

4. The surgical navigation system of claim 1, wherein at least one of the first parameter and the second parameter is a physical parameter of the surgical object.

5. The surgical navigation system of claim 4, wherein the physical parameter is one of the following: the geometry or shape, contour, color, envelope, surface roughness, surface markings, or color or brightness of the object.

6. The surgical navigation system of claim 1, wherein at least one of the first parameter and the second parameter is a motion parameter of the surgical object.

7. The surgical navigation system of claim 6, wherein the motion parameters include at least one of the following: the velocity or speed of the surgical object, the acceleration of the surgical object, the rotation of the surgical object, and the displacement of the surgical object.

8. The surgical navigation system of claim 1, wherein the first parameter is a first color of the surgical object, and the second parameter is a second color of the surgical object.

9. The surgical navigation system of claim 8, wherein the controller is configured to detect environmental conditions and normalize the detected parameters according to the environmental conditions.

10. The surgical navigation system of claim 9, wherein the environmental conditions are the illumination of the surgical object, the illumination of the tracking geometry, and / or the total illumination of the space within the line of sight of the locator.

11. The surgical navigation system of claim 1, wherein the second parameter is the color change of the surgical object.

12. The surgical navigation system of claim 1, wherein the controller is configured to: operate in a first state, in which the controller associates the tracker geometry with the surgical object after a triggering event; and operate in a second state, in which the controller automatically associates the tracker geometry with the surgical object.

13. The surgical navigation system of claim 12, wherein the triggering event is input from a user.

14. The surgical navigation system of claim 12, wherein the triggering event occurs when the surgical object is in a predefined posture.

15. The surgical navigation system of claim 1, wherein the locator includes a first sensor configured to detect visible light and a second sensor configured to detect infrared or near-infrared light.

16. The surgical navigation system of claim 15, wherein the attitude of the tracker geometry is detected by the second sensor.

17. The surgical navigation system of claim 16, wherein the first parameter is detected by the first sensor.

18. The surgical navigation system of claim 15, wherein the controller detects the attitude of the tracker geometry in a first coordinate system and detects the first parameter in a second coordinate system.

19. The surgical navigation system of claim 18, wherein the controller is configured to register at least one of the first coordinate system and the second coordinate system to the third coordinate system.

20. The surgical navigation system of claim 18, wherein the controller is configured to register one of the first coordinate system and the second coordinate system to the other of the first coordinate system and the second coordinate system.

21. The surgical navigation system of claim 1, wherein the updated temporary descriptor includes the shape of the surgical object and the color of the surgical object.

22. The surgical navigation system of claim 1, wherein the controller is configured to: Detecting a third parameter of the surgical subject, The updated temporary descriptor is updated based on the third parameter, and The pose of the surgical object is tracked based on a combination of the updated temporary descriptor and the pose of the tracker geometry.

23. The surgical navigation system of claim 1, wherein the controller is configured to determine that the orientation of the tracker geometry cannot be detected by the locator.

24. The surgical navigation system of claim 23, wherein the controller is configured to track the posture of the surgical object solely based on the updated temporary descriptor in response to determining that the posture of the tracker geometry cannot be detected by the locator.

25. The surgical navigation system of claim 24, wherein the controller is configured to determine, after determining that the pose of the tracker geometry can be detected by the locator, that the pose of the tracker geometry cannot be detected by the locator.

26. The surgical navigation system of claim 25, wherein the controller is configured to track the posture of the surgical object based on at least one of the updated temporary descriptor and the posture of the tracker geometry in response to determining that the posture of the tracker geometry can be detected by the locator.

27. The surgical navigation system of claim 25, wherein the controller is configured to register or re-register the tracker to the surgical object by comparing new parameters of the surgical object with the updated temporary descriptor.

28. The surgical navigation system of claim 1, wherein the controller provides the first parameter and the second parameter to the machine learning module, and the machine learning module outputs calculated parameters of the surgical object based on the first parameter and the second parameter.

29. The surgical navigation system of claim 28, wherein the controller is configured to: The updated temporary descriptor is updated based on the calculated parameters to create a learned temporary descriptor describing the association between the surgical object and the detected tracker geometry. The pose of the surgical object is tracked based on a combination of the learned temporary descriptor and the pose of the tracker geometry.

30. The surgical navigation system of claim 29, wherein the controller is configured to: Receive the third parameter of the surgical subject. The calculated parameters are compared with the third parameter. The learned temporary descriptor is updated based on comparisons to create an updated learned temporary descriptor describing the association between the surgical object and the detected tracker geometry. The pose of the surgical object is tracked based on a combination of the updated learned temporary descriptor and the pose of the tracker geometry.

31. The surgical navigation system of claim 1, wherein the second parameter is implemented as a motion parameter, and the controller is configured to: Detect the attitude changes of the tracker's geometry. The updated temporary descriptor is created based on the determination that the motion parameters are sufficiently consistent with the attitude changes of the tracker geometry.

32. The surgical navigation system of claim 1, wherein the controller is configured to replace the first parameter with the second parameter in response to determining that the second parameter is inconsistent with the first parameter.

33. The surgical navigation system of claim 1, wherein the tracker is implemented as a first tracker, and the surgical object is implemented as a first surgical object, and the surgical navigation system further comprises: A second tracker is coupled to a second surgical object and includes a tracking element arranged with the same tracker geometry as the first tracker; and The locator is configured to track the first tracker and the second tracker, and to detect parameters associated with the first surgical object and the second surgical object; and The controller is configured as follows: The locator is used to detect the pose of the first tracker and a first parameter associated with the first surgical object. A first tracking entity is created based on the pose of the first tracker and the first parameters. The locator is used to detect the pose of the second tracker and a second parameter associated with the second surgical object. A second tracking entity is created based on the pose of the second tracker and the second parameters, and The poses of the first surgical object and the second surgical object are tracked based on the movements of the first tracking entity and the second tracking entity, respectively.

34. The surgical navigation system of claim 1, further comprising: A database containing descriptors of surgical objects; and The controller is configured as follows: The first parameter of the surgical object is compared with the descriptor stored in the database. Based on comparison, it is determined that no descriptor describing the appearance of the surgical object exists in the database, and in response, a temporary descriptor describing the surgical object is created based on the detected parameters of the surgical object.

35. The surgical navigation system of claim 1, further comprising a display that communicates with the controller and is configured to depict the surgical object relative to a surgical target.

36. The surgical navigation system of claim 35, wherein the surgical object is represented by computer graphics, and the computer graphics are depicted relative to the surgical target.

37. The surgical navigation system of claim 36, wherein the computer graphics are based on one of the temporary descriptor and the parameters.

38. The surgical navigation system of claim 36, wherein the computer graphics include a unique identifier that identifies the surgical object.

39. The surgical navigation system of claim 1, wherein the controller is configured to create a tracking entity based on the updated temporary descriptor and the tracker geometry.

40. The surgical navigation system of claim 39, wherein the controller is configured to track the surgical object by tracking the tracking entity based on a combination of the pose of the updated temporary descriptor and the tracker geometry.

41. The surgical navigation system of claim 1, wherein the updated temporary descriptor is implemented as a database entry containing at least the first parameter and the second parameter.

42. A surgical navigation system for tracking a surgical object during a surgical procedure, the system comprising: A tracker coupled to the surgical object and including tracker geometry; A locator configured to detect the tracker and detect the presence of the surgical object and parameters of the surgical object; as well as The controller communicates with the locator and is configured to: Receive the parameters of the surgical subject. A temporary descriptor describing the appearance of the surgical object is created based on the parameters of the surgical object. The locator is used to detect the movement of the tracker and the movement of the temporary descriptor. Determine whether the movement of the tracker is sufficiently correlated with the movement of the temporary descriptor, and if so, create a tracking entity including the tracker geometry and the temporary descriptor. The movement of the surgical object is tracked based on the movement of the tracking entity.

43. A surgical navigation system for tracking a surgical subject during a surgical procedure, the system comprising: A locator configured to detect the presence of the surgical object and to detect parameters of the surgical object; as well as The controller communicates with the locator and is configured to: The parameters of the surgical subject are received from the locator at the first moment. A temporary descriptor describing the appearance of the surgical object is created based on the parameters of the surgical object received from the locator at the first time. The parameters of the surgical subject are received from the locator at a second time. Determine whether the parameters received at the first time point are different from those received at the second time point, and if they are different, create an updated temporary descriptor describing the appearance of the surgical object based on the parameters received at both the first and second times points. The pose of the surgical object is tracked based on the updated temporary descriptor.

44. A surgical navigation system, the surgical navigation system comprising: A first tracker, the first tracker being coupled to a first surgical object and including a tracking element arranged in the geometry of the first tracker; A second tracker, the second tracker being coupled to a second surgical object and including a tracking element arranged in the geometry of the first tracker; A locator configured to track the first tracker and the second tracker, and to detect parameters associated with the first surgical object and the second surgical object; as well as The controller communicates with the locator and is configured to: The locator is used to detect the pose of the first tracker and a first parameter associated with the first surgical object. A first tracking entity is created based on the pose of the first tracker and the first parameters. The locator is used to detect the pose of the second tracker and a second parameter associated with the second surgical object. A second tracking entity is created based on the pose of the second tracker and the second parameters, and The poses of the first surgical object and the second surgical object are tracked based on the movements of the first tracking entity and the second tracking entity, respectively.

45. A surgical navigation system for tracking a surgical subject during a surgical procedure, the system comprising: A locator configured to detect the presence of the surgical object and to detect parameters of the surgical object; A database containing descriptors of surgical objects; as well as The controller communicates with the locator and is configured to: Receive the parameters of the surgical subject from the locator. The parameters of the surgical object are compared with the descriptors stored in the database. Based on comparison, it is determined that no descriptor describing the appearance of the surgical object exists in the database, and in response, a temporary descriptor describing the surgical object is created based on the detected parameters of the surgical object. The posture of the surgical object is tracked based on the temporary descriptor.

46. ​​A surgical navigation system for tracking a surgical object during a surgical procedure, the system comprising: A tracker coupled to the surgical object; The locator includes a first sensor and a second sensor and is configured to: The first sensor is used to detect the attitude of the tracker in the first tracking mode, and The second sensor is used to detect parameters of the surgical object in the second tracking mode; and The controller communicates with the locator and is configured to: Receive the attitude of the tracker in the first imaging mode. Receive the parameters of the surgical object in the second imaging mode. A temporary descriptor describing the appearance of the surgical object is created based on the detected parameters of the surgical object, and The surgical object is tracked based on the pose of the tracker and the temporary descriptor.

47. A surgical navigation system for tracking a surgical object during a surgical procedure, the system comprising: A tracker coupled to the surgical object; The locator includes an NIR sensor and a visible light sensor and is configured to: The NIR sensor is used to detect the attitude of the tracker in NIR space, and The parameters of the surgical object are detected using the visible light sensor in the visible light space; and The controller communicates with the locator and is configured to: Receive the attitude of the tracker in the NIR space. Receive the parameters of the surgical object in the visible light space. A temporary descriptor describing the appearance of the surgical object is created based on the detected parameters of the surgical object. Determine that the tracker's pose is no longer known in the NIR space; In response, the surgical object is tracked based on the parameters of the surgical object in the visible light space.

48. A surgical navigation system for tracking a surgical object during a surgical procedure, the system comprising: A tracker coupled to the surgical object; The locator includes a first sensor and a second sensor and is configured to: The first sensor is used to detect the attitude of the tracker in the first tracking mode, and The second sensor is used to detect parameters of the surgical object in the second tracking mode; and The controller communicates with the locator and is configured to: Receive the attitude of the tracker detected in the first imaging mode. Receive the parameters of the surgical object detected in the second imaging modality. A temporary descriptor describing the appearance of the surgical object is created based on the detected parameters of the surgical object. A tracking entity describing the association between the tracker and the surgical object is created based on the tracker's pose and the temporary descriptor. The surgical object is tracked based on the pose of the tracked entity. In response to determining that the tracker is occluded from the perspective of the locator, the tracking entity is disabled. In response to determining that, from the locator's perspective, the tracker is no longer occluded and the tracker remains attached to the surgical object, the tracking entity is reactivated, and The surgical object is recovered based on the pose of the tracked entity.

49. The surgical navigation system of claim 48, wherein the controller is configured to determine that the tracker is still associated with the surgical object by comparing new parameters of the surgical object with the temporary descriptor.

50. The surgical navigation system of claim 49, wherein the parameters of the surgical object are implemented as initial parameters, and the controller is configured to compare the new parameters with the temporary descriptor by comparing the new parameters with the initial parameters.

51. A method for tracking a surgical object using a locator, the surgical object being coupled to the tracker, the tracker including tracking elements arranged in tracker geometry, the method comprising: The locator is used to detect the orientation of the tracker's geometry. Receive a first parameter of the surgical object detected by the locator. A temporary descriptor describing the appearance of the surgical object is created based on the first parameter. Receive a second parameter of the surgical object detected by the locator. The temporary descriptor is updated based on the second parameter to create an updated temporary descriptor describing the appearance of the surgical object. The surgical object is tracked based on a combination of the updated temporary descriptor and the pose of the tracker geometry.

52. A non-transitory computer-readable medium storing instructions configured to be executed by a controller connected to a locator to cause the controller to perform operations including: The locator is used to detect the attitude of the tracker's geometry. Receive a first parameter of the surgical object detected by the locator. A temporary descriptor describing the appearance of the surgical object is created based on the first parameter. Receive a second parameter of the surgical object detected by the locator. The temporary descriptor is updated based on the second parameter to create an updated temporary descriptor describing the appearance of the surgical object. The surgical object is tracked based on a combination of the updated temporary descriptor and the pose of the tracker geometry.

Images