Obstacle avoidance technology for surgical navigation

Abstract

This paper describes a system and method in which a locator is configured to detect the position of a first object and a vision device is configured to generate a depth map of a surface near the first object. A virtual model corresponding to the first object is accessed, and the positional relationship between the locator and the vision device in a common coordinate system is identified. A desired depth map of the vision device is then generated based on the detected position of the first object, the virtual model, and the positional relationship. Portions of the actual depth map that fail to match the desired depth map are identified, and a second object is identified based on the identified portions.

Description

[0001] Cross-reference of related applications

[0002] This application claims priority and all benefits to U.S. Provisional Patent Application No. 62 / 870,284, filed July 3, 2019, the contents of which are hereby incorporated by reference in their entirety. Technical Field

[0003] This disclosure generally relates to surgical navigation systems. Background Technology

[0004] Surgical navigation systems help locate surgical instruments relative to the target volume of patient tissue for treatment. During surgical procedures, the target volume to be treated is often located near sensitive anatomical structures and surgical instruments that should be avoided. Due to the flexible nature of these structures, tracking these adjacent anatomical structures using attached trackers is often difficult. Furthermore, attaching trackers to each object adjacent to the target volume makes the surgical workspace crowded and increases the cost and complexity of the surgical navigation system. Summary of the Invention

[0005] In a first aspect, a navigation system is provided, the navigation system comprising: a locator configured to detect a first object; a vision device configured to generate an actual depth map of a surface near the first object; and a controller coupled to the locator and the vision device, the controller being configured to: access a virtual model corresponding to the first object; identify a positional relationship between the locator and the vision device in a common coordinate system; generate a desired depth map of the vision device based on the detected position of the first object, the virtual model, and the positional relationship; identify portions of the actual depth map that fail to match the desired depth map; and identify a second object based on the identified portions.

[0006] In the second aspect, the robotic manipulator is used in conjunction with the navigation system of the first aspect, wherein the robotic manipulator supports a surgical tool and includes a plurality of links and a plurality of actuators configured to move the links to move the surgical tool, and wherein the robotic manipulator is controlled to avoid the second object.

[0007] In a third aspect, a method for operating a navigation system is provided, the navigation system comprising: a locator configured to detect the position of a first object; a vision device configured to generate an actual depth map of a surface near the first object; and a controller coupled to the locator and the vision device, the method comprising: accessing a virtual model corresponding to the first object; identifying a positional relationship between the locator and the vision device in a common coordinate system; generating a desired depth map of the vision device based on the detected position of the first object, the virtual model, and the positional relationship; identifying portions of the actual depth map that fail to match the desired depth map; and identifying a second object based on the identified portions.

[0008] In a fourth aspect, a computer program product is provided, comprising a non-transitory computer-readable medium having instructions stored thereon, the instructions being configured to implement the method of the third aspect when executed by one or more processors.

[0009] According to any of the above aspects, the locator is configured to be: an optical locator configured to detect optical features associated with the first object; an electromagnetic locator configured to detect electromagnetic features associated with the first object; an ultrasonic locator configured to detect the first object with or without a tracker; an inertial locator configured to detect inertial features associated with the first object; or any combination thereof.

[0010] According to any of the above aspects, the first object may be any of the following: the patient's anatomical structure or skeleton; equipment in the operating room, such as, but not limited to: a robotic manipulator, a handheld instrument, an end effector or tool attached to the robotic manipulator, an operating table, a mobile trolley, an operating table on which a patient may be placed, an imaging system, a retractor, or any combination thereof.

[0011] According to one implementation of any of the above aspects: the vision device is coupled to any of the following: the locator; a unit separate from the locator; the camera unit of the navigation system; an adjustable arm; the robot manipulator; an end effector; a handheld tool; a surgical boom system, such as a ceiling-mounted boom, a limb fixation device, or any combination of the foregoing.

[0012] According to any of the above aspects, the surface near the first object can be: a surface adjacent to the first object; a surface spaced apart from the first object by a certain distance; a surface in contact with the first object; a surface directly on top of the first object; a surface in the environment near the first object; a surface in the environment behind or around the first object; a surface within a threshold distance of the first object; a surface within the field of view of the locator; or any combination of the foregoing.

[0013] According to one implementation of any of the foregoing aspects: the second object may be an object capable of forming an obstacle, comprising any of the following: a second part of the patient's anatomical structure, such as surrounding soft tissue; equipment in the operating room, such as, but not limited to: a robotic manipulator, one or more arms of the robotic manipulator, a second robotic manipulator, a handheld instrument, an end effector or tool attached to the robotic manipulator or handheld instrument, an operating table, a mobile trolley, an operating table on which the patient may be placed, an imaging system, a retractor, the body of a tracking device; a body part of a person in the operating room, or any combination thereof.

[0014] According to one implementation of any of the foregoing aspects: the controller may be one or more controllers or control systems. According to one implementation, the controller is configured to identify the position of the second object relative to the first object in the common coordinate system. According to one implementation, the controller identifies the position based on the detected position of the first object, the location of the second object in the actual depth map, and the positional relationship.

[0015] According to one implementation, the first object defines a target volume of patient tissue to be treated according to the surgical plan. According to one implementation, the controller is configured to: determine whether the second object is an obstacle to treating the target volume according to the surgical plan, based on the position of the second object relative to the target volume in the common coordinate system and the surgical plan. According to one implementation, in response to determining that the second object is an obstacle to the surgical plan, the controller is configured to: modify the surgical plan and / or trigger notifications and / or stop surgical navigation.

[0016] According to one implementation, the tracker is coupled to the first object. According to one implementation, the controller is configured to detect the position of the tracker in a first coordinate system specific to the locator via the locator. According to one implementation, the controller can identify the position of the virtual model in the first coordinate system based on the detected position of the tracker in the first coordinate system and the positional relationship between the tracker and the first object in the first coordinate system. According to one implementation, the controller converts the position of the virtual model in the first coordinate system to a position of the virtual model in a second coordinate system specific to the vision device based on the position of the virtual model in the first coordinate system and the positional relationship between the locator and the vision device in a second coordinate system. According to one implementation, the controller can generate the expected depth map based on the position of the virtual model in the second coordinate system.

[0017] According to one implementation, the controller is configured to identify portions of the actual depth map that fail to match the expected depth map by comparing the actual depth map and the expected depth map. In some implementations, the controller calculates the difference between the actual depth map and the expected depth map. According to one implementation, the controller determines whether a first segment determining the difference indicates an absolute depth greater than a threshold depth. According to one implementation, in response to determining that the first segment of the difference indicates an absolute depth greater than the threshold depth, the controller identifies a second segment of the actual depth map corresponding to the first segment of the difference as the portion. According to one implementation, the threshold depth is non-zero.

[0018] In one implementation, the controller is configured to identify portions of the actual depth map that fail to match the expected depth map. In some implementations, the controller does this by being configured to determine whether the size of the first segment is greater than a minimum size threshold. In some implementations, the controller identifies the second segment as the portion in response to determining that the size of the first segment is greater than the minimum size threshold.

[0019] According to one implementation, the controller is configured to identify a second object based on the identified portion by being configured to perform the following: matching the identified portion with a predetermined contour corresponding to the second object.

[0020] According to one implementation, the portion of the actual depth map includes an arrangement of features corresponding to the second object and located at a first position in the actual depth map. According to one implementation, the controller is configured to track the movement of the second object by monitoring whether the arrangement of the features has moved to a second position, different from the first position. According to one implementation, the controller monitors such content in additional actual depth maps subsequently generated by the vision device.

[0021] In one implementation, the controller is configured to generate a virtual boundary corresponding to the second object in the common coordinate system. In another implementation, the virtual boundary provides constraints. In some instances, the constraints are for the movement of objects such as surgical instruments, robotic manipulators, the working end of a robotic handheld surgical device, imaging equipment, or any other movable equipment in the operating room. In some instances, the constraints are no-entry or no-exit boundaries.

[0022] In one implementation, the controller is configured to crop the actual depth map to a region of interest based on the virtual model, the detected position of the first object, and the positional relationship between the locator and the vision device in a common coordinate system. In some implementations, the controller is configured to compare the actual depth map by being configured to: compare the cropped actual depth map.

[0023] According to one implementation, the controller is configured to identify the positional relationship between the locator and the vision device in the common coordinate system by being configured to project a pattern onto a surface within the field of view of the vision device (and optionally also within the field of view of the locator). In some implementations, the controller uses the locator to generate positioning data indicating the position of the pattern in a first coordinate system specific to the locator. In some implementations, the controller receives a calibration depth map generated by the vision device showing the projected pattern. In some implementations, the controller identifies the position of the projected pattern in a second coordinate system specific to the vision device based on the calibration depth map. In some implementations, the controller identifies the positional relationship between the locator and the vision device in the common coordinate system based on the position of the pattern in the first coordinate system and the position of the pattern in the second coordinate system. In some implementations, the controller is configured to operate in a first spectral band to detect the position of the first object, the vision device is configured to operate in a second spectral band to generate the actual depth map of the surface near the first object, and the first spectral band is different from the second spectral band.

[0024] Any of the above aspects may be combined in whole or in part.

[0025] The above summary provides a simplified overview of some aspects of the invention to offer a basic understanding of certain aspects of the invention discussed herein. The summary is not intended to provide a broad overview of the invention, nor is it intended to identify any significant or critical elements of the invention or to define its scope. The sole purpose of the summary is merely to present some concepts in a simplified form as an introduction to the specific embodiments presented below. Attached Figure Description

[0026] Figure 1 It is a perspective view of a surgical navigation system that includes a locator and a visual device.

[0027] Figure 2 It is used for control Figure 1 A schematic diagram of the control system of the surgical navigation system.

[0028] Figure 3 Is Figure 1 A perspective view of the coordinate system used in the surgical navigation system.

[0029] Figure 4 This is a flowchart of an example of a method for using tracker-based localization and machine vision navigation of a target part.

[0030] Figure 5 This is an illustration of an example of a target site, such as an anatomical structure being treated during a surgical procedure.

[0031] Figure 6 Is with Figure 5 A diagram showing the position of the virtual model corresponding to the object in the target area.

[0032] Figure 7 Based on Figure 6 An illustration of the expected depth map of the virtual model.

[0033] Figure 8 It is by Figure 1 An illustration of the actual depth map captured by the vision device.

[0034] Figure 9 yes Figure 8 The actual depth map is cropped into an illustration of the region of interest.

[0035] Figure 10 yes Figure 7 Expected depth map and Figure 9 A diagram illustrating the difference between the actual depth maps and the actual depth maps.

[0036] Figure 11 Is with Figure 9 The illustration shows the virtual model corresponding to the surgical retractor identified in the actual depth map.

[0037] Figure 12 Is with Figure 9 The illustration shows the virtual model corresponding to the ligaments marked in the actual depth map.

[0038] Figure 13 Is with Figure 9 An illustration of the virtual model corresponding to the epidermal tissue identified in the actual depth map.

[0039] Figure 14 yes Figure 6 and Figures 11 to 13 A diagram of the virtual model in a common coordinate system.

[0040] Figure 15 It was subsequently by Figure 1 An illustration of the actual depth map captured by the vision device.

[0041] Figure 16 yes Figure 14 Based on Figure 15 The actual depth map is illustrated with an updated virtual model of the localization. Detailed Implementation

[0042] Figure 1 A surgical system 10 for treating a patient is illustrated. The surgical system 10 may be located in a surgical setting such as an operating room in a medical facility. The surgical system 10 may include a surgical navigation system 12 and a robotic manipulator 14. The robotic manipulator 14 may be coupled to a surgical instrument 16 and may be configured to manipulate the surgical instrument 16, for example, under the guidance of a surgeon and / or the surgical navigation system 12, to treat a target volume of patient tissue. For example, the surgical navigation system 12 may cause the robotic manipulator 14 to manipulate the surgical instrument 16 to remove a target volume of patient tissue while avoiding other objects adjacent to the target volume, such as other medical instruments and adjacent anatomical structures. Alternatively, the surgeon may manually hold and manipulate the surgical instrument 16 while receiving guidance from the surgical navigation system 12. As some non-limiting examples, the surgical instrument 16 may be a deburring instrument, an electrosurgical instrument, an ultrasonic instrument, a reamer, an impactor, or a sagittal saw.

[0043] During the surgical procedure, the surgical navigation system 12 can use a combination of tracker-based localization and machine vision to track the position (localization and orientation) of the object of interest within the surgical workspace. The surgical workspace for the surgical procedure can be considered to include the target volume of patient tissue to be treated and the area immediately surrounding the target volume where treatment obstacles may exist. The tracked object may include, but is not limited to, the patient's anatomy, the target volume of the anatomy to be treated, surgical instruments such as surgical instruments 16, and the surgeon's anatomy such as the surgeon's hand or fingers. The tracked patient's anatomy and target volume may include soft tissues such as ligaments, muscles, and skin, and may include hard tissues such as bones. The tracked surgical instruments may include retractors, cutting tools, and waste management devices used during the surgical procedure.

[0044] Attaching a tracker to an object of interest in the surgical workspace provides the surgical navigation system 12 with an accurate and efficient mechanism for determining the position of such an object within the workspace. During the procedure, the tracker may generate a known signal pattern, such as in a specific invisible light band (e.g., infrared, ultraviolet). The surgical navigation system 12 may include a locator specifically designed to detect signals within a specific invisible light band and ignore light signals outside that band. In response to the locator detecting a signal pattern associated with a given tracker, the surgical navigation system 12 may determine the position of the tracker relative to the locator based on the angle of the detected pattern. The surgical navigation system 12 may then infer the position of the object to which the tracker is attached based on the determined position of the tracker and the fixed positional relationship between the object and the tracker.

[0045] While the trackers described above enable the surgical navigation system 12 to accurately and effectively track hard tissue objects, such as bones and surgical instruments, in the surgical workspace, these trackers are generally insufficient for tracking soft tissue objects, such as skin and ligaments. Specifically, due to the flexible nature of soft tissue objects, maintaining a fixed positional relationship between the entire soft tissue object and the tracker during the surgical procedure is difficult. Furthermore, attaching a tracker to each of the numerous patient tissues and instruments involved in the surgical procedure would clutter the surgical workspace, making navigation difficult and increasing the cost and complexity of the surgical navigation system 12. Therefore, in addition to tracker-based localization, the surgical navigation system 12 may also implement machine vision to track objects in the surgical workspace during the surgical procedure.

[0046] Specifically, in addition to using locators and attached trackers to detect the position of objects in the surgical workspace, the surgical navigation system 12 may include a vision device configured to generate a depth map of the surface in the workspace (also referred to herein as the target site). The target site can be a variety of different objects or parts. In one instance, the target site is a surgical site, such as a part of an anatomical structure (e.g., bone) that requires treatment or tissue removal. In other instances, the target site can be equipment in the operating room, such as a robotic manipulator, an end effector or tool attached to the robotic manipulator, an operating table, a trolley, a worktable on which a patient can be placed, an imaging system, etc.

[0047] The surgical navigation system 12 can also be configured to identify the positional relationship between the locator and the vision device in a common coordinate system, and can be configured to generate a predicted depth map for the vision device based on the position of an object detected by the locator in the target site, a virtual model corresponding to the object, and the positional relationship. Subsequently, the surgical navigation system 12 can be configured to compare the predicted depth map with the actual depth map generated by the vision device, and based on the comparison, identify portions of the actual depth map that do not match the estimated depth map. The surgical navigation system 12 can then be configured to identify objects in the target site based on the identified portions, and is configured to determine whether the object is an obstacle to the current surgical plan.

[0048] The surgical navigation system 12 can display the relative position of the tracked object during the surgical procedure to assist the surgeon. The surgical navigation system 12 can also control and / or constrain the movement of the robotic manipulator 14 and / or surgical instruments 16 to a virtual boundary associated with the tracked object. For example, the surgical navigation system 12 can identify the target volume of the patient tissue to be treated and potential obstacles in the surgical workspace based on the tracked object. The surgical navigation system 12 can then restrict surgical tools (e.g., the end effector EA of the surgical instrument 16) from contacting anything beyond the target volume of the patient tissue to be treated, thereby improving patient safety and surgical accuracy. The surgical navigation system 12 can also eliminate damage to surgical instruments caused by unintentional contact with other objects, which could also result in undesirable debris at the target site.

[0049] like Figure 1As shown, the surgical navigation system 12 may include a locator 18 and a navigation cart assembly 20. The navigation cart assembly 20 may house a navigation controller 22 configured to implement the functions, features, and processes of the surgical navigation system 12 described herein. In particular, the navigation controller 22 may include a processor 23 programmed to implement the functions, features, and processes of the navigation controller 22 and the surgical navigation system 12 described herein. For example, the processor 23 may be programmed to convert optically based signals received from the locator 18 into locator data, which represents the position of an object attached to a tracker in the surgical workspace.

[0050] The navigation controller 22 is operatively communicative with the user interface 24 of the surgical navigation system 12. The user interface 24 facilitates interaction between the user and the surgical navigation system 12 and the navigation controller 22. For example, the user interface 24 may include one or more output devices that provide information to the user, such as information from the navigation controller 22. The output devices may include a display 25 adapted to be located outside a sterile area including the surgical workspace and a display 26 adapted to be located inside the sterile area. The displays 25, 26 are adjustablely mountable to the navigation cart assembly 20. The user interface 24 may also include one or more input devices that enable the user to input information into the surgical navigation system 12. The input devices may include a keyboard, mouse, and / or touchscreen 28 that the user can interact with to input surgical parameters and control aspects of the navigation controller 22. The input devices may also include a microphone that enables user input via voice recognition technology.

[0051] The locator 18 can be configured to detect the position of one or more objects attached to a tracker in the surgical workspace, such as by detecting the position of the tracker attached to the object. Specifically, the locator 18 can be coupled to the navigation controller 22 of the surgical navigation system 12 and can generate an optical-based signal indicating the position of one or more trackers in the surgical workspace and transmit the optical-based signal to the navigation controller 22. The navigation controller 22 can then be configured to generate locator data based on the optical-based signal and a fixed positional relationship between the object and the tracker, the locator data indicating the position of the object attached to the tracker in the surgical workspace. The object in the target site tracked by the locator 18 may be referred to herein as a "locating object".

[0052] The locator 18 may have a housing 30 that accommodates at least two optical sensors 32. Each of the optical sensors 32 is adapted to detect signals in a specific invisible light band, such as infrared or ultraviolet light, characteristic of the tracker. Although Figure 1Positioner 18 is shown as a single unit with multiple optical sensors 32, but in alternative instances, positioner 18 may comprise separate units arranged around the surgical workspace, each unit having a separate housing and one or more optical sensors 32.

[0053] The optical sensor 32 can be a one-dimensional or two-dimensional charge-coupled device (CCD). For example, the housing 30 can accommodate two two-dimensional CCDs for triangulation of the tracker's position in the surgical workplace, or it can accommodate three one-dimensional CCDs for triangulation of the tracker's position in the surgical workplace. Alternatively, the locator 18 can employ other optical sensing technologies, such as complementary metal-oxide-semiconductor (CMOS) active pixels.

[0054] In some implementations, the navigation system and / or locator 18 is electromagnetic (EM) based. For example, the navigation system may include an EM transceiver coupled to the navigation controller 22 and / or coupled to another computing device, controller, etc. Here, the tracker may include EM components attached thereto (e.g., various types of magnetic trackers, electromagnetic trackers, inductive trackers, etc.), which may be passive or actively excited. The EM transceiver generates an EM field, and the EM components respond with EM signals, such that the tracked state is communicated to (or interpreted by) the navigation controller 22. The navigation controller 22 may analyze the received EM signals to correlate with the relevant state. It should also be understood that implementations of EM-based navigation systems may have different structural configurations than the active tag-based navigation systems shown herein.

[0055] In other implementations, the navigation system and / or locator 18 may be based on one or more types of imaging systems that do not necessarily require the tracker to be fixed to the object in order to determine the associated positioning data. For example, an ultrasound-based imaging system may be provided to facilitate the acquisition of ultrasound images (e.g., specific known structural features of the tracked object, tags or stickers attached to the tracked object, etc.) such that the tracked state (e.g., position, orientation, etc.) is communicated to (or interpreted by) the navigation controller 22 based on the ultrasound images. The ultrasound images may be 2D, 3D, or a combination thereof. The navigation controller 22 may process the ultrasound images in near real-time to determine the tracked state. The ultrasound imaging device may have any suitable configuration and may be compatible with, for example, Figure 1The camera units shown are different. As another example, a fluorescence-based imaging system can be provided to facilitate the acquisition of X-ray images of radiopaque markers (e.g., stickers, labels, etc., with known structural features attached to the tracked object), such that the tracked state is communicated to (or interpreted by) the navigation controller 22 based on the X-ray images. The navigation controller 22 can process the X-ray images in near real-time to determine the tracked state. Similarly, other types of optical-based imaging systems can be provided to facilitate the acquisition of digital images, videos, etc., of a specific known object and / or markers (e.g., stickers, labels, etc., attached to the tracked object) (e.g., based on comparison with a virtual representation of the tracked object or its structural components or features), such that the tracked state is communicated to (or interpreted by) the navigation controller 22 based on the digital images. The navigation controller 22 can process the digital images in near real-time to determine the tracked state.

[0056] Therefore, it should be understood that various types of imaging systems (including multiple imaging systems of the same or different types) can form part of a navigation system without departing from the scope of this disclosure. Those skilled in the art will appreciate that the navigation system and / or locator 18 can have any other suitable components or structures not specifically listed herein. For example, the navigation system can utilize inertial tracking or any combination of tracking technologies alone. Furthermore, with Figure 1 Any of the technologies, methods and / or components associated with the navigation system shown can be implemented in a variety of different ways, and other configurations are also envisioned in this disclosure.

[0057] Positioner 18 can be mounted to an adjustable arm to selectively position optical sensor 32 in an ideal, unobstructed field of view with surgical workspace and target volume. Positioner 18 can be adjusted in at least one degree of freedom by rotating about a rotary joint, and can be adjusted in two or more degrees of freedom.

[0058] As previously described, locator 18 can cooperate with multiple tracking devices (also referred to herein as trackers) to determine the position of an object to which the tracker is attached within the surgical workspace. Typically, the object to which each tracker is attached can be rigid and inflexible, such that movement of the object cannot or should not alter the positional relationship between the object and the tracker. In other words, the relationship between the tracker and the object to which the tracker is attached can remain fixed even if the position of the object changes within the surgical workspace. For example, the tracker can be securely attached to the patient's bones and surgical instruments, such as retractors and surgical instruments 16. In this way, in response to determining the position of the tracker within the surgical workspace using locator 18, navigation controller 22 can infer the position of the object to which the tracker is attached based on the determined position of the tracker.

[0059] For example, when the target volume to be treated is located in the patient's knee joint region, tracker 34 can be securely attached to the patient's femur F, tracker 36 can be securely attached to the patient's tibia T, and tracker 38 can be securely attached to surgical instrument 16. Trackers 34 and 36 can be attached to the femur F and tibia T as shown in U.S. Patent No. 7,725,162, which is hereby incorporated by reference. Trackers 34 and 36 can also be installed as shown in U.S. Patent Application Publication No. 2014 / 0200621, filed January 16, 2014, entitled "Navigation Systems and Methods for Indicating and Reducing Line-of-Sight Errors," which is hereby incorporated by reference. During manufacture, tracker 38 can be integrated into surgical instrument 16 or can be installed separately on surgical instrument 16 in preparation for surgical procedures.

[0060] Before the start of a surgical procedure using surgical system 10, preoperative images of the anatomical structures of interest can be generated, such as anatomical structures defining and / or adjacent to the target volume of patient tissue to be treated by surgical instruments 16. For example, when the target volume of patient tissue to be treated is located in the patient's knee joint region, preoperative images of the patient's femoral F and tibial T can be taken. These images can be based on MRI, radiographic, or computed tomography (CT) scans of the patient's anatomy and can be used to develop virtual models of the anatomical structures. Each virtual model of the anatomical structure may include a three-dimensional model (e.g., point cloud, mesh, CAD) including data representing the whole or at least a portion of the anatomical structure and / or data representing the target volume of the anatomical structure to be treated. These virtual models can be provided to and stored in navigation controller 22 before the surgical procedure.

[0061] In addition to or as an alternative to taking preoperative images, treatment plans can be developed in the operating room based on kinematic studies, skeletal tracking, and other methods. These same methods can also be used to generate the aforementioned virtual models.

[0062] In addition to the virtual model corresponding to the anatomical structures of the patient of interest, prior to the surgical procedure, the navigation controller 22 may receive and store virtual models of other tracked objects of interest in the surgical procedure, such as surgical instruments and other objects potentially present in the surgical workspace (e.g., the surgeon's hand and / or fingers). The navigation controller 22 may also receive and store surgical procedure-specific data, such as the positional relationship between the tracker and objects fixed to the tracker, the positional relationship between the locator 18 and the vision device, and the surgical plan. The surgical plan identifies the patient's anatomical structures involved in the surgical procedure, identifies the instruments used in the surgical procedure, and defines the planned trajectories of the instruments and the planned movement of patient tissues during the surgical procedure.

[0063] During the surgical procedure, the optical sensor 32 of the locator 18 can detect light signals from trackers 34, 36, and 38, such as those in the invisible light band (e.g., infrared or ultraviolet light), and can output optical-based signals indicating the position of trackers 34, 36, and 38 relative to the locator 18 to the navigation controller 22 based on the detected light signals. The navigation controller 22 can then generate locator data indicating the position of the object fixed to trackers 34, 36, and 38 relative to the locator 18 based on the determined positions of trackers 34, 36, and 38 and the known positional relationship between trackers 34, 36, and 38 and the object.

[0064] To supplement tracker-based object tracking provided by locator 18, surgical navigation system 12 may also include a vision device 40. Vision device 40 may be capable of generating three-dimensional images of the surgical workspace in real time. Unlike locator 18, which is limited to detecting and precisely locating the position of invisible light signals transmitted from trackers 34, 36, 38, vision device 40 may be configured to generate three-dimensional images, such as in the form of depth maps, of the surfaces in and around the target volume within the field of view of vision device 40. Vision device 40 may include one or more image sensors 42 and a light source 44. Each of the image sensors 42 may be a CMOS sensor.

[0065] For example, vision device 40 can generate a depth map of the surgical workspace by illuminating exposed surfaces in the surgical workspace with invisible light, such as infrared or ultraviolet light. The surface can then reflect the invisible light back, which can be detected by one or more image sensors 42 of vision device 40. Based on the time-of-flight of the invisible light from its transmission to its detection by vision device 40, vision device 40 can determine the distances between vision device 40 and several points on the exposed surface of the surgical workspace. Vision device 40 can then generate a depth map indicating the distance and angle between vision device 40 and each surface point. Alternatively, vision device 40 can utilize other modalities to generate the depth map, such as, but not limited to, structured light projection, laser ranging, or stereoscopic methods.

[0066] Similar to locator 18, visual device 40 can be positioned as a field of view, preferably unobstructed, with a surgical workspace before the surgical procedure. Visual device 40 can be integrated with locator 18, such as... Figure 1 As shown. Alternatively, vision device 40 may be mounted to a separate adjustable arm to position vision device 40 separately from positioner 18. Vision device 40 may also be directly attached to robot manipulator 14, such as, for example, as described in U.S. Patent 10,531,926 entitled "Systems and Methods for Identifying and Tracking Physical Objects During a Robotic Surgical Procedure," the contents of which are hereby incorporated by reference in their entirety. Vision device 40 may also operatively communicate with navigation controller 22.

[0067] As described above, the navigation controller 22 can be configured to track objects and identify obstacles in the surgical workspace based on tracker-based positioning data generated using the locator 18 and depth maps generated by the vision device 40. Specifically, while the vision device 40 generates the visual depth map of the surgical workspace, the locator 18 can generate optically based data for generating locator data indicating the position of an object fixed to the tracker in the surgical workspace relative to the locator 18. The depth maps generated by the vision device 40 and the locator data generated using the locator 18 can therefore be interleaved in time. In other words, each instance of the locator data generated using the locator 18 can be temporally associated with a different depth map generated by the vision device 40, such that the position of the object indicated in the locator data and the positions of those objects in the associated depth map correspond to the same moment during the surgical procedure.

[0068] In response to determining the locator data, the navigation controller 22 can be configured to generate a anticipated depth map to be captured by the vision device 40 and associated with the locator data. The anticipated depth map may be a temporally associated depth map expected to be generated by the vision device 40, assuming only objects fixed to the tracker exist in the surgical workspace. The navigation controller 22 can be configured to determine the anticipated depth map based on the detected position of the objects fixed to the tracker in the surgical workspace as indicated in the locator data, a virtual model corresponding to the objects, and the positional relationship between the locator 18 and the vision device 40.

[0069] Subsequently, the navigation controller 22 can retrieve the actual depth map generated by the vision device 40 that is temporally associated with the locator data, and can identify portions of the actual depth map that fail to match the expected depth map. The navigation controller 22 can then identify objects in the surgical workspace based on the identified portions, such as objects adjacent to the target volume of the patient tissue to be treated, in addition to objects fixed to the tracker, and can determine whether any such objects constitute an obstacle to the current surgical trajectory.

[0070] The surgical instrument 16 may form part of the end effector of the robotic manipulator 14. The robotic manipulator 14 may include a base 46, a plurality of links 48 extending from the base 46, and a plurality of movable joints for moving the surgical instrument 16 relative to the base 46. The links 48 may be formed as follows: Figure 1 The series arm structure and parallel arm structure shown (e.g.) Figure 3 (as shown) or other suitable structures. The robot manipulator 14 may include the ability to operate in a manual mode, in which a user grasps the end effector of the robot manipulator 14 to cause the surgical instrument 16 to move (e.g., directly or via force / torque sensor measurements causing the robot manipulator 14 to be actively driven). The robot manipulator 14 may also include a semi-autonomous mode, in which the surgical instrument 16 is moved by the robot manipulator 14 along a predefined toolpath (e.g., the active joint of the robot manipulator 14 is actuated to move the surgical instrument 16 without requiring force / torque from the user's end effector). An example of operation in semi-autonomous mode is described in U.S. Patent No. 9,119,655 to Bowling et al., which is hereby incorporated by reference. A separate tracker may be attached to the base 46 of the robot manipulator 14 to track the movement of the base 46 via a locator 18.

[0071] Similar to the surgical navigation system 12, the robot manipulator 14 may house a manipulator controller 50, which includes a processor 52 programmed to implement the processes of the robot manipulator 14, or more specifically, the processes of the manipulator controller 50 described herein. For example, the processor 52 may be programmed to control the operation and movement of the surgical instrument 16, such as by moving the link 48 under the guidance of the surgical navigation system 12.

[0072] During the surgical procedure, the manipulator controller 50 may be configured to determine, for example, the desired location where the surgical instrument 16 should be moved, based on navigation data received from the navigation controller 22. Based on this determination and information associated with the current position of the surgical instrument 16, the manipulator controller 50 may be configured to determine the extent to which each of the links 48 needs to be moved to reposition the surgical instrument 16 from its current position to the desired position. Data indicating the repositioning position of the links 48 may be forwarded to the joint motor controllers (e.g., a joint motor controller for controlling each motor) of the active joints of the robot manipulator 14. In response to receiving such data, the joint motor controllers may be configured to move the links 48 according to the data, and thus move the surgical instrument 16 to the desired position.

[0073] Now for reference Figure 2 The locator 18 and vision device 40 may each include a locator controller 62 and a vision controller 64, respectively. The locator controller 62 may be communicatively coupled to the optical sensor 32 of the locator 18 and to the navigation controller 22. During the surgical procedure, the locator controller 62 may be configured to operate the optical sensor 32 such that it generates optical-based data indicating light signals received from trackers 34, 36, and 38.

[0074] Trackers 34, 36, and 38 may be active trackers, each with at least three active markers for transmitting light signals to optical sensor 32. Trackers 34, 36, and 38 may be powered by an internal battery or may have leads that receive power via navigation controller 22. The active markers of each tracker 34, 36, and 38 may be light-emitting diodes (LEDs) 65 that transmit light such as infrared or ultraviolet light. Each of trackers 34, 36, and 38 may also include a tracker controller 66 connected to the LEDs 65 of trackers 34, 36, and 38 and connected to navigation controller 22. Tracker controller 66 may be configured to, for example, control the rate and sequence of illumination of the LEDs 65 of trackers 34, 36, and 38 under the guidance of navigation controller 22. For example, the tracker controller 66 of trackers 34, 36, and 38 may cause the LEDs 65 of each tracker 34, 36, and 38 to light up at different rates and / or times to facilitate the navigation controller 22 in distinguishing trackers 34, 36, and 38.

[0075] The sampling rate of the optical sensor 32 is the rate at which the optical sensor 32 receives light signals from the sequentially emitting LEDs 65. The optical sensor 32 may have a sampling rate of 100 Hz or higher, more preferably 300 Hz or higher, or most preferably 500 Hz or higher. For example, the optical sensor 32 may have a sampling rate of 8000 Hz.

[0076] Trackers 34, 36, and 38 are not active trackers, but rather passive trackers that include passive markers (not shown), such as those reflecting light emitted from locator 18 (e.g., from light source 44). Figure 1 The reflected light is then received by the optical sensor 32.

[0077] In response to receiving optical signals from trackers 34, 36, and 38, optical sensor 32 can output optical-based data to locator controller 62. This optical-based data indicates the position of trackers 34, 36, and 38 relative to locator 18, and correspondingly, indicates the position of an object securely attached to trackers 34, 36, and 38 relative to locator 18. Specifically, each optical sensor 32 may include a one-dimensional or two-dimensional sensor region for detecting optical signals from trackers 34, 36, and 38, and responsively indicates the position where each optical signal is detected within the sensor region. The detection position of each optical signal within a given sensor region may be based on the angle at which the optical sensor 32 receiving the optical signal, including the sensor region, and similarly may correspond to the position of the source of the optical signal in the surgical workspace.

[0078] Therefore, in response to receiving light signals from trackers 34, 36, and 38, each optical sensor 32 can generate optical-based data indicating the location where a light signal is detected within its sensor area. The optical sensor 32 can transmit this optical-based data to the locator controller 62, which in turn can transmit it to the navigation controller 22. The navigation controller 22 can then generate tracker position data indicating the positions of trackers 34, 36, and 38 relative to the locator 18 based on the optical-based data. For example, the navigation controller 22 can triangulate the position of LED 65 relative to the locator 18 based on the optical-based data and can apply stored positional relationships between trackers 34, 36, and 38 and markers to the determined position of LED 65 relative to the locator 18 to determine the positions of trackers 34, 36, and 38 relative to the locator 18.

[0079] Subsequently, the navigation controller 22 can generate locator data based on the tracker position data, indicating the position of the object securely attached to the trackers 34, 36, 38 relative to the locator 18. Specifically, the navigation controller 22 can retrieve stored positional relationships between the trackers 34, 36, 38 and the objects to which they are attached, and can apply these positional relationships to the tracker position data to determine the position of the object fixed to the trackers 34, 36, 38 relative to the locator 18. Alternatively, the locator controller 62 can be configured to determine the tracker position data and / or locator data based on received optical-based data, and can transmit the tracker position data and / or locator data to the navigation controller 22 for further processing.

[0080] The vision controller 64 is communicatively coupled to the light source 44 and one or more image sensors 42 of the vision device 40, and is also coupled to the navigation controller 22. While the locator controller 62 causes the locator 18 to generate optical-based data indicating the position of trackers 34, 36, 38 in the surgical workspace, the vision controller 64 can cause the vision device 40 to generate a depth map of the exposed surfaces of the surgical workspace. Specifically, the vision controller 64 can cause the image sensors 42 to generate image data that forms the basis of the depth map, and can generate the depth map based on the image data. The vision controller 64 can then forward the depth map to the navigation controller 22 for further processing. Alternatively, the vision controller 64 can transmit image data to the navigation controller 22, which can then generate a depth map based on the received image data.

[0081] Typically, the depth map generated by the vision device 40 indicates the distance between the vision device 40 and surfaces within its field of view. In other words, the depth map shows the topography of surfaces in the surgical workspace from the viewpoint of the vision device 40. Each depth map generated by the vision device 40 may include multiple image components that form an image frame of the vision device 40. Each image component may approximate a pixel of the depth map and may define a vector from the center of the vision device 40 to a point on the surface within its field of view. For example, the location of an image component within an image frame of the vision device 40 may correspond to the horizontal and vertical components of the vector defined by the image component, and the color of the image component may correspond to the depth component of the vector defined by the image component. As an example, image components representing surface points closer to the vision device 40 in the surgical workspace may have a brighter color compared to those representing surface points farther from the vision device 40.

[0082] The vision device 40 may be a depth camera including one or more depth sensors 68. The depth sensors 68 may be adapted to detect light, such as invisible light, reflected from surfaces exiting the field of view of the depth sensor 68. During the surgical procedure, the vision controller 64 may cause the light source 44 to illuminate the target area using invisible light, such as infrared or ultraviolet light. The depth sensors 68 can then detect the reflection of the invisible light from the surface of the target area, which allows the vision controller 64 to generate a depth map.

[0083] For example, the vision controller 64 can generate a depth map based on the time it takes for light transmitted from the light source 44 to reflect off points on the exposed surface of the target site (i.e., a time-of-flight method), where the time corresponds to the distance between the vision device 40 and the respective points. The vision controller 64 can then use these determined distances to generate the depth map. As an alternative example, the light source 44 can project a known structured invisible light pattern onto the exposed surface of the surgical site. The depth sensor 68 can then detect the reflection of the known pattern, which may be distorted based on the topography of the surface in the target site. The vision controller 64 can thus be configured to generate a depth map of the target site based on a comparison between the known pattern and a distorted version of the pattern detected by the depth sensor 68.

[0084] Alternatively, the vision device 40 may be an RGB camera including one or more RGB sensors 70. The RGB sensors 70 may be configured to generate color images of exposed surfaces in the target area, and the vision controller 64 may be configured to generate depth maps based on the color images.

[0085] For example, similar to the structured light method described above, the vision controller 64 can be configured to cause the light source 44 to project a known structured light pattern onto the target region, such as with a color deviating from the color in the target region. The RGB sensor 70 can then generate an RGB image of the target region, which can depict a distorted version of the known structured light pattern based on the surface topography of the target region. The vision controller 64 can extract the distorted version of the known structured light pattern from the RGB image, such as using pattern recognition, edge detection, and color recognition, and can determine a depth map based on a comparison between the known structured light pattern and the extracted distorted version.

[0086] As an alternative, the vision device 40 can be configured to generate a depth map of the target area using principles of stereoscopic vision. More specifically, multiple image sensors 42, such as multiple depth sensors 68 or RGB sensors 70, can be positioned to have fields of view of the target area from different angles. The vision controller 64 can be configured to cause each image sensor 42 to simultaneously generate an image of the target area from a different angle. For example, when the image sensor 42 is a depth sensor 68, the vision controller 64 can be configured to cause a light source 44 to illuminate the exposed surface of the target area with a pattern of invisible light, and each depth sensor 68 can image the pattern of invisible light reflected from the exposed surface from different angles. The vision controller 64 can then determine the three-dimensional position of points on the surface of the target area relative to the vision device 40 based on the positions of surface points in each image and the known positional relationship between the image sensors 42. The vision controller 64 can then generate a depth map based on the determined three-dimensional positions.

[0087] To reduce interference between the locator 18 and the vision device 40 during surgical procedures, the locator 18 and the vision device 40 may be configured to operate in different spectral bands to detect the position of an object in the target area. Alternatively, when the vision device 40 uses a light source 44 to illuminate an exposed surface in the target area, such as when the vision device 40 operates in an invisible light band, the locator 18 may be configured to operate with a sufficiently short exposure time such that the light source 44 of the vision device 40 is invisible to the locator 18.

[0088] As previously described, navigation controller 22 may include processor 23, which is programmed to perform the functions, features, and processes of navigation controller 22 described herein, such as calculating a desired depth map based on locator data generated using locator 18, and determining objects adjacent to a target volume of patient tissue to be treated at the surgical site by comparing the desired depth map with an actual depth map generated by vision device 40. In addition to processor 23, navigation controller 22 may include memory 72 and non-volatile storage devices 74, each operatively coupled to processor 23.

[0089] Processor 23 may include one or more devices selected from the group consisting of a microprocessor, microcontroller, digital signal processor, microcomputer, central processing unit, field-programmable gate array, programmable logic device, state machine, logic circuit, analog circuit, digital circuit, or any other device that manipulates (analog or digital) signals based on operating instructions stored in memory 72. Memory 72 may include a single memory device or multiple memory devices, including but not limited to: read-only memory (ROM), random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information. Non-volatile storage device 74 may include one or more persistent data storage devices, such as hard disk drives, optical disk drives, magnetic tape drives, non-volatile solid-state devices, or any other device capable of persistently storing information.

[0090] Non-volatile storage device 74 can store software, such as positioning engine 76, conversion engine 78, vision engine 80, and surgical navigator 81. The software may be embodied in computer-executable instructions compiled or interpreted by a variety of programming languages ​​and / or technologies, including but not limited to one or a combination of the following: Java, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL.

[0091] Processor 23 can operate under the control of software stored in non-volatile storage device 74. Specifically, processor 23 can be configured to execute the software as an active process by reading into memory 72 and executing computer-executable instructions of the software. When executed by processor 23, the computer-executable instructions can be configured to cause processor 23 to implement the configuration functions, features, and processes of the navigation controller 22 described herein. Therefore, the software can be configured to cause navigation controller 22 to implement the functions, features, and processes of navigation controller 22 described herein by means of the computer-executable instructions of the software being configured, which, when executed by processor 23, are configured to cause processor 23 of navigation controller 22 to implement the processes of navigation controller 22 described herein.

[0092] The non-volatile storage device 74 of the navigation controller 22 can also store data that facilitates the operation of the navigation controller 22. Specifically, the software of the navigation controller 22 can be configured to access the data stored in the non-volatile storage device 74 and to implement the functions, features, and processes of the navigation controller 22 described herein based on the data.

[0093] For example, but not limited to, the data stored in the non-volatile storage device 74 may include model data 82, conversion data 83, and surgical plan 84. Model data 82 may include virtual models of anatomical structures of interest in the surgical procedure, including virtual models of potential obstacles such as the surgeon's hand or fingers, and virtual models of surgical instruments being used in the surgical procedure, as described above. Conversion data 83 may include positional relationships that enable the conversion of the position of an object in the surgical workspace relative to one device, such as trackers 34, 36, 38 or locator 18 or vision device 40, into the position of the object relative to another device. For example, conversion data 83 may clarify the fixed positional relationships between trackers 34, 36, 38 and objects securely attached to trackers 34, 36, 38, and the positional relationship between locator 18 and vision device 40. Surgical plan 84 may identify the target volumes of patient anatomy involved in the surgical procedure, identify the instruments used in the surgical procedure, and define the planned trajectories of instruments and the planned movement of patient tissues during the surgical procedure.

[0094] Referring again to the software running on the navigation controller 22, the positioning engine 76 can be configured to generate positioning data, such as based on optical data generated by the optical sensor 32 of the locator 18, indicating the position of an object securely attached to trackers 34, 36, 38 relative to the locator 18. The conversion engine 78 can be configured to convert the position of an object relative to one device of the surgical system 10 to the position of an object relative to another device of the surgical system 10, such as based on positional relationships represented by conversion data 83. The vision engine 80 can be configured to generate a desired depth map based on the positioning data generated by the positioning engine 76 and the conversion data 83, and is configured to compare the desired depth map with an actual depth map generated by the vision device 40 to identify and track objects in the surgical workspace. The surgical navigator 81 can be configured to provide surgical guidance based on the identification and tracking determined by the vision engine 80. Further details regarding the functionality of these software components are discussed below.

[0095] Although not shown, each of the manipulator controller 50, the positioner controller 62, and the vision controller 64 may also include a processor, a memory, and a non-volatile storage device including data and software configured to implement the functions, features, and processes of the controllers described herein when executing their computer-executable instructions.

[0096] Although the exemplary surgical system 10 is Figure 1 As shown in, and in Figure 2Further details are provided below, but this example is not intended to be limiting. In practice, the surgical system 10 may have more or fewer components and may use alternative components and / or implementations. For example, all or part of the locator engine 76 may be implemented by the locator controller 62. As an example, the locator controller 62 may be configured to generate locator data based on model data 82 that indicates the position of an object securely attached to trackers 34, 36, 38.

[0097] Figure 3 The coordinate systems of various objects and devices used with the surgical system 10 are shown. The navigation controller 22 can be configured to, for example via a transformation engine 78, transform the position of an object in one coordinate system to the position of an object in another coordinate system based on positional relationships defined in transformation data 83 stored in the navigation controller 22. Such transformations enable the navigation controller 22 to track objects in the surgical system 10 relative to a common coordinate system. Furthermore, the transformations enable the navigation controller 22, for example via a vision engine 80, to calculate a desired depth map to be generated by the vision device 40 based on positioning data generated by the positioning engine 76, and to identify and track objects based on a comparison between the actual depth map generated by the vision device 40 and the desired depth map. As a non-limiting example, each positional relationship defined by the transformation data 83 and capable of transformation between coordinate systems can be represented by a transformation matrix defined by the transformation data 83.

[0098] The navigation controller 22 can be configured to reference a locator coordinate system LCLZ to track objects in a target area, such as objects attached to trackers 34, 36, and 38. The locator coordinate system LCLZ may include an origin and orientation defined by the positions of the x, y, and z axes relative to the surgical workspace. The locator coordinate system LCLZ may be fixed to and centered on locator 18. Specifically, the center point of locator 18 may define the origin of the locator coordinate system LCLZ. The locator data described above, which indicates the position of an object relative to locator 18 determined using locator 18, can similarly indicate the position of such objects in the locator coordinate system LCLZ.

[0099] During the procedure, one objective is to maintain the locator coordinate system (LCLZ) in a known position. An accelerometer may be mounted to the locator 18 to detect sudden or accidental movement of the LCLZ, which may occur if the locator 18 is unintentionally bumped by the surgeon. In response to a detected movement of the LCLZ, the navigation controller 22 may be configured to, for example, present an alarm to the surgeon via the user interface 24 through the surgical navigator 81 to stop surgical navigation and / or be configured to transmit a signal to the manipulator controller 50 causing the manipulator controller 50 to stop the movement of the surgical instrument 16 until the surgical system 10 is recalibrated.

[0100] Each object tracked by the surgical system 10 may also have its own coordinate system, which is fixed to the object and centered on the object, and is separate from the locator coordinate system LCLZ. For example, trackers 34, 36, and 38 may be fixed and centered within the bone tracker coordinate system BTRK1, the bone tracker coordinate system BTRK2, and the instrument tracker coordinate system TLTR, respectively. The patient's femur F may be fixed and centered within the femoral coordinate system FBONE, and the patient's tibia T may be fixed and centered within the tibial coordinate system TBONE. Prior to the surgical procedure, preoperative images and / or virtual models of each tracked object, such as the femur F, tibia T, and surgical instruments 16, may be mapped to the object based on its fixed position in the coordinate system, such as by mapping to and fixing it within the object's coordinate system.

[0101] During the initial phase of the surgical procedure, trackers 34 and 36 can be securely attached to the patient's femur F and tibia T, respectively. The positions of coordinate systems FBONE and TBONE can then be mapped to coordinate systems BTRK1 and BTRK2, respectively. For example, pointer-type instruments P with their own trackers PT are disclosed in U.S. Patent No. 7,725,162 to Malackowski et al., which is hereby incorporated by reference. Figure 1 This can be used to register the femoral coordinate system FBONE and the tibial coordinate system TBONE to the bone tracker coordinate systems BTRK1 and BTRK2, respectively. The fixed positional relationship between the femoral coordinate system FBONE and the bone tracker coordinate system BTRK1, and the fixed positional relationship between the tibial coordinate system TBONE and the bone tracker coordinate system BTRK2, can be stored as transformation data 83 on the navigation controller 22.

[0102] Given the fixed spatial relationship between the femur F and tibia T and their trackers 34 and 36, the navigation controller 22 can, for example via the transformation engine 78, convert the position of the femur F in the femoral coordinate system FBONE to the position of the femur F in the bone tracker coordinate system BTRK1, and can convert the position of the tibia T in the tibial coordinate system TBONE to the position of the tibia T in the bone tracker coordinate system BTRK2. Therefore, by determining the positions of trackers 34 and 36 in the positioning coordinate system LCLZ using the locator 18, the navigation controller 22 can determine the positions of the femoral coordinate system FBONE and the tibial coordinate system TBONE in the positioning coordinate system LCLZ, and can correspondingly determine the positions of the femur F and the tibia T in the positioning coordinate system LCLZ.

[0103] Similarly, the treatment end of the surgical instrument 16 can be fixed within its coordinate system EAPP and centered within it. For example, the origin of the coordinate system EAPP can be fixed to the centroid of the surgical cutting drill. Before the program begins, the position of the coordinate system EAPP and the corresponding position of the treatment end of the surgical instrument 16 can be fixed within the instrument tracker coordinate system TLTR of the tracker 38. The fixed positional relationship between the coordinate system EAPP and the instrument tracker coordinate system TLTR can also be stored in the navigation controller 22 as transformation data 83. Therefore, by determining the position of the instrument tracker coordinate system TLTR in the positioning coordinate system LCLZ using the locator 18, the navigation controller 22 can determine the position of the coordinate system EAPP in the positioning coordinate system LCLZ, for example via the transformation engine 78, based on the positional relationship between the instrument tracker coordinate system TLTR and the coordinate system EAAP, and can correspondingly determine the position of the treatment end of the surgical instrument 16 in the positioning coordinate system LCLZ.

[0104] The vision device 40 can also be fixed within and centered in the vision coordinate system VIS. The origin of the vision coordinate system VIS can represent the centroid of the vision device 40. Each actual depth map generated by the vision device 40, as described above, indicating the position of the exposed surface in the target area relative to the vision device 40, can similarly indicate the position of the exposed surface in the coordinate system VIS.

[0105] When the vision device 40 is integrated with the positioner 18, such as Figure 1 As shown, the visual coordinate system VIS and the locator coordinate system LCLZ can be considered equivalent. In other words, the position of an object or coordinate system in the locator coordinate system LCLZ can be so close to or equal to the position of the object or coordinate system in the visual coordinate system VIS that no transformation is required. Alternatively, since the visual coordinate system VIS can be fixed within the locator coordinate system LCLZ when the vision device 40 is integrated with the locator 18, or vice versa, the positional relationship between the visual coordinate system VIS and the locator coordinate system LCLZ, and the corresponding positional relationship between the vision device 40 and the locator 18, can be determined during the manufacture of the surgical system 10 and can be stored in the navigation controller 22 at the factory as transformation data 83.

[0106] When the vision device 40 is separated from the locator 18, the vision device 40 may include a tracker (not shown) rigidly mounted to the housing of the vision device 40 to establish positional relationships between the visual coordinate system VIS and the locator coordinate system LCLZ, and correspondingly between the vision device 40 and the locator 18. The navigation controller 22 may preload the positional relationship between the tracker's coordinate system and the visual coordinate system VIS as transformation data 83. Therefore, by determining the position of the tracker's coordinate system in the positioning coordinate system LCLZ using the locator 18, the navigation controller 22 can determine the position of the visual coordinate system VIS in the positioning coordinate system LCLZ based on the stored positional relationship between the tracker's coordinate system and the visual coordinate system VIS, and correspondingly determine the position of the vision device 40 in the locator coordinate system LCLZ. Further correspondingly, the navigation controller 22 can determine the position of the vision device 40 relative to the locator 18 in the locator coordinate system LCLZ and the visual coordinate system VIS.

[0107] Alternatively, the navigation controller 22 may be configured to identify the positional relationship between the locator's common coordinate system LCLZ and the visual coordinate system VIS based on a common light pattern inserted into the target location and detectable by both the locator 18 and the vision device 40. For example, after the locator 18 and the vision device 40 are positioned to have a field of view of the target location, a pattern of light (such as invisible light) may be projected onto the target location, which may reflect the light pattern back to the locator 18 and the vision device 40. The navigation controller 22 may cause the light source 44 of the vision device 40 to project the light pattern into the target location, or the light pattern may be projected using a separate light projector (not shown). As another example, a tracker or other physical device, such as a pointer PT, may be used with a marker configured to transmit a light pattern detectable by both the locator 18 and the vision device 40. Figure 1 It can be placed inside the target area and within the field of view of the locator 18 and the vision device 40.

[0108] The navigation controller 22 can be configured to generate positioning data, such as via the locator engine 76 and using the locator 18, indicating the position of the light pattern in a locator coordinate system LCLZ specific to the locator 18. The navigation controller 22 can also receive a calibrated depth map showing the light pattern from the vision device 40 and can be configured to identify the position of the light pattern in the visual coordinate system VIS based on the calibrated depth map, such as via the transformation engine 78. The navigation controller 22 can then be configured to determine the positional relationship between the positioning coordinate system LCLZ and the visual coordinate system LCLZ, such as via the transformation engine 78, based on the determined position of the projected pattern in both the positioning coordinate system LCLZ and the visual coordinate system LCLZ, such as using a regression algorithm.

[0109] Figure 4A method 100 is shown that uses tracker-based localization and machine vision to track objects in a surgical workspace and determine whether an object obstructs surgical planning. Method 100 may be performed by a surgical navigation system 12, or more specifically by a navigation controller 22.

[0110] In box 102, the positional relationship between the locator 18 and the vision device 40 in a common coordinate system can be identified. Specifically, the navigation controller 22 can be configured to identify the positional relationship between the locator 18 and the vision device 40, and correspondingly the positional relationship between the locator coordinate system LCLZ and the vision coordinate system VIS, using any of the methods described above, such as via the transformation engine 78. For example, the tracker can be fixed to the vision device 40, or a light pattern can be placed in a target area that can be detected by both the locator 18 and the vision device 40. Alternatively, during manufacturing, when the locator 18 is integrated with or otherwise fixed relative to the vision device 40, the positional relationship can be determined during manufacturing and pre-stored as transformation data 83 in the navigation controller 22.

[0111] In box 104, virtual models corresponding to one or more objects in the target site can be accessed, for example, based on transformation data 83. Transformation data 83 may indicate the objects to which trackers such as trackers 34, 36, and 38 are attached in the target site. Navigation controller 22 may be configured to retrieve a virtual model of each of the objects attached to the trackers. In some cases, one or more of these retrieved virtual models may also define the target volume to be treated during the surgical procedure.

[0112] Figure 5 A target site 200 is shown for a patient undergoing knee replacement surgery. The target site 200 may include a portion of the patient's femur F, which may contain a target volume 202 of bone tissue to be removed using surgical instruments (e.g., surgical instrument 16). The target site 200 may also include soft tissues adjacent to the target volume 202, such as ligaments 204 and epidermal tissue 206. The target site 200 may also include surgical tools, such as a retractor 208 positioned to retract the epidermal tissue 206 and provide access to the patient's femur F. The target site 200 may additionally include a tracker 209 securely attached to the patient's femur F. The navigation controller 22 can therefore retrieve a virtual model corresponding to the femur F from model data 82, an instance of which is... Figure 6 As shown in the image.

[0113] Refer again Figure 4In block 106, locator 18 can be used to detect the position of an object attached to a tracker at the target location. Specifically, locator 18 can be configured to generate optically based data indicating the position of each tracker coordinate system and, correspondingly, the position of each tracker in the locator coordinate system LCLZ, as described above. Navigation controller 22 can then be configured, for example via positioning engine 76, to identify the position of the object attached to the tracker in the locator coordinate system LCLZ, and, correspondingly, the position of the object in the locator coordinate system LCLZ, based on the detected positions of the trackers and their coordinate systems, and the positional relationship between the trackers and the object indicated in transformation data 83. Because a virtual model of each object attached to a tracker can be mapped to the object's coordinate system in transformation data 83, navigation controller 22 can similarly identify the position of the virtual model of each object attached to the tracker in the locator coordinate system LCLZ based on the detected positions of the trackers to which the object is attached and the positional relationship between the trackers and the object.

[0114] Figure 6 Show Figure 5 The target site 200 is a continuation and shows a virtual model 210 that corresponds to the patient's femur F. The virtual model 210 can be attached to the tracker 209 in the target site 200. The virtual model 210 can be positioned in the locator coordinate system LCLZ based on the position of the tracker 209 and the femur F in the locator coordinate system LCLZ determined using the locator 18. The virtual model 210 can define a virtual target volume 212 to be removed from the femur F during treatment, the virtual target volume 212 being correlated with... Figure 5 The target volume 202 shown corresponds to this.

[0115] In box 108, a desired depth map is generated, such as based on the accessed virtual model, the detected position of the object corresponding to the virtual model in the locator coordinate system LCLZ, and the positional relationship between the locator 18 and the vision device 40 in the common coordinate system. As previously described, the position of the virtual model in the locator coordinate system LCLZ can correspond to the position of the object in the locator coordinate system LCLZ determined using the locator 18. The navigation controller 22 can be configured to, for example, convert the position of the virtual model in the locator coordinate system LCLZ to the position of the virtual model in the vision coordinate system VIS via the vision engine 80 based on the positional relationship between the locator 18 and the vision device 40 and the corresponding positional relationship between the locator coordinate system LCLZ and the vision coordinate system VIS in the common coordinate system.

[0116] Subsequently, navigation controller 22 can generate a desired depth map based on the position of the virtual model in the visual coordinate system (VIS). As described above, the depth map generated by vision device 40 can show the position (e.g., depth and location) of the exposed object surface in the target area relative to vision device 40. The position of the virtual model in the visual coordinate system (VIS) can generally indicate the position of the object surface represented by the virtual model relative to vision device 40, which can be fixed within the visual coordinate system (VIS). Therefore, navigation controller 22 can be configured to simulate, for example, the depth map expected to be generated by vision device 40 with the field of view of the target area based on the determined position of the virtual model in the visual coordinate system (VIS) via vision engine 80, assuming that there are no other objects in the target area.

[0117] Figure 7 It shows that it can be based on Figure 6 The expected depth map generated by the virtual model 210. Figure 7 The expected depth map can be simulated by the visual device 40 based on the transformed position of the virtual model 210 in the visual coordinate system VIS to simulate the expected depth map of the patient's femur F, assuming that there are no other objects in the target area, such as ligaments 204, epidermal tissue 206, retractor 208 and tracker 209. Figure 7 The expected depth map can also be cropped to the region of interest, which will be discussed in more detail below.

[0118] In box 110, the actual depth map captured by vision device 40 can be received. Specifically, while locator 18 generates locator data in box 106 indicating the position of the object to be tracked and attached to the target site, vision device 40 can generate a depth map of the target site, as described above. In this way, the depth map can be interleaved with the locator data in time, and also with the estimated depth map generated based on the locator data in time. In other words, both the actual depth map and the expected depth map can represent the target site at substantially the same time during the surgical procedure.

[0119] Figure 8 It can be shown that it can be made by having Figure 5A depth map generated by a vision device 40 of the field of view of the depicted target region 200, wherein tracker 209 is removed for simplicity. The depth map may include several image components approximating pixels, which are arranged in a matrix to form an image frame of the depth map. Grid 214 has been artificially placed on the illustrated depth map to highlight exemplary image components. The location of each image component in the depth map image frame may represent the horizontal and vertical distances from the central viewpoint of the vision device 40, and the brightness of each image component may correspond to the distance of the object surface point represented by the image component from the vision device 40. In the illustrated example, brighter image components represent surface points in the target region closer to the vision device 40, while darker components represent surface points in the target region farther from the vision device 40.

[0120] In box 112, the actual depth map can be cropped into a region of interest (ROI) for the surgical procedure, based on the virtual model accessed in box 104, the detected position of the object corresponding to the virtual model in the locator coordinate system LCLZ, and the positional relationship between the locator 18 and the vision device 40 in the common coordinate system. As explained in further detail below, the actual depth map can be compared with the expected depth map to identify objects in the target site and determine whether any such objects might obstruct treatment of the target volume in the target site. The larger the size of the actual depth map and the expected depth map being compared, the greater the computational cost involved in the comparison. Therefore, the navigation controller 22 can be configured to crop the actual depth map into the ROI, such as via the vision engine 80, based on the position of the virtual model in the visual coordinate system VIS, to reduce the size of the compared depth images. As described above, the position of the virtual model in the visual coordinate system VIS can be determined based on the determined position in the object locator coordinate system LCLZ and the positional relationship between the locator 18 and the vision device 40 in the common coordinate system.

[0121] For example, the virtual model accessed in box 104 can define the target volume to be treated during the surgical procedure. Therefore, the position of the virtual model in the visual coordinate system (VIS) can indicate the position of the target volume in the VIS, and correspondingly, can indicate the position of the target volume in the actual depth map generated by the vision device 40. The navigation controller 22 can be configured to, for example, crop the actual depth map via the vision engine 80 to remove any region that is more than a threshold distance from the position of the target volume in the VIS. Alternatively or additionally, the navigation controller 22 can be configured to, for example, center a user-selected or program-specific shape at the position of the target volume in the actual depth map via the vision engine 80, and is configured to remove any region outside the shape of the actual depth map. The navigation controller 22 can be configured to, for example, limit the size and shape of the expected depth map to the size and shape of the cropped actual depth map during or after the calculation of the expected depth map.

[0122] Figure 9 Show Figure 8 Based on and Figure 6 The virtual model 210 corresponding to the femur F in the image is cropped to the actual depth map of the ROI. The virtual model 210 can define a virtual target volume 212 corresponding to the target volume 202 to be treated during the surgical procedure. The navigation controller 22 can crop based on the determined position of the target volume in the visual coordinate system (VIS), such as by centering the selected shape on the target volume in the depth map and removing areas outside the shape in the depth map. Figure 8 The position of the target volume in the visual coordinate system (VIS) can indicate the position of the target volume in the actual depth map. Figure 7 The expected depth map is similarly limited to the size and shape of the actual depth map being clipped.

[0123] In boxes 114 and 116, portions of the depth map that fail to match the expected depth map are identified. Specifically, in box 114, the actual depth map can be compared to the expected depth map, for example, by calculating the difference between the actual depth map and the expected depth map. The navigation controller 22 can be configured to calculate the difference between the expected depth map and the actual depth map, for example, via the vision engine 80, by calculating the difference between the depths at each pair of corresponding image components in the expected depth map and the actual depth map. A pair of corresponding image components in the expected depth map and the actual depth map may include the image components of each depth map at the same horizontal and vertical positioning. Assuming that the actual depth map has been cropped to the ROI in box 112, the depth map compared in box 114 may be the cropped depth map.

[0124] The difference between the actual depth map and the estimated depth can indicate objects in the target area that have not yet been identified and tracked, such as objects that were not or could not be adequately tracked using the attached tracker (e.g., soft tissue, a surgeon's hand). This difference can be represented by a difference depth map, where each image component of the difference depth map indicates the depth difference calculated for a corresponding image component located at the same horizontal and vertical position as the image component in the difference depth map within both the actual and expected depth maps. Figure 10 Shown in Figure 9 Actual depth map and Figure 7 The difference depth map is calculated between the expected depth maps.

[0125] Corresponding image components from the actual depth map and the expected depth map, indicating the same depth, will result in zero depth difference, and can correspond to objects previously identified and tracked, such as those using trackers and locators 18. Zero depth difference can be represented in the depth map of the difference by image components with maximum brightness or color and / or hue specific to zero depth. Figure 10 In the difference depth map, regions 216, 218, and 220 represent corresponding image components from the actual depth map and the expected depth map that have zero depth difference.

[0126] The discrepancy between corresponding image components indicating the same depth in the actual depth map and the expected depth map will result in a non-zero depth difference, which may correspond to objects not previously identified and tracked, such as those using trackers and locators 18. This non-zero depth difference can be represented in the depth map by image components with brightness less than the maximum brightness or colors and / or hues different from those specific to zero depth. Figure 10 In the difference depth map, the darker areas adjacent to regions 216, 218, and 210 represent the corresponding image components of the actual depth map and the expected depth map that have non-zero depth differences.

[0127] In box 116, the calculated difference can be filtered based on one or more object thresholds. The object thresholds can be designed to distinguish between non-zero differences due to noise or insignificant calibration inaccuracies and non-zero differences due to the presence of other objects in the target area. Object thresholds may include, but are not limited to, threshold depth and / or minimum size thresholds, each of which can be non-zero.

[0128] As an example, for each of one or more non-zero segments in a difference depth map, navigation controller 22 may be configured to determine whether the non-zero segment indicates an absolute depth greater than a depth threshold. Specifically, the difference depth map may include one or more non-zero segments, each of which includes a set of consecutive image components, each of which indicates a non-zero depth difference. If the magnitude (without reference sign) of the non-zero depth difference indicated by each image component in the non-zero segment is greater than a depth threshold, then the non-zero segment of the difference depth map is considered to have an absolute depth greater than the depth threshold. In response to determining that a non-zero segment of the difference indicates an absolute depth greater than the threshold depth, navigation controller 22 may be configured to, for example, identify the segment of the actual depth map corresponding to the non-zero segment of the difference as a portion of the actual depth map that fails to match the estimated depth map, by means of a segment of the actual depth map being located at the same horizontal and vertical position in the actual depth map as the non-zero segment in the difference depth map.

[0129] As another example, for each non-zero segment of the difference, navigation controller 22 can be configured to determine whether the size (e.g., area) of the non-zero segment is greater than a minimum size threshold. In response to determining that the size of the non-zero segment is greater than the minimum size threshold, navigation controller 22 can be configured, for example via vision engine 80, to identify the segment of the actual depth map corresponding to the non-zero segment of the difference as a portion of the actual depth map that fails to match the expected depth map, by means of a segment in the actual depth map being located at the same horizontal and vertical position in the actual depth map as the non-zero segment in the difference depth map.

[0130] In another instance, the navigation controller 22 may be configured to identify the segment of the actual depth map corresponding to the non-zero segment of the difference as a portion of the actual depth map that fails to match the estimated depth map in response to determining that the size of the non-zero segment is greater than a minimum size threshold and that the non-zero segment indicates an absolute depth greater than the threshold depth.

[0131] In box 118, the presence of an object in the target area can be determined based on the filtered differences. Specifically, the navigation controller 22 can be configured to determine, for example, via the vision engine 80, whether any portion of the actual depth map that does not match the expected depth map is identified. If no (the "No" branch of box 118), method 100 can return to box 106 to again use the locator 18 to detect the location of the object attached to the tracker. If yes (the "Yes" branch of box 118), method 100 can proceed to box 120 to identify the object in the target area by applying machine vision techniques to the portion of the actual depth map that does not match the expected depth map.

[0132] In block 120, navigation controller 22 may be configured to apply machine vision techniques, such as via vision engine 80, to identified portions of the actual depth map that fail to match the expected depth map, to identify objects in the target area based on the identified portions. For example, but not limited to, navigation controller 22 may be configured to utilize pattern recognition, edge detection, color recognition, wavelength analysis, image component intensity analysis (e.g., pixel or voxel intensity analysis), depth analysis, and metrics generated by machine learning to segment objects represented in the identified portions of the actual depth map. As some examples, edge-separated regions of the identified portions that have different regular patterns, different color palettes, and / or indicate different depth ranges may correspond to different objects. As another example, the surfaces of different objects (e.g., different tissues) in the target area may generate signals of different wavelengths and / or different intensities, which are reflected to and detected by vision device 40. The vision device 40 may be configured to output such information for each image component of the actual depth map, and based on this information, the navigation controller 22 may be configured to segment different objects in the identified portion based on the changing wavelength and / or signal intensity occurring across the identified portion of the actual depth map. If the navigation controller 22 cannot detect multiple objects in the identified portion using machine vision, the navigation controller 22 may be configured to treat the entire identified portion as a single object in the target area.

[0133] Alternatively, the navigation controller 22 may be configured to identify objects in the identified portion of the actual depth, such as via the vision engine 80, based on model data 82 stored in the navigation controller 22. Identification differs from segmentation in that identification may assign a label describing the object type to each object represented in the actual depth map, such as identifying an object as a ligament, retractor, epidermal tissue, etc. Identification of each object in the target area allows the navigation controller 22, such as via the vision engine 80, to model the entire object rather than just its surface, and to better predict object movement and other reactions during the surgical procedure. This enables the surgical navigator 81 of the navigation controller 22 to make increasingly informed navigation decisions.

[0134] As described above, the model data 82 stored in the navigation controller 22 can define a three-dimensional model corresponding to a potential object in the target area. The model data 82 can also define predetermined contours for various potential objects in the target area, each contour illustrating one or more object-specific features that help the navigation controller 22 identify the object based on the actual depth map. For example, the contour of a given object may include, but is not limited to, one or more of a palette, wavelength range, signal strength range, distance or depth range, area, volume, shape, polarization, and depth measures output from a learning or statistical model corresponding to the object. The contour of a given object may also include a three-dimensional model of the object, such as a three-dimensional model generated from the patient scan described above.

[0135] The navigation controller 22 can therefore be configured to identify an object based on the identified portion of the actual depth map that fails to match the estimated depth map, by matching at least a portion of the identified portion with one of the predefined contours (i.e., the predefined contour corresponding to the object). The navigation controller 22 can then be configured to mark at least a portion of the identified portion of the actual depth map as a specific object corresponding to the contour, which can then be considered adjacent to the locator object.

[0136] In an alternative instance, a user may interact with user interface 24 to manually select objects in identified segments divided by navigation controller 22, and / or select predefined contours for the selected objects. The user may also interact with user interface 24 to manually track objects, such as those represented by actual depth maps within the identified segments, and / or select predefined contours for the tracked objects. Navigation controller 22 may then be configured to mark the selected segmented or tracked objects using markers corresponding to the selected predefined contours, and to track the selected or tracked objects accordingly.

[0137] In box 122, the position of each object identified based on the actual depth map can be determined in a common coordinate system, such as the visual coordinate system (VIS) or the locator coordinate system (LCLZ), having the objects being located. For example, navigation controller 22 can be configured to determine, for example, via vision engine 80, the position of each object identified based on the depth map and the position of each located object relative to the target volume in the common coordinate system, the position being defined by the object being located, such that navigation controller 22 can determine whether either the identified object and / or the located object constitutes an obstacle to the treatment target volume.

[0138] The navigation controller 22 can be configured to determine the position of each identified object relative to the identified object based on the positioning of the identified object detected by the locator 18 in the locator coordinate system LCLZ, the positioning of the identified object in the actual depth map, and the positional relationship between the locator 18 and the vision device 40 in a common coordinate system, which can be defined by the transformation data 83 stored in the navigation controller 22. As previously described, the position of the identified object in the actual depth map can indicate the position of the identified object in the visual coordinate system VIS. For example, the formation of the actual depth map: each image component of the identified object can represent a vector from the central viewpoint of the vision device 40 to the position in the visual coordinate system VIS. The position of each image component in the image frame of the actual depth map can indicate the horizontal and vertical components of the vector, and the depth indicated by each image component can represent the depth component of the vector.

[0139] The navigation controller 22 can therefore be configured to determine the position of each identified object in the vision component system (VIS) based on the position of the object in the actual depth map, and can then be configured to determine the position of each identified object relative to the locator object in the common coordinate system using the positional relationship between the locator 18 and the vision device 40, the position of each identified object in the visual coordinate system (VIS), and / or the position of each locator object in the locator coordinate system (LCLZ).

[0140] In box 124, for each tracked object (including objects identified based on the actual depth map and objects located using locator 18), a virtual boundary corresponding to the object in the common coordinate system can be generated, for example, based on the object's determined position in the common coordinate system. Specifically, navigation controller 22 can be configured to generate the virtual boundary in the common coordinate system, for example, via vision engine 80, to provide constraints on the movement of surgical instruments such as surgical instrument 16. For this purpose, navigation controller 22 can also be configured to track the movement of surgical instrument 16 in the common coordinate system, for example, using locator 18. The virtual boundary generated by navigation controller 22 can define areas in the common coordinate system that surgical instrument 16 should not enter or approach, as said space may be occupied by other objects, including sensitive anatomical structures and other surgical tools.

[0141] For example, navigation controller 22 may be configured to insert a stored 3D virtual model for each located object into a common coordinate system based on the determined position of the located object in the common coordinate system. When model data 82 stored in navigation controller 22 defines a 3D virtual model for a given object identified based on the identified portion of the actual depth map, navigation controller 22 may be configured to insert the 3D virtual model into the common coordinate system based on the determined position of the given identified object in the common coordinate system. Alternatively or additionally, model data 82 may indicate one or more original geometries (e.g., sphere, cylinder, cuboid) of a given object identified based on the identified portion of the actual depth map. In this case, navigation controller 22 may be configured to size and / or arrange the indicated original geometry based on the surface topography of the object indicated by the actual depth map, and is configured to insert the size-sized and / or arranged original geometry into the common coordinate system based on the determined position of the given object in the common coordinate system. Alternatively, when no virtual model or original geometry is indicated for a given object identified according to the identified portion of the actual depth map, the navigation controller 22 may be configured to construct a mesh boundary based on the surface topography indicated by the object in the actual depth map, and to insert the mesh boundary into the common coordinate system according to the determined position of the given object in the common coordinate system.

[0142] As another example, in addition to or as an alternative to one or more of the techniques described above, navigation controller 22 may be configured to approximate the boundary of a given object in a common coordinate system by inserting force particles into the common coordinate system based on the determined position of the given object in the common coordinate system. Specifically, navigation controller 22 may be configured to select individual points on the surface of the identified object and to place force particles at the determined positions of the individual points in the common coordinate system. Each of the force particles may be configured to repel other objects moving close to the force particle in the common coordinate system, such as entering within a predetermined distance. Thus, during the tracked movement of surgical instrument 16 in the common coordinate system, the force particles may repel surgical instrument 16, thereby preventing surgical instrument 16 from colliding with the object represented by the force particles. Inserting force particles corresponding to individual points on the surface of the identified object, rather than representing the virtual boundary of the entire surface of the identified object, into the common coordinate system results in generating the virtual boundary of the object using relatively reduced processing bandwidth and less data.

[0143] As an example, Figures 11 to 13 Showing the coordinates of the target part 200 in the common coordinate system Figure 5 The virtual boundary corresponding to the object in the () can be based on Figure 10 The differences shown are based on Figure 9 The actual depth map failed to match Figure 7The identified portion was determined by matching the expected depth map. Specifically, Figure 11 A virtual model of the retractor, corresponding to the retractor 208 in the target location 200 and depicted in the actual depth map, is shown in a common coordinate system. Figure 12 A virtual model of the ligament, corresponding to the ligament 204 in the target region 200 and depicted in the actual depth map, is shown in a common coordinate system. Figure 13 A virtual model of the epidermal tissue, corresponding to the epidermal tissue 206 in the target region 200 and depicted in the actual depth map, is shown in a common coordinate system. As an alternative example, the virtual boundary of the ligament 204 in the target region 200 can be in the form of an original geometric object, such as a cylinder, inserted into the common coordinate system based on the determined position of the ligament 204 in the common coordinate system, and the virtual boundary of the epidermal tissue 206 in the target region 200 can be in the form of a mesh surface or force particle inserted into the common coordinate system based on the determined position of the epidermal tissue 206 in the common coordinate system.

[0144] Figure 14 The diagram illustrates the relative positions of the femur F of the object identified based on the actual depth map and the patient located using locator 18 in a common coordinate system. Specifically, the illustration includes... Figures 11 to 13 The virtual model, and the virtual model 210 of the patient's femur F in Figure 6 As shown in the diagram. During the surgical procedure, the navigation controller 22 can be configured to, for example, display via the surgical navigator 81. Figure 14 Illustrations and images or virtual models of surgical instruments 16, such as those tracked by locator 18, at their current position in a common coordinate system, are provided to help surgeons guide surgical instruments 16 to the target volume 202.

[0145] In box 126, the presence of potential obstacles in the target site can be determined based on the tracked object and / or surgical plan 84. Specifically, navigation controller 22 can be configured, for example via surgical navigator 81, to determine whether one of the tracked objects (such as an object identified according to the actual depth map) is an obstacle to surgical plan 84 based on the object's position relative to the target volume in a common coordinate system and surgical plan 84. For example, surgical plan 84 may define a planned trajectory of surgical instrument 16 through the common coordinate system to treat the target volume. If the planned trajectory results in a collision with one of the virtual boundaries of the tracked object, navigation controller 22 can be configured to determine the presence of an obstacle.

[0146] In response to the determination of an obstacle's presence (the "Yes" branch of box 126), a remedial action can be triggered in box 128. The navigation controller 22 can be configured to trigger the remedial action, such as by performing one or more of a number of available actions via the surgical navigator 81. As an example, in response to the determination that the object is an obstacle to surgical plan 84, the navigation controller 22 can be configured to modify surgical plan 84 to avoid the obstacle. For example, the navigation controller 22 can be configured to change the trajectory of the surgical instrument 16 to avoid the obstacle and is configured to transmit the modified surgical plan 84 to the manipulator controller 50 for implementation. As another example, the navigation controller 22 can be configured to halt surgical guidance and movement of the robot manipulator 14 provided by the surgical navigation system 12 until an obstacle, as detected by the navigation controller 22, is cleared. The navigation controller 22 can also be configured to trigger an obstacle alarm and / or notification via the user interface 24 of the surgical navigation system 12. As another example, when the object causing the obstacle is identified as soft tissue, the navigation controller 22 can be configured to provide soft tissue guidance via the user interface 24. For example, navigation controller 22 may be configured to show the position of the soft tissue object causing the obstacle relative to other objects in the target area and provide suggestions for moving the soft tissue to clear the obstacle. Navigation controller 22 may be configured to continue monitoring the position of the soft tissue in a common coordinate system while providing soft tissue guidance and to notify the user when the obstacle threat is cleared.

[0147] Following the triggering and / or overcoming of a remedial action (box 128), or in response to an unmarked obstacle (the "No" branch of box 126), in box 130, the vision device 40 can be used to track the movement of objects identified based on actual depth. Specifically, the navigation controller 22 can be configured to track the movement of each identified object, such as via the vision engine 80, by being configured to: monitor the state of the portion of the actual depth map corresponding to the identified object in a subsequent actual depth map generated by the vision device 40. By focusing on changes in the portion of the previously determined actual depth map corresponding to the identified object in the subsequently generated depth map, rather than generating a desired depth map for each subsequently generated actual depth map, calculating the difference between the desired depth map and the subsequent actual depth map, and matching stored contours with the difference, the navigation controller 22 may be able to monitor the movement of the identified object at an increasing speed.

[0148] More specifically, each portion of the actual depth map corresponding to the identified object can depict an arrangement of features specific to the object and located at a specific position on the actual depth map. For example, but not limited to, the arrangement of features can be an arrangement of vertices with object-specific geometric relationships, an arrangement of edges or lines with object-specific geometric relationships, or an arrangement of depths with object-specific relative and geometric relationships. Furthermore, the spatial relationship between the arrangement of the object's features and the rest of the object can be fixed.

[0149] The navigation controller 22 can therefore be configured to monitor the movement of an object identified according to the actual depth map by monitoring whether the arrangement of object-specific features in the actual depth map has moved to a position in another depth map that differs from the position of the arrangement in the actual depth map. If so, the navigation controller 22 can be configured to determine the new position of the object in a common coordinate system based on the new position of the arrangement of features corresponding to the object in the other depth map, and is configured to update the virtual boundary associated with the object in the common coordinate system accordingly. The arrangement of features used to monitor the movement of a given object can be indicated in the object's model data 82, or can be manually set by the user by selecting points in the portion of the actual depth map corresponding to the object using the user interface 24.

[0150] For example, Figure 15 Shown in the generation Figure 9 The actual depth map shown is followed by another actual depth map generated by the vision device 40. Figure 15 The other depth map representation puller 208 ( Figure 5 The position of the part and the part in Figure 9 The position of the retractor 208 differs in the depth map, thus indicating that the retractor 208 has moved. The navigation controller 22 can be configured to track this movement of the retractor 208 by monitoring changes in the position of specific feature arrangements in another depth map relative to a previous actual depth map. These features are specifically arranged in the portion representing the retractor 208 and are fixed in position relative to the rest of the retractor 208. For example, the navigation controller 22 can monitor another depth map to understand changes in the position of the arrangement of the apex 222 between the head and body of the retractor 208.

[0151] In response to changes in the position of the determined vertex 222, the navigation controller 22 can be configured to adjust the position of the vertex 222 based on its arrangement. Figure 15 The updated position of the puller 208 in the common coordinate system is determined by the updated position in another depth map and the fixed positional relationship between the arrangement of vertex 222 and the rest of the puller 208. The navigation controller 22 can then be configured to adjust the virtual boundary associated with the puller 208 in the common coordinate system based on the updated position. Figure 16 Showing according to Figure 15 The updated position of the virtual boundary of the puller 208 (i.e., the virtual model corresponding to the puller 208) in the common coordinate system of the new position of the arrangement of vertex 222 depicted in another depth map.

[0152] This paper discloses a system and method for tracking objects in a surgical workspace using a combination of machine vision and tracker-based localization. Due to the flexible nature of soft tissues such as muscles, skin, and ligaments, tracker-based localization is often insufficient for tracking soft tissues. Therefore, in addition to using tracker-based localization to detect the position of rigid objects in the surgical workspace, the surgical navigation system may also include a vision device configured to generate depth maps of exposed surfaces in the surgical workspace. The surgical navigation system may be further configured to generate a predicted depth map of the vision device based on the position of an object in a target site detected using localization, a virtual model corresponding to the object, and the positional relationship between the locator and the vision device in a common coordinate system. The surgical navigation system may then be configured to identify portions of the actual depth map that do not match the estimated depth map, and to identify objects, including soft tissue, in the target site based on the identified portions. The surgical navigation system may then be configured to determine whether the object constitutes an obstacle to the current surgical plan.

[0153] Generally, routines executed to implement embodiments of the present invention, whether as part of an operating system or as a particular application, component, program, object, module, or sequence of instructions, or even a subset thereof, are referred to herein as "computer program code" or simply "program code." Program code typically comprises computer-readable instructions residing in various memories and storage devices in a computer at different times, and when read and executed by one or more processors in the computer, these computer-readable instructions cause the computer to perform operations necessary to implement the operations and / or elements embodying various aspects of embodiments of the present invention. Computer-readable program instructions for performing operations of embodiments of the present invention can be, for example, source code or object code written in assembly language or any combination of one or more programming languages.

[0154] The various program codes described herein can be identified based on the applications implemented therein in specific embodiments of the invention. However, it should be understood that any particular program nomenclature used below is merely for convenience, and therefore the invention should not be limited to use only in any particular application identified and / or implied by such nomenclature. Furthermore, considering the virtually limitless ways in which computer programs can be organized into routines, procedures, methods, modules, objects, etc., and the virtually endless ways in which program functionality can be distributed among various software layers residing within a typical computer (e.g., operating systems, libraries, APIs, applications, applets, etc.), it should be understood that embodiments of the invention are not limited to the specific organization and distribution of program functionality described herein.

[0155] The program code embodied in any application / module described herein can be distributed individually or collectively as a program product in a variety of different forms. In particular, the program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of embodiments of the invention.

[0156] Inherently non-transitory computer-readable storage media can include volatile and non-volatile media, as well as removable and non-removable media, implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media may also include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technologies, portable optical disc read-only memory (CD-ROM) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is readable by a computer. Computer-readable storage media should not be construed as transient signals themselves (e.g., radio waves or other propagating electromagnetic waves, electromagnetic waves propagating through a transmission medium such as a waveguide, or electrical signals transmitted through wires). Computer-readable program instructions can be downloaded from the computer-readable storage media to a computer, another type of programmable data processing device, or another apparatus, or downloaded via a network to an external computer or external storage device.

[0157] Computer-readable program instructions stored in a computer-readable medium can be used to direct a computer, other type of programmable data processing apparatus, or other means to operate in a particular manner, causing the instructions stored in the computer-readable medium to produce an article of writing that includes instructions that implement the functions, actions, and / or operations specified in flowcharts, sequence diagrams, and / or block diagrams. The computer program instructions can be provided to one or more processors of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the one or more processors, cause a series of calculations to be performed to implement the functions, actions, and / or operations specified in the flowcharts, sequence diagrams, and / or block diagrams.

[0158] In some alternative embodiments, the functions, actions, and / or operations specified in the flowcharts, sequence diagrams, and / or block diagrams may be reordered, processed sequentially, and / or concurrently, consistent with embodiments of the present invention. Furthermore, any of the flowcharts, sequence diagrams, and / or block diagrams may include more or fewer blocks than those illustrated consistent with embodiments of the present invention.

[0159] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit embodiments of the invention. As used herein, the singular forms “a,” “an,” and “described” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Furthermore, if the terms “comprising,” “having,” “has,” “possess,” “consist of,” or variations thereof are used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “comprising.”

[0160] While the invention has been illustrated by description of various embodiments, and while these embodiments have been described in considerable detail, the applicant does not intend to limit the scope of the appended claims or restrict them in any way to such details. Other advantages and modifications will be readily apparent to those skilled in the art. Therefore, the invention, in its broader aspects, is not limited to the specific details shown and described, representative devices and methods, and illustrative examples. Consequently, deviations from such details may be made without departing from the spirit or scope of the applicant's overall inventive concept.

Claims

1. A navigation system comprising: A locator configured to detect the position of a first object; A vision device configured to generate an actual depth map of the surface near the first object; as well as A controller, coupled to the locator and the vision device, is configured to: Access the virtual model corresponding to the first object; The positional relationship between the locator and the vision device in a common coordinate system is identified; The expected depth map of the vision device is generated based on the detected position of the first object, the virtual model, and the positional relationship. Identify the portions of the actual depth map that do not match the expected depth map; and The second object is identified based on the identified portion.

2. The navigation system of claim 1, wherein the controller is configured to identify the position of the second object relative to the first object in the common coordinate system based on the detected position of the first object, the positioning of the second object in the actual depth map, and the positional relationship.

3. The navigation system of claim 2, wherein the first object defines a target volume of patient tissue to be treated according to the surgical plan, and the controller is configured to: Based on the position of the second object relative to the target volume in the common coordinate system and the surgical plan, it is determined whether the second object is an obstacle that will be treated according to the surgical plan for the target volume; and In response to determining that the second object is an obstacle of the target volume to be treated according to the surgical plan, one or more of the following are performed: modifying the surgical plan, triggering a notification, and stopping surgical navigation.

4. The navigation system of claim 1, wherein the tracker is rigidly coupled to the first object, and the controller is configured to: The position of the tracker in a first coordinate system specific to the locator is detected via the locator; The position of the virtual model in the first coordinate system is identified based on the position detected by the tracker in the first coordinate system and the positional relationship between the tracker and the first object in the first coordinate system; Based on the position of the virtual model in the first coordinate system and the positional relationship between the locator and the vision device in the second coordinate system, the position of the virtual model in the first coordinate system is converted into the position of the virtual model in the second coordinate system specific to the vision device; and The expected depth map is generated based on the position of the virtual model in the second coordinate system.

5. The navigation system of any one of claims 1-4, wherein the controller is configured to identify portions of the actual depth map that fail to match the expected depth map by being configured to: Calculate the difference between the actual depth map and the expected depth map; Determine whether the first segment of the difference indicates an absolute depth greater than a threshold depth; and In response to the first segment indicating an absolute depth greater than the threshold depth, the second segment of the actual depth map corresponding to the first segment indicating the difference is identified as the portion.

6. The navigation system of claim 5, wherein the threshold depth is non-zero.

7. The navigation system of claim 5, wherein the controller is configured to identify portions of the actual depth map that fail to match the expected depth map by being configured to: Determine whether the size of the first segment is greater than the minimum size threshold; and In response to determining that the size of the first segment is greater than the minimum size threshold, the second segment is identified as the portion.

8. The navigation system of any one of claims 1-4, wherein the controller is configured to identify the second object based on the identified portion by being configured to match the identified portion with a predetermined contour corresponding to the second object.

9. The navigation system of any one of claims 1-4, wherein the portion of the actual depth map includes an arrangement of features corresponding to the second object and located in a first position in the actual depth map, and the controller is configured to track the movement of the second object by being configured to monitor whether the arrangement of the features moves to a second position, the second position being different from the first position in a subsequent actual depth map generated by the vision device.

10. The navigation system of any one of claims 1-4, wherein the controller is configured to generate a virtual boundary corresponding to the second object in the common coordinate system, the virtual boundary being configured to provide constraints on the movement of the surgical tool.

11. The navigation system of any one of claims 1-4, wherein the controller is configured to Based on the virtual model, the detected position of the first object, and the positional relationship between the locator and the vision device in the common coordinate system, the actual depth map is cropped into a region of interest; and The portion of the actual depth map that fails to match the expected depth map is identified by comparing the cropped actual depth map with the expected depth map.

12. The navigation system of any one of claims 1-4, wherein the controller is configured to identify the positional relationship between the locator and the vision device in the common coordinate system by being configured to: A light pattern is projected onto a surface within the field of view of the locator and the vision device; The locator is used to generate positioning data, which indicates the position of the projected light pattern in a first coordinate system specific to the locator. Receive a calibrated depth map generated by the vision device, showing the projected light pattern; The position of the projected light pattern in a second coordinate system specific to the vision device is identified based on the calibration depth map; and The positional relationship between the locator and the vision device in the common coordinate system is identified based on the position of the projected light pattern in the first coordinate system and the position of the projected light pattern in the second coordinate system.

13. The navigation system of any one of claims 1-4, wherein the locator is configured to operate in a first spectral band to detect the position of the first object, the vision device is configured to operate in a second spectral band to generate the actual depth map of the surface near the first object, and the first spectral band is different from the second spectral band.

14. A robotic manipulator used in conjunction with a navigation system as claimed in any one of claims 1-13, wherein the robotic manipulator supports a surgical tool and includes a plurality of links and a plurality of actuators configured to move the links to move the surgical tool, and wherein the robotic manipulator is controlled to avoid the second object.

15. A method of operating a navigation system, the navigation system comprising: A locator configured to detect the position of a first object; A vision device configured to generate an actual depth map of the surface near the first object; The method includes a controller coupled to the locator and the vision device. Access the virtual model corresponding to the first object; The positional relationship between the locator and the vision device in a common coordinate system is identified; The expected depth map of the vision device is generated based on the detected position of the first object, the virtual model, and the positional relationship. Identify the portions of the actual depth map that do not match the expected depth map; as well as The second object is identified based on the identified portion.

16. The method of claim 15, further comprising a controller: identifying the position of the second object relative to the first object in the common coordinate system based on the detected position of the first object, the positioning of the second object in the actual depth map, and the positional relationship.

17. The method of claim 16, wherein the first object defines a target volume of patient tissue to be treated according to a surgical plan, and the method further includes a controller: Based on the position of the second object relative to the target volume in the common coordinate system and the surgical plan, it is determined that the second object is an obstacle to be treated according to the surgical plan for the target volume; and In response to determining that the second object is an obstacle of the target volume to be treated according to the surgical plan, one or more of the following are performed: modifying the surgical plan, triggering a notification, and stopping surgical navigation.

18. The method of claim 15, wherein the tracker is rigidly coupled to the first object, and the method further includes a controller: The position of the tracker in a first coordinate system specific to the locator is detected via the locator; The position of the virtual model in the first coordinate system is identified based on the position detected by the tracker in the first coordinate system and the positional relationship between the tracker and the first object in the first coordinate system; Based on the position of the virtual model in the first coordinate system and the positional relationship between the locator and the vision device in the second coordinate system, the position of the virtual model in the first coordinate system is converted into the position of the virtual model in the second coordinate system specific to the vision device; as well as The expected depth map is generated based on the position of the virtual model in the second coordinate system.

19. The method of any one of claims 15-18, wherein identifying portions of the actual depth map that fail to match the expected depth map includes a controller: Calculate the difference between the actual depth map and the expected depth map; The first segment determining the difference indicates an absolute depth greater than a threshold depth; and In response to the first segment indicating an absolute depth greater than a threshold depth, the second segment of the actual depth map corresponding to the first segment indicating the difference is identified as the portion.

20. The method of claim 19, wherein the threshold depth is non-zero.

21. The method of claim 19, wherein identifying portions of the actual depth map that fail to match the expected depth map further includes a controller: It is determined that the size of the first segment is greater than the minimum size threshold; and In response to determining that the size of the first segment is greater than the minimum size threshold, the second segment is identified as the portion.

22. The method of any one of claims 15-18, wherein identifying the second object based on the identified portion includes a controller: matching the identified portion with a predetermined contour corresponding to the second object.

23. The method of any one of claims 15-18, wherein the portion of the actual depth map includes an arrangement of features corresponding to the second object and located at a first position in the actual depth map, and the method further includes a controller tracking movement of the second object by monitoring whether the arrangement of the features moves to a second position, the second position being different from the first position in a subsequent additional actual depth map generated by the vision device.

24. The method of any one of claims 15-18, further comprising a controller: generating a virtual boundary in the common coordinate system corresponding to the second object, the virtual boundary being configured to provide constraints on the movement of the surgical tool.

25. The method of any one of claims 15-18, further comprising a controller: Based on the virtual model, the detected position of the first object, and the positional relationship between the locator and the vision device in a common coordinate system, the actual depth map is cropped into a region of interest; and The portion of the actual depth map that fails to match the expected depth map is identified by comparing the cropped actual depth map with the expected depth map.

26. The method of any one of claims 15-18, wherein identifying the positional relationship between the locator and the vision device in the common coordinate system includes a controller: A light pattern is projected onto a surface within the field of view of the locator and the vision device; The locator is used to generate positioning data, which indicates the position of the projected light pattern in a first coordinate system specific to the locator. Receive a calibrated depth map generated by the vision device that corresponds to the projected light pattern; The position of the projected light pattern in a second coordinate system specific to the vision device is identified based on the calibration depth map; as well as The positional relationship between the locator and the vision device in the common coordinate system is identified based on the position of the projected light pattern in the first coordinate system and the position of the projected light pattern in the second coordinate system.

27. The method of any one of claims 15-18, further comprising a controller: Operate the locator in the first spectral band to detect the position of the first object; and The vision device is operated in a second spectral band to generate the actual depth map of the surface near the first object, the second spectral band being different from the first spectral band.

28. A computer program product comprising a non-transitory computer-readable medium having instructions stored thereon, the instructions being configured, when executed by one or more processors, to perform the method as described in any one of claims 15-27.

29. A surgical system comprising: Surgical instruments; The robotic manipulator, comprising multiple links and joints, is configured to support and move surgical instruments; The manipulator controller is configured to control the robot manipulator; and Navigation systems, including: The locator is configured to detect the position of the first object; The vision device is configured to generate a map of the actual depth of the surface near the first object; and The navigation controller, coupled to the manipulator controller, positioner, and vision device, is configured as follows: Access the virtual model corresponding to the first object; Identify the positional relationship between the locator and the vision device in a common coordinate system; Generate the expected depth map of the vision device based on the detected position, virtual model, and positional relationships of the first object; Identify portions of the actual depth map that do not match the expected depth map; The second object is identified based on the recognition; and The information related to the second object is conveyed to the manipulator controller. The manipulator controller is configured, based on the information conveyed, to control the robot manipulator to avoid the second object.

Images