Obstacle avoidance techniques for surgical navigation

Abstract

To provide a system and method in which a localizer is configured to detect a position of a first object and a vision device is configured to generate a depth map of surfaces near the first object.SOLUTION: In a navigation system, a virtual model corresponding to a first object is accessed to identify a positional relationship in a common coordinate system between a localizer and a vision device. Next, an expected depth map of the vision is generated based on the detected position of the first object, the virtual model, and the positional relationship. A portion of an actual depth map that fails to match the expected depth map is recognized, and a second object is recognized based on the identified portion.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Provisional Patent Application No. 62 / 870,284, filed July 3, 2019. This provisional application claims priority to and the full benefit of this provisional application, 10, No. 10 / 109,493, filed Dec. 1, 2004, incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to surgical navigation systems. [Background technology]

[0003] The surgical navigation system locates surgical instruments to a target volume of a patient's tissue for treatment. During a surgical procedure, the target volume to be treated is located in an area that is difficult to avoid. These are often located adjacent to sensitive anatomical structures and surgical tools that must be removed. Tracking of adjacent anatomical structures using attached trackers is difficult due to the flexibility of the structures. Furthermore, it is often difficult to attach a tracker to each object adjacent to the target volume. This leads to a crowded surgical workspace and increases the cost and complexity of surgical navigation systems. do. Summary of the Invention

[0004] In a first aspect, a localizer configured to detect a first object; a vision device configured to generate a real depth map of a surface in the vicinity of an object; and a controller coupled to the localizer and the vision device, the controller accessing a virtual model corresponding to a first object and localizing the virtual model in a common coordinate system; determining a positional relationship between the riser and the vision device; and determining a detected position of a first object; generating a predicted depth map for the visual device based on the virtual model and the positional relationship; , identifying portions of the actual depth map that do not match the expected depth map, and and recognizing a second object based on the detected object. A navigation system is provided comprising:

[0005] In a second aspect, a robot manipulator is combined with the navigation system of the first aspect. The robot manipulator supports a surgical tool and has multiple links and a and a plurality of actuators configured to move the surgical tool. The robot is controlled to avoid the second object.

[0006] In a third aspect, a method of operating a navigation system is provided, comprising: The system includes a localizer configured to detect a position of the first object and a a vision device configured to generate a real depth map of a surface in the vicinity of the object; a controller coupled to the imager and the visual device, the method comprising: Access to a virtual model corresponding to the object and coordinate the localizer and the visual determining a positional relationship between the device and the detected position of the first object, a virtual model, generating an expected depth map for the visual device based on the positional relationship; identifying portions of the real depth map that do not match the map, and and recognizing the object.

[0007] In a fourth aspect, the method of the third aspect when executed by one or more processors a computer including a non-transitory computer-readable medium having stored thereon instructions configured to perform A computer program product is provided.

[0008] According to one embodiment of any of the above aspects, the localizer an optical localizer configured to detect an optical signature associated with the first object; an electromagnetic localizer configured to detect electromagnetic signatures associated with any an ultrasonic probe configured to detect a first object with or without a detector; an inertial load configured to detect an inertial feature associated with a first object; a colorizer, or any combination of the above.

[0009] According to one embodiment of any of the above aspects, the first object comprises a patient's Anatomical structures or bones, robotic manipulators, hand-held instruments, attachments to robotic manipulators attached end effectors or tools, surgical table, mobile cart, patient loading surgical equipment, such as an operating table, imaging system, retractor, or any combination of the foregoing. The device may be any of the devices in the room, but is not limited to these.

[0010] According to one embodiment of any of the above aspects, the vision device comprises: a localizer; A separate unit from the localizer, a navigation system camera unit, adjustable Arms, robotic manipulators, end effectors, hand tools, ceiling mounted a surgical boom system, such as a surgical boom, a limb support device, or any combination of the foregoing. It is combined with either

[0011] According to one embodiment of any of the above aspects, a surface near the first object are surfaces adjacent to the first object, surfaces at a distance from the first object, surface, a surface that touches the first object, a surface that is directly on top of the first object, a surface that is directly on the first object, Surfaces in the environment near the object, behind the first object or in the environment surrounding it surface, within a threshold distance of the first object, within the field of view of the localizer, or any combination of the above. It could be that.

[0012] According to one embodiment of any of the above aspects, the second object is a second portion of the patient's anatomy, such as tissue; a robotic manipulator; one or more arms of a robotic manipulator, a second robotic manipulator, a handheld instrument, a robotic manipulator End effectors or tools attached to manipulators or handheld instruments, surgical tapes a mobile cart, an operating table on which the patient can be placed, an imaging system, a retractor, a tracking device (but not limited to) equipment in the operating room, such as the vise itself, and people in the operating room. or any combination of the foregoing. It can be a project.

[0013] According to one embodiment of any of the above aspects, the controller includes one or more controllers. According to one embodiment, the controller may be a common coordinate system. configured to determine a position of a second object relative to a first object in According to one embodiment, the controller is configured to: Such identification is based on the location and positional relationship of the second object in the depth map. Determine.

[0014] According to one embodiment, the first object is a representation of the patient's tissue to be treated according to the surgical plan. According to one embodiment, the controller defines a target volume in a common coordinate system. Based on the position of the second object relative to the intersection and the surgical plan, the second object is to determine whether the target volume is an obstacle to treating it according to the surgical plan. According to one embodiment, the second object is an obstacle to the surgical plan. In response to determining that there is a problem, the controller may modify the surgical plan; and / or configured to trigger notifications and / or stop surgical navigation will be done.

[0015] According to one embodiment, the tracker is coupled to the first object. If so, the controller, via the localizer, calculates the coordinates in a first coordinate system specific to the localizer. According to one embodiment, the controller is configured to detect the position of the tracker. The detected position of the tracker in the first coordinate system and the position of the tracker and the first coordinate system determining a position of the virtual model in the first coordinate system based on a positional relationship with the object; According to one embodiment, the controller may calculate the position of the virtual model in the first coordinate system. and a positional relationship between the localizer and the visual device in the second coordinate system. The position of the virtual model in the coordinate system is converted into a virtual coordinate system in a second coordinate system specific to the viewing device. According to one embodiment, the controller converts the virtual coordinate system in the second coordinate system into the position of the model. Based on the position of the model, a predicted depth map can be generated.

[0016] According to one embodiment, the controller is configured to compare the actual depth map with the expected depth map. configured to identify portions of the actual depth map that do not match the expected depth map. In some embodiments, the controller is configured to: In one embodiment, the controller calculates a difference between the first section of the difference and the second section of the difference. Determine whether the object exhibits an absolute depth greater than a threshold depth. The controller determines that the first section of the difference indicates an absolute depth greater than the threshold depth. In response to the first section of the difference, a second section of the real depth map corresponding to the first section of the difference is generated. According to one embodiment, the threshold depth is non-zero.

[0017] According to one embodiment, the controller detects portions of the actual depth map that do not match the expected depth map. In some embodiments, the controller is configured to identify the first section. configured to determine whether the size of the session is greater than a minimum size threshold. In some embodiments, the controller does this by: In response to determining that the size is greater than the minimum size threshold, the second section is partitioned. Specify as minutes.

[0018] According to one embodiment, the controller maps the identified portion to a second object. The identified portion is then matched to a predetermined profile. The second object is configured to recognize the second object.

[0019] According to one embodiment, a portion of the real depth map corresponds to a second object, and According to one embodiment, the controller includes an array of features located at a first location of the chip. , by monitoring whether the sequence of features moves to a second position different from the first position. According to one embodiment, the computer is configured to track the movement of the second object. The controller then calculates the real depth map in the additional real depth map that is subsequently generated by the vision device. Monitor things like:

[0020] According to one embodiment, the controller generates a virtual coordinate system corresponding to the second object in the common coordinate system. According to one embodiment, the virtual boundary provides constraints. In some embodiments, the constraints may include a surgical tool, a robotic manipulator, a robotic hand, or a robotic hand. Objects such as the working end of surgical equipment, imaging devices, or any other moving equipment in the operating room In some embodiments, constraints are keep-out boundaries. Or a keep-in boundary.

[0021] According to one embodiment, the controller comprises: a virtual model; a detected position of the first object; and a real depth map based on the positional relationship between the localizer and the visual device in the common coordinate system. In some embodiments, the controller is configured to trim the image to a region of interest. is configured to compare the cropped real depth map to the real depth map. The method is configured to compare the groups.

[0022] According to one embodiment, the controller is within the field of view of the vision device and is optionally routed. configured to project the pattern onto a surface that is also within the field of view of the colorizer, configured to determine the positional relationship between the localizer and the visual device in the universal coordinate system; In some embodiments, the controller calculates the coordinates in a first coordinate system specific to the localizer. The location data is generated using a localizer that indicates the location of the pattern in the In an embodiment, the controller may generate a comparison image indicative of the projection pattern generated by the vision device. In some embodiments, the controller receives a calibration depth map. and determining a position of the projected pattern in a second coordinate system specific to the vision device based on the In some embodiments, the controller determines the position of the pattern in the first coordinate system and the position of the pattern in the second coordinate system. Based on the position of the pattern in the two coordinate systems, the localizer and the observer in the common coordinate system are In some embodiments, the controller determines a positional relationship with the sensory device. configured to operate in a spectral band to detect a location of a first object; The vision device operates in a second spectral band to visualize the real world of a surface near the first object. configured to generate a depth map, the first spectral band being is different.

[0023] Any of the above aspects can be combined in whole or in part. Any of these can be combined in whole or in part.

[0024] The above summary has been provided to provide a basic understanding of certain aspects of the invention discussed herein. This summary may provide a brief overview of some aspects of the invention in order to better understand the present invention. It is not intended to provide an extensive overview of the invention, nor does it include all of the essential elements. It is not intended to identify critical elements or to define the scope of the invention. The sole purpose of this summary is to provide some general information as a prelude to the detailed description that follows. The idea is to present the idea in a simplified form. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a perspective view of a surgical navigation system including a localizer and a vision device. [Figure 2] FIG. 2 is a schematic diagram of a control system for controlling the surgical navigation system of FIG. 1. [Figure 3] FIG. 2 is a perspective view of a coordinate system used in the surgical navigation system of FIG. 1. [Figure 4] 1 is a flowchart of a method for navigating an example target region using tracker-based localization and machine vision. [Figure 5] 1 is an illustration of an example of a target site, e.g., an anatomical structure to be treated during a surgical procedure. [Figure 6] FIG. 6 is an illustration of the location of a virtual model corresponding to an object within the target region of FIG. 5. [Figure 7] FIG. 7 is an explanatory diagram of a predicted depth map based on the virtual model of FIG. 6. [Figure 8] FIG. 2 is an illustration of a real depth map captured by the visual device of FIG. 1; [Figure 9] FIG. 9 is an illustration of the actual depth map of FIG. 8 cropped to a region of interest. [Figure 10] 10 is an explanatory diagram of the difference between the predicted depth map of FIG. 7 and the actual depth map of FIG. 9. FIG. [Figure 11] FIG. 10 is an illustration of a virtual model corresponding to the surgical retractor identified in the real depth map of FIG. 9. [Figure 12] FIG. 10 is an illustration of a virtual model corresponding to a ligament identified in the real depth map of FIG. 9. [Figure 13] FIG. 10 is an illustration of a virtual model corresponding to the epidermal tissue identified in the real depth map of FIG. 9. [Figure 14] FIG. 14 is an explanatory diagram of the virtual models of FIGS. 6 and 11 to 13 in a common coordinate system. [Figure 15]FIG. 2 is an illustration of a real depth map subsequently captured by the vision device of FIG. 1; [Figure 16] 16 is an illustration of the virtual model of FIG. 14 with updated positioning based on the real depth map of FIG. 15. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0026] FIG. 1 shows a surgical system 10 for treating a patient. The surgical system 10 is typically used in medical facilities. The surgical system 10 may be installed in a surgical environment such as a hospital operating room. The robotic manipulator 14 may include a system 12 and a robotic manipulator 14. The controller 14 may be coupled to a surgical instrument 16, which the surgeon uses to treat a target volume of patient tissue. and / or to operate the surgical instrument 16 under the direction of the surgical navigation system 12, etc. For example, the surgical navigation system 12 may be configured to The target volume is then scanned to detect other objects adjacent to the target volume, such as other medical devices and adjacent anatomical structures. The surgical instrument 16 can be manipulated to ablate a target volume of patient tissue while avoiding the Alternatively, the surgeon can perform the following operations while receiving guidance from the surgical navigation system 12: The surgical instrument 16 can be manually held and manipulated. The surgical instrument 16 may be a burring instrument, an electrosurgical instrument, an ultrasonic instrument, a reamer, an impactor, or the like. , or a sagittal saw.

[0027] During a surgical procedure, the surgical navigation system 12 uses tracker-based localization Using a combination of machine vision and 3D modeling, the position of the object of interest ( The surgical workspace for the surgical procedure can be tracked by a target volume of patient tissue and a target volume to be treated where obstacles to treatment may exist; The tracked object may be considered to include the patient's anatomy and the area immediately surrounding the patient's anatomy. the structure, the target volume of the anatomical structure to be treated, the surgical instruments, such as surgical instrument 16, and the surgeon's hands or These may include, but are not limited to, surgical body anatomies such as fingers. The anatomical structures and target volumes may include soft tissues such as ligaments, muscles, and skin, and may include bones, The surgical instruments that are tracked may include retractors, cutters, and the like used during surgical procedures. tools, and waste disposal devices.

[0028] By fixing the tracker to the object of interest in the surgical workspace, surgical navigation is possible. The surgical system 12 may be configured to determine the location of such objects within the surgical workspace. During the procedure, the tracker detects the presence of a specific non-visible light band. Known signal patterns may be generated in various wavelengths (e.g., infrared, ultraviolet), etc. The application system 12 is specialized to detect signals within a specific non-visible light band and does not detect signals outside this band. The localizer may include a localizer that ignores optical signals from a given tracker. In response to detecting a signal pattern associated with , determining the position of the tracker relative to the localizer based on the angle at which the pattern is detected. The surgical navigation system 12 then calculates the determined position of the tracker. The tracker is attached based on the position of the object and the fixed positional relationship between the object and the tracker. The position of the object can be determined.

[0029] The tracker allows the surgical navigation system 12 to track bones and surgical objects within the surgical workspace. This may allow for accurate and efficient tracking of hard tissue objects such as instruments. These trackers are generally not suitable for tracking soft tissue objects such as skin and ligaments. Specifically, due to the flexible nature of the soft tissue object, It is difficult to maintain a constant positional relationship between the entire object and the tracker. Attaching trackers to each of the several patient tissues and instruments involved creates a surgical workspace. The area is congested, making navigation difficult, and the cost and This increases the complexity of surgical navigation in addition to tracker-based localization. The navigation system 12 is used to track objects within the surgical workspace during a surgical procedure. ,Machine vision can also be implemented.

[0030] Specifically, the localizer and attached tracker are used to locate the object within the surgical workspace. In addition to detecting the location of the object, the surgical navigation system 12 also detects the location of the object in the working space. configured to generate a depth map of a surface within the target region (also referred to herein as the target area). The target site may be a variety of different objects or sites. In one embodiment, the target site may be an anatomical structure requiring treatment or tissue ablation. In other embodiments, the target site is a surgical site, such as a portion of a structure (e.g., a bone). Manipulator, end effector or tool attached to robot manipulator surgical table, mobile cart, patient-supported operating table, imaging system It may be an operating room device such as a stem, or the like.

[0031] The surgical navigation system 12 also includes a localizer and a vision device in a common coordinate system. The localizer may be configured to identify a positional relationship with a virtual object corresponding to the object. Using the model and the positional relationship, the visual The device may be configured to generate a predicted depth map of the surgical navigation device. The imaging system 12 compares the predicted depth map with the actual depth map generated by the vision device. and based on the comparison, identifying portions of the actual depth map that do not match the estimated depth map. The surgical navigation system 12 may then be configured to: Identify objects at the target site based on the current surgical plan and The system may be configured to determine whether the object is an obstacle based on the detected obstacle.

[0032] The surgical navigation system 12 is configured to track and visualize the surgical site during the surgical procedure to assist the surgeon. The relative positions of the objects may be displayed. The movement of the robot manipulator 14 and / or surgical instrument 16 relative to the tracked object is For example, surgical navigation systems can be used to control and / or constrain the virtual boundaries of the surgical site. The system 12 determines the target volume of patient tissue to be treated and the surgical procedure based on the tracked object. The surgical navigation system 1 may then identify potential obstacles within the surgical space. 2 is a diagram showing a surgical instrument (e.g., an end effector EA of the surgical instrument 16) that is inserted into the patient tissue to be treated. Limits contact with anything beyond the target volume, improving patient safety and surgical accuracy The surgical navigation system 12 can also be used to visualize the position of the object in relation to other objects. This can occur due to accidental contact, resulting in unwanted debris at the target site. This eliminates potential damage to surgical instruments.

[0033] As shown in FIG. 1, the surgical navigation system 12 includes a localizer 18 and a navigation system. The navigation cart assembly 20 may include a navigation cart assembly 20. , performing the functions, features, and processes of the surgical navigation system 12 described herein. The navigation controller 22 may be configured to In particular, navigation controller 22 may be a navigation controller as described herein. and the surgical navigation system 12. For example, the processor 23 may include a processor 23 programmed to The optically based signal received from the localizer 18 is then picked up by a tracker within the surgical workspace. It is programmed to convert the location of the object into localizer data that indicates the location of the attached object. This may also be done.

[0034] The navigation controller 22 is a user interface for the surgical navigation system 12. The user interface 24 may be in operative communication with the surgical navigation system. and facilitating user interaction with the application system 12 and the navigation controller 22. For example, the user interface 24 may be a navigation controller 22 or the like. It may also include one or more output devices that provide information to a user, such as The chair is fitted with a display adapted to be located outside the sterile field containing the surgical workspace. A display 26 adapted to be located inside the sterile field may be included. The displays 25, 26 are adjustable to the navigation cart assembly 20. The user interface 24 may also be mounted to facilitate surgical navigation. The computer system 12 may include one or more input devices that allow user input to the computer system 12. The input device is interacted with by the user to input surgical parameters and A keyboard, mouse, and Input devices may include a user interface via voice recognition technology and / or a touch screen 28. A microphone may also be included to allow user input.

[0035] The localizer 18 detects the position of a tracker attached to an object. This allows the position of one or more objects attached to trackers within the surgical workspace to be determined. Specifically, the localizer 18 may be configured to detect 12, and can be coupled to the navigation controller 22 of the system 12 to access one or more and generates an optically based signal indicating the position of the tracker, which is then transmitted to the navigation controller 2. 2. In that case, the navigation controller 22 may transmit the optical-based signal to , and the tracker in the surgical workspace based on a certain positional relationship between the object and the tracker. configured to generate localizer data indicative of the location of an object attached to the The objects within the target region that are tracked by the localizer 18 may be referred to herein as Sometimes referred to as a "local object."

[0036] The localizer 18 comprises an outer casing 3 which houses at least two optical sensors 32. Each of the optical sensors 32 may have a tracker-specific light source, such as infrared or ultraviolet light. It can be adapted to detect signals in specific non-visible light bands. Although the localizer 18 is shown as a single unit with the optical sensor 32, alternative In an embodiment, localizer 18 comprises separate units positioned around the surgical workspace. , each may have a separate outer casing and one or more optical sensors 32 .

[0037] The optical sensor 32 may be a one-dimensional or two-dimensional charge-coupled device (CCD). For example, the outer casing 30 may be used to triangulate the position of the tracker in the surgical workspace. The tracker may house two 2D CCDs or triangulate the position of the tracker in the surgical workspace. Additionally or alternatively, the localizer may house three one-dimensional CCDs for determining the 18 is compatible with other light-sensing devices such as complementary metal-oxide semiconductor (CMOS) active pixels. Guiding techniques may also be employed.

[0038] In some embodiments, the navigation system and / or localizer 18 may It is based on magnetic (EM) technology. For example, navigation systems connected to the controller 22 and / or another computing device, controller, etc. The tracker may include an EM transceiver attached to it. EM components (e.g., various magnetic trackers, electromagnetic trackers, inductive trackers, etc.) It can be passive or actively energized. The EM transceiver generates an EM field, and the EM component detects the state being tracked and navigates the state. responsive to the EM signal to be transmitted to (or read by) the location controller 22. The navigation controller 22 analyzes the received EM signals and determines the relevant conditions. Again, the EM-based navigation system An embodiment of the system is an active marker-based navigation system as described herein. It will be understood that the system may have different structural configurations than the system.

[0039] In other embodiments, the navigation system and / or localizer 18 may The tracker does not necessarily need to be fixed to the object to determine the associated location data. It can be based on one or more types of imaging systems, e.g. The tracking state (e.g., position, orientation, etc.) is based on the ultrasound image and is then displayed to the navigation controller. The ultrasound image (e.g., the image of the tracked object) is transmitted to (or read by) the sensor 22. specific known structural features of the object, markers or stickers fixed to the tracked object, etc. It is possible to provide an ultrasound-based imaging system that facilitates the acquisition of The ultrasound images may be 2D, 3D, or a combination thereof. The application controller 22 processes the ultrasound images in near real time to determine the tracked state. The ultrasound imaging device may have any suitable configuration. The camera unit may be different from that shown in FIG. The tracked state is communicated (or read) to the navigation controller 22 based on the X-ray image. radiopaque markers (e.g., attached to the tracked object) Fluorescence imaging, which facilitates the acquisition of X-ray images of objects (stickers, tags, etc.) with known structural features It is possible to provide a perspective-based imaging system. The roller 22 can process the x-ray images in near real time to determine the tracked state. Similarly, the tracked state is transmitted to the navigation controller 22 based on the digital image. Digital images, video, or other images of specific known objects, as they are obtained (or read) (e.g., a virtual representation of the tracked object or its structural components or features) based on a comparison with the current state) and / or markers (e.g., tracked objects) It facilitates the capture of digital images, videos, etc. of stickers, tags, etc. attached to objects. It is possible to provide other types of optically based imaging systems that facilitate The navigation controller 22 processes the digital images in near real time to A tracking state can be determined.

[0040] Therefore, various types of imaging systems, including multiple imaging systems of the same or different types, may be used. The imaging system may be integrated into a navigation system without departing from the scope of this disclosure. It will be understood that the navigation system and the and / or localizer 18 may have any other suitable configuration not specifically described herein. It will be understood that the navigation system may have any component or structure. , inertial tracking alone or any combination of tracking techniques can be utilized. , of techniques, methods, and / or components related to the navigation system shown in FIG. Both can be implemented in a variety of ways, and other configurations are contemplated by this disclosure.

[0041] The localizer 18 ideally views the surgical workspace and target volume free of obstructions. The optical sensor 32 may be mounted on an adjustable arm for selective positioning. The localizer 18 rotates about a rotary joint to move at least one It may be adjustable in one degree of freedom, or it may be adjustable in two or more degrees of freedom.

[0042] As noted above, the localizer 18 comprises a plurality of tracking devices, also referred to herein as trackers. In conjunction with the chair, the tracker determines the position of the object within the surgical workspace. Generally, the object to which each tracker is attached is rigid and flexible. Therefore, the positional relationship between the object and the tracker changes depending on the object's movement. In other words, the operating room Regardless of the position change of the object, the tracker in the surgical workspace and the tracker attached The relationship to the object being tracked can remain fixed. For example, a tracker can and surgical instruments such as retractors and surgical instruments 16. In this way, the localizer 18 can be used to track the surgical workspace. In response to determining the position of the tracker, navigation controller 22 Determines the position of the object to which the tracker is attached based on the specified position It is possible.

[0043] For example, if the target volume to be treated is located in the knee region of the patient, the tracker 34 is The tracker 36 can be securely attached to the femur F and the tibia T of the patient. and the tracker 38 can be securely attached to the surgical instrument 16. The trackers 34, 36 are described in U.S. Pat. No. 7,725, The tracker 34 may be attached to the femur F and tibia T as shown in US Pat. No. 6,162. , 36 is also incorporated herein by reference. ms and Methods for Indicating and Reducing ng Line-of-Sight Errors,” on January 16, 2014. Equipped with a device such as that shown in filed U.S. Patent Application Publication No. 2014 / 0200621 The tracker 38 may be integrated into the surgical instrument 16 at the time of manufacture or may be externally mounted. It may be separately attached to the surgical instrument 16 in preparation for the surgical procedure.

[0044] Prior to the start of a surgical procedure using the surgical system 10, the surgical instrument 16 may be used to treat the area to be treated. Anatomical structures defining a target volume of patient tissue and / or anatomical structures adjacent to that target volume. Preoperative images may be generated for the anatomy of interest, such as: If the target volume of patient tissue is the patient's knee, preoperative images of the patient's femur (F) and tibia (T) are taken. These images may be taken using MRI scans, radiological scans, or other methods of imaging the patient's anatomy. It may be based on a virtual model of the anatomy, or a computed tomography (CT) scan. Each virtual model of an anatomical structure may be used to construct a complete model of the anatomy. Data representing the body or at least a portion thereof and / or data representing a target volume of the anatomical structure to be treated. This may include 3D models (e.g., point clouds, meshes, CAD) containing data. These virtual models are provided to the navigation controller 22 prior to the surgical procedure, It may be preserved.

[0045] In addition to, or instead of, preoperative imaging, kinematic studies, bone tracing, and other This same method can be used to plan treatment in the operating room. It is also possible to generate a virtual model of the above.

[0046] Prior to the surgical procedure, in addition to a virtual model corresponding to the patient's anatomy, The navigation controller 22 is configured to display the surgical instruments and Other objects of interest to the surgical procedure, such as the surgeon's hands and / or fingers It can receive and store virtual models of other tracked objects. The position controller 22 also controls the position of the tracker and the object fixed to the tracker. The positional relationship between the localizer 18 and the visual device, and the surgical plan are also important for the surgical procedure. Specific surgical data can also be received and stored. Surgical planning allows the patient to participate in the surgical procedure. Identify the anatomy of the patient involved and identify the instruments used in the surgical procedure. This allows for the definition of the planned trajectory of the instrument and the planned movement of the patient's tissue during the surgical procedure. This can be done.

[0047] During the surgical procedure, the optical sensor 32 of the localizer 18 receives signals from the trackers 34, 36, 38. It is possible to detect optical signals in the non-visible light band (e.g., infrared or ultraviolet light) from the The position of the trackers 34, 36, 38 relative to the localizer 18 is indicated based on the received optical signals. The optically based signal can be output to the navigation controller 22. The navigation controller 22 determines the positions of the trackers 34, 36, 38 and Based on the known positional relationship between the object and the camera 34, 36, 38, the localizer 1 A localizer indicating the position of the object fixed to the trackers 34, 36, 38 relative to the The data can be generated.

[0048] To complement the tracker-based object tracking provided by the localizer18 To achieve this, the surgical navigation system 12 may also include a vision device 40. The device 40 may be capable of generating a three-dimensional image of the surgical workspace area in real time. The position of the invisible light signals transmitted from the trackers 34, 36, and 38 can be detected and identified. Unlike the localizer 18, which may be limited to a depth map, the vision device 40 and so on to obtain a three-dimensional image of the interior and surrounding surfaces of a target volume within the field of view of the vision device 40. The vision device 40 may be configured to generate one or more image sensors 42 and optical Each of the image sensors 42 may be a CMOS sensor. good.

[0049] For example, the vision device 40 may illuminate the surgical workspace with non-visible light, such as infrared or ultraviolet light. By illuminating the exposed surfaces of the surgical workspace, a depth map can be generated. In this case, the surface reflects the non-visible light back into one or more image cells of the vision device 40. The transmission of invisible light can be detected by the visual device 40. Based on the time of flight, the visual device 40 determines whether the visual device 40 and the exposed surfaces of the surgical workspace are aligned. As a result, the distance between the visual device 40 and several points on the may generate a depth map showing the distance and angle between the vision device 40 and each surface point. Alternatively, the vision device 40 may be, for example, a structured light projection, laser ranging, or stereoscopic vision device. Other modalities may be used to generate the depth map, including but not limited to: stomach.

[0050] Similar to the localizer 18, prior to the surgical procedure, the visual device 40 is preferably The visual device 40 may be positioned to view a clear surgical workspace. Alternatively, the visual device 40 may be provided separately. and positioning the visual device 40 separately from the localizer 18 by attaching it to an adjustable arm. The visual device 40 may be, for example, for Identifying and Tracking Physical Ob jects During a Robotic Surgical Procedure No. 10,531,926 entitled "Method and Method for Manufacturing a Microwave Oven," the contents of which are incorporated by reference in their entirety. (the entire disclosure of which is incorporated herein by reference) The vision device 40 may also be attached to the navigation controller 22. The device may be in operative communication with the

[0051] As described above, the navigation controller 22 is generated using the localizer 18. The tracker-based localization data generated by the vision device 40 and the depth map generated by the vision device 40 are used to configured to track objects and identify obstacles within a surgical workspace based on a Specifically, the vision device 40 may simultaneously generate a depth map of the surgical workspace. The localizer 18 is then fixed to a tracker in the surgical workspace relative to the localizer 18. Optically based radars are used to generate localizer data indicating the location of objects. The data may be generated by the vision device 40. Thus, the depth map generated by the vision device 40 may be generated by the and the localizer data generated by the localizer 18 are interleaved in time. In other words, each input of the localizer data generated by the localizer 18 may be The stance is temporally associated with different depth maps generated by the vision device 40. and the location of the object as indicated in the localizer data. The positions of these objects in the depth map are then compared to the same instant during the surgical procedure. Respond.

[0052] In response to determining the localizer data, navigation controller 22: The expected data to be captured by the vision device 40 and associated with the location data The tracker may be configured to generate a depth map. The localizer data and time are compared, assuming that only the selected object is present in the surgical workspace. , a depth map expected to be generated by the visual device 40. The navigation controller 22 may be configured to determine the surgical operation space indicated by the localizer data. The detected position of the object fixed to the tracker in the area, the virtual model corresponding to the object, and a predicted depth map based on the positional relationship between the localizer 18 and the visual device 40. It may be configured to determine

[0053] The navigation controller 22 then temporally associates the localizer data with The actual depth map generated by the vision device 40 can be obtained, and the expected Portions of the real depth map that do not match the depth map can be identified. The application controller 22 then applies a laser beam to a target volume of patient tissue to be treated based on the identified portion. Objects in the surgical workspace, such as objects other than those fixed to adjacent trackers objects can be identified, and any such object can be It can be determined whether it will cause any disruption to the track.

[0054] The surgical instrument 16 may form part of the end effector of the robotic manipulator 14. The robot manipulator 14 includes a base 46 and several arms extending from the base 46. and several active links 48 for moving the surgical instrument 16 relative to the base 46. The link 48 may include a serial arm structure as shown in FIG. The robot may be configured to form a structure (e.g., as shown in FIG. 3), or any other suitable structure. The robot manipulator 14 is operated by a user holding the end effector of the robot manipulator 14 (for example, For example, a force / torque sensor may be used to directly or actively drive the robot manipulator 14. The invention may include the ability to operate in a manual mode, which initiates movement of the surgical instrument 16 (via sensor measurement). The robotic manipulator 14 also controls the surgical instrument 16 to move in a predetermined manner. a semi-autonomous mode in which the robot is moved by the robot manipulator 14 along a fixed path; (For example, the active joints of the robot manipulator 14 can be controlled from the user to the end (operated to move the surgical instrument 16 without requiring force / torque on the effector) An example of operation in semi-autonomous mode is provided in Bowling, et al., incorporated herein by reference. No. 9,119,655 to t al. to the base 46 of the robot manipulator 14 to track the movement of the base 46 due to A separate tracker may also be attached.

[0055] Similar to the surgical navigation system 12, the robotic manipulator 14 is The processing of the robot manipulator 14, more particularly the manipulator control a manipulator including a processor 52 programmed to perform the processing of the controller 50; The controller 50 may house, for example, a processor 52. The movement of the link 48 controls the operation and movement of the surgical instrument 16, such as in response to a command from the system 12. It may be programmed as follows.

[0056] During a surgical procedure, the manipulator controller 50 controls the navigation controller 22 The surgical instrument 16 is moved to a location based on navigation data received from the This determination, together with the current position of the surgical instrument 16, may be configured to determine the desired location of the surgical instrument 16. Based on this information, the manipulator controller 50 moves the surgical instrument 16 from its current position. Determine the extent to which each of the links 48 needs to be moved to reposition it from the desired position. The data indicating where the link 48 should be relocated may be , a joint motor controller that controls the active joint of the robot manipulator 14 The data can be transferred to a controller (e.g., one for controlling each motor). In response to receiving the data, the joint motor controllers move the links 48 in accordance with the data. However, it can be configured to move the surgical instrument 16 to a desired position as a result.

[0057] Referring now to FIG. 2, the localizer 18 and the vision device 40 each A riser controller 62 and a vision controller 64 may be included. The controller 62 is connected to the optical sensor 32 of the localizer 18 and to the navigation controller. During a surgical procedure, the localizer controller 62 may be communicatively coupled to the localizer controller 22. The optical sensor 32 is operated to receive the signals received from the trackers 34, 36, and 38. The optical signal may be configured to generate optical-based data indicative of the optical signal.

[0058] The trackers 34, 36, 38 include at least three optical sensors for transmitting optical signals to the optical sensor 32. The tracker may be an active tracker, each having two active markers. 34, 36, 38 may be powered by an internal battery or may be powered by a navigation controller. Each tracker may have a lead to receive power via the controller 22. The active markers 34, 36, and 38 are light-emitting diodes that emit light such as infrared or ultraviolet rays. Each of the trackers 34, 36, and 38 may be a tracker. The LEDs 65 of the cameras 34, 36, and 38 are connected to the navigation controller 22. The tracker controller 66 may also include a navigation controller. The LEDs 65 of the trackers 34, 36, and 38 are illuminated by instructions from the tracker controller 22 or the like. For example, the trackers 34, 36, and 38 may be configured to control the route and order of The tracker controller 66 controls the trackers 34, 35 by the navigation controller 22. To facilitate differentiation between trackers 34, 36, and 38, the LEDs 65 of each tracker 34, 36, and 38 are set to different values. The light can be emitted at a rate and / or time.

[0059] The sampling rate of the optical sensor 32 is determined by the LEDs 65 that sequentially emit light. The optical sensor 32 receives optical signals at a rate of 100 Hz or greater, or more preferably or have a sampling rate of 300 Hz or higher, or most preferably 500 Hz or higher. For example, the optical sensor 32 may have a sampling rate of 8000 Hz. That's fine.

[0060] The trackers 34, 36, and 38 are not active trackers, but are generated by the localizer 18. A reflective object or the like that reflects the light that is irradiated onto the light source (e.g., light emitted from the light source 44 (FIG. 1)) It may also be a passive tracker including a passive marker (not shown). The light may be received by an optical sensor 32 .

[0061] In response to receiving optical signals from the trackers 34, 36, and 38, the optical sensor 32 The optical sensor 32 indicates the position of the trackers 34, 36, 38 relative to the localizer 18. Also, correspondingly, objects firmly attached to trackers 34, 36, and 38 The optically based data indicating the position of the object relative to the localizer 18 is sent to the localizer controller. Specifically, each optical sensor 32 may output to a tracker 34, 36, 38. 1, which detects optical signals from the sensor and accordingly indicates the position within the sensor area where each optical signal is detected. The detected position of each optical signal within a given sensor area may be determined by: The angle at which the optical signal is received by the optical sensor 32 including the sensor area can be based on the angle at which the optical signal is received by the optical sensor 32 including the sensor area. Similarly, the position of the source of the optical signal in the surgical workspace can be addressed.

[0062] Thus, in response to receiving an optical signal from the trackers 34, 36, 38, each optical sensor 32 is an optically based detector that indicates the location within the sensor area of the optical sensor 32 where the optical signal is detected. The optical sensor 32 can generate such optically based data. The localizer controller 62 can then , and the optically based data can be communicated to the navigation controller 22. Then, the navigation controller 22 controls the localizer based on the optical-based data. 18. Tracker position data may be generated that indicates the position of the trackers 34, 36, 38 relative to the optical fiber 18. For example, the navigation controller 22 may provide local color information based on optically based data. The position of the LED 65 relative to the isometer 18 may be triangulated, and the trackers 34, 36, 37 Applying the stored positional relationship between the LED 8 and the marker, the measured localization of the LED 65 is performed. and determining the position of the trackers 34, 36, 38 relative to the localizer 18. The position may be determined.

[0063] The navigation controller 22 then determines the location of the tracker based on the tracker position data. The object is firmly attached to the camera 34, 36, 38 relative to the localizer 18. Specifically, localizer data can be generated to indicate the location of the navigation system. The controller 22 includes trackers 34, 36, and 38. You can retrieve the stored positional relationships with the objects that are currently being used, and you can The function is applied to the tracker position data to determine the position of the object fixed to the tracker 34, 36, 38. The position of the object relative to the localizer 18 can be determined. The controller 62 calculates the tracker position data and and / or localizer data, and tracker position data and / or transmitting the localizer data to the navigation controller 22 for further processing. It is also possible to understand

[0064] The vision controller 64 controls the light source 44 and one or more image sensors 42 of the vision device 40. and may be communicatively coupled to the navigation controller 22. The tracker 62 controls the localizer 18 to track the trackers 34, 36, 37 in the surgical workspace. 8. The vision controller 64 simultaneously generates optically based data indicating the position of the A vision device 40 is capable of generating a depth map of the exposed surfaces of the surgical workspace. Specifically, the vision controller 64 provides the image sensor 42 with image data that forms the basis of the depth map. The depth map can be generated based on the image data. The vision controller 64 then navigates through the depth map for further processing. Alternatively, the vision controller 64 may transfer the image data to the vision controller 22. The navigation controller 22 can then The imager 22 may generate a depth map based on the received image data.

[0065] In general, the depth map generated by the vision device 40 is generated by the vision device 40 and the vision The depth map may indicate the distance between the surface and the object that is within the field of view of the device 40. In other words, the depth map may indicate the distance between the surface and the object that is within the field of view of the device 40. , which can show the topography of surfaces within the surgical workspace from the perspective of the vision device 40. Each depth map generated by the vision device 40 corresponds to an image frame of the vision device 40. The image component may include a plurality of image components forming a depth map. , which may be similar to a pixel from the center of the visual device 40 to the A vector can be determined to a point on a surface that is within the field of view. For example, The location of an image component within a frame is determined by the horizontal component of the vector subtended by the image component. and vertical components, and the color of the image component may correspond to the In one embodiment, the distance to the visual device 40 may correspond to the depth component of the vector. Image components representing surface points in the surgical workspace that are farther from the vision device 40 are more likely to be image components representing surface points in the surgical workspace that are farther from the vision device 40. may have a lighter color than the image component representing

[0066] The vision device 40 may be a depth camera that includes one or more depth sensors 68. The depth sensor 68 detects light, such as non-visible light, reflected from surfaces within the field of view of the depth sensor 68. During a surgical procedure, the light source 44 may be adapted to illuminate red by the vision controller 64. The target area may be illuminated with non-visible light such as ultraviolet light. The intensity sensor 68 can detect the reflection of non-visible light from the surface of the target site, thereby This may enable the vision controller 64 to generate a depth map.

[0067] For example, the vision controller 64 may be configured to handle distances between the vision device 40 and various points. based on the time it takes for light emitted from the light source 44 to reflect off a point on an exposed surface within the target site. The vision controller 64 may then generate a depth map (i.e., time-of-flight). These determined distances can be used to generate a depth map. Illustratively, light source 44 projects a known structured invisible light pattern onto the exposed surface of the surgical site. The depth sensor 68 may then measure the distortion based on the topography of the surface within the target region. Therefore, the vision controller can detect reflections of known patterns that may be present. 64 is a barge with a known pattern and a distorted pattern detected by a depth sensor 68. The imaging device may be configured to generate a depth map of the target region based on a comparison with the imaging.

[0068] Alternatively, the vision device 40 may be an RGB camera that includes one or more RGB sensors 70. The RGB sensor 70 may be configured to generate a color image of the exposed surface at the target site. The vision controller 64 may be configured to generate a depth map based on the color image. The device may be configured to:

[0069] For example, similar to the structured light methodology described above, the vision controller 64 may direct the light source 44 to The device projects a known structured light pattern onto the target site, e.g., in a color that is out of sync with the color at the site. The RGB sensor 70 may then generate an RGB image of the target area. This image can be imaged using known structured light patterns based on the surface topography of the target area. The vision controller 64 may perform pattern recognition, edge detection, , and distorted versions of known structured light patterns from RGB images using color recognition, etc. can be extracted and compared with a known structured light pattern and the extracted distorted version. The depth map may be determined based on:

[0070] As a further alternative, the vision device 40 may use the principles of stereoscopic vision to obtain a depth map of the target area. More specifically, multiple depth sensors 68 or RGB Multiple image sensors 42, such as sensor 70, may be used to view the target area from different angles. The vision controller 64 may be configured to direct each image sensor 42 to capture images of the target from different angles. For example, the image sensor 42 may be configured to simultaneously generate an image of the target area. If the sensor 68 is a target, the vision controller 64 causes the light source 44 to emit a pattern of invisible light. The depth sensors 68 may be configured to illuminate the exposed surface of the site, and each of the depth sensors 68 may be configured to illuminate the exposed surface. The pattern of invisible light reflected by the object may be captured from different angles. The imager 64 calculates the surface points based on the positions of the surface points in each image and the known positional relationship between the image sensors 42. The three-dimensional position of a point on the surface of the target area relative to the vision device 40 can be determined. The vision controller 64 then generates a depth map based on the determined three-dimensional position. It can be generated.

[0071] To reduce interference between the localizer 18 and the visual device 40 during surgery, The colorizer 18 and the visual device 40 operate in different spectral bands to visualize the object within the target region. Additionally or alternatively, a vision device may be configured to detect the position of the object. When the vision device 40 uses a light source 44 to illuminate the target area, such as when the vision device 40 operates in the non-visible light band. When illuminating an exposed surface at a position, the localizer 18 detects that the light source 44 of the vision device 40 It is constructed to operate at a sufficiently short temporal exposure rate so as not to be visible to the localizer 18. It may be made.

[0072] As mentioned above, the navigation controller 22 uses the localizer 18 to generate calculating a predicted depth map based on the localizer data; and The surgical site can be visualized by comparing the depth map with the actual depth map generated by the vision device 40. The present disclosure also provides methods for determining objects adjacent to a target volume of patient tissue to be treated within the target volume. The functions, features, and processes of the navigation controller 22 are described in the In addition to the processor 23, the navigation system may include a The operation controllers 22 each include a memory controller operatively coupled to a processor 23. The memory may include a memory 72 and a non-volatile storage 74.

[0073] The processor 23 may be a microprocessor, a microcontroller, a digital signal processor, or the like. processors, microcomputers, central processing units, field programmable gate arrays, Programmable logic devices, state machines, logic circuits, analog circuits, digital circuits or manipulate signals (analog or digital) based on operating instructions stored in memory 72 The memory 72 may include one or more devices selected from any other device that supports the memory 72. is read-only memory (ROM), random access memory (RAM), and volatile memory Non-volatile memory, static random access memory (SRAM), dynamic Random Access Memory (DRAM), Flash Memory, Cache Memory, or Information a single memory device, including but not limited to any other device capable of storing Non-volatile storage 74 may include a hard drive or multiple memory devices. drives, optical drives, tape drives, non-volatile solid state devices, or any other devices that store information permanently One or more persistent data storage devices, such as any other device that can store data continuously may include:

[0074] The non-volatile storage 74 includes a localization engine 76, a transformation engine 78, a vision engine 80, and surgical navigator 81. Java, C, C++, C#, Objective C, Fortran, Pas. cal, Java Script, Python, Perl, and PL / SQL, in or in combination with various programming languages and / or may be embodied by computer-executable instructions compiled or interpreted from the technology. .

[0075] The processor 23 operates under the control of software stored in non-volatile storage 74. Specifically, the processor 23 can read the software into the memory 72. as an actively running process by executing its computer-executable instructions The processor 23 may be configured to execute software that, when executed by the processor 23, The computer-executable instructions cause the processor 23 to execute the navigation controls described herein. The controller 22 may be configured to perform configured functions, features, and processes. Thus, the software, when executed by the processor 23, performs the navigation described herein. The processing of the navigation controller 22 is transmitted to the processor 23 of the navigation controller 22. The computer-executable instructions of the software are configured to cause the The functions, features, and operations of the navigation controller 22 described in the specification are The application controller 22 may be configured to implement the above.

[0076] The non-volatile storage 74 of the navigation controller 22 It may also store data that facilitates the operation of the roller 22. The software in application controller 22 uses the data stored in non-volatile storage 74. and based on that data, the navigation controller 2 described herein. 2. The functions, features, and processes of the present invention may be configured to perform the functions, features, and processes of the present invention.

[0077] For example, but not by way of limitation, data stored in non-volatile storage 74 may include: This may include model data 82, transformation data 83, and surgical plan 84. As mentioned above, the virtual model of potential obstacles, such as the surgeon's hand or fingers, and the Virtual models of anatomical structures of interest for the surgical procedure, including virtual models of the surgical instruments used The transformation data 83 may include a positional relationship in this specification, This allows the trackers 34, 36, 38, or the localizer 18, or the vision device 40 The position of an object in the surgical workspace relative to one device, such as For example, the transformation data 83 may be Trackers 34, 36, 38 and objects firmly attached to trackers 34, 36, 38 The positional relationship between the object and the localizer 18 and the visual device 40 is The surgical plan 84 identifies target volumes of the patient's anatomy that will be involved in the surgical procedure. The instruments used in the surgical procedure can be identified and the ... A planned trajectory of the tool and a planned movement of the patient tissue can be defined.

[0078] Referring again to the software running on the navigation controller 22, the position specific The localizer 18 optical sensor 32 generates the optical-based Based on the data, objects firmly attached to trackers 34, 36, and 38 The system may be configured to generate location data indicative of the location of the object relative to the localizer 18. The conversion engine 78 converts the data into a sequence of steps based on the positional relationship represented by the conversion data 83. The position of the object relative to one device of the surgical system 10 is calculated relative to another device of the surgical system 10. The Vision Engine 8 may be configured to convert the position of the object relative to the device. 0 is based on the location data generated by the location engine 76 and the transformation data 83. a predicted depth map based on the depth map generated by the vision device 40; It is constructed to identify and track objects within the surgical workspace by comparing them with a real-world depth map. The surgical navigator 81 can then use the identification and tracking determined by the vision engine 80. These software components can be configured to provide surgical guidance based on the The details of the element's functions will be explained in detail later.

[0079] Although not shown, the manipulator controller 50 and the localizer controller 62 , and the vision controller 64, each of which, upon execution of its computer-executable instructions, configured to perform the functions, features, and operations of the controller described herein Includes a processor, memory, and non-volatile storage containing data and software It is also possible.

[0080] An embodiment of a surgical system 10 is shown in FIG. 1 and in more detail in FIG. 2. are not intended to be limiting. In practice, surgical system 10 may include many more components. or may have fewer components and may use alternative components and / or implementations. For example, all or part of the localizer engine 76 may be a localizer controller. This may be implemented by the localizer controller 62. In one embodiment, 62 is securely attached to the trackers 34, 36, 38 based on the model data 82. may be configured to generate localizer data indicative of the location of the selected object. .

[0081] FIG. 3 illustrates the coordinate systems of various objects and devices used with surgical system 10. The navigation controller 22 converts the navigation data into a The positional relationship determined by the conversion data 83 stored in the motion controller 22 is Based on this, we can map the position of an object in one coordinate system to the position of the object in another coordinate system. Such a conversion may be performed by the navigation controller. 22 to track objects within the surgical system 10 relative to a common coordinate system. Furthermore, this transformation can be transmitted to the navigation controller via the visual engine 80 or the like. The positioning engine 76 generates a visual data based on the positioning data. and calculating an expected depth map that will be generated by the vision device 40. Identify objects based on a comparison of the actual depth map generated by the algorithm with the predicted depth map As one non-limiting example, the conversion data 83 may be defined as Each of the positional relationships that allow transformation between coordinate systems is defined by transformation data 83. It may be represented by a transformation matrix that is

[0082] The navigation controller 22 controls the track based on the localizer coordinate system LCLZ. Objects within the target site, such as objects within the target site attached to the racks 34, 36, 38 The localizer coordinate system LCLZ can be configured to track an object. These can include the position of the x-axis, y-axis, and z-axis relative to the surgical workspace. The localizer coordinate system LCLZ is fixed to the localizer 18 and can be determined by the Specifically, the center point of the localizer 18 is the The origin of the localizer coordinate system LCLZ may be determined. The position of the object relative to the localizer 18 is determined using the localizer 18. The position of such an object in the localizer coordinate system LCLZ can be calculated by It can be shown in the following way.

[0083] Through this procedure, one of the objectives is to maintain the localizer coordinate system LCLZ in a known position. This can happen if the localizer 18 is inadvertently hit by operating personnel. To detect sudden or unexpected movements of the localizer coordinate system LCLZ, such as A speedometer may be attached to the localizer 18. The detected localizer coordinates LCLZ In response to the movement, the navigation controller 22 may, for example, via the surgical navigator 81. and presenting a warning to the surgeon via the user interface 24, Stop navigation and / or manipulating the surgical system 10 until it is recalibrated. The manipulator controller 50 receives a signal to stop the movement of the surgical instrument 16. The signal may be configured to be transmitted to the controller 50.

[0084] Each object tracked by the surgical system 10 has an object fixed to it. It has its own coordinate system centered on the object, separate from the localizer coordinate system LCLZ. For example, trackers 34, 36, and 38 each have a bone tracker coordinate system BTRK1. , the bone tracker coordinate system BTRK2, and the instrument tracker coordinate system TLTR, The patient's femur F is fixed in the femoral coordinate system FBONE, within which The patient's tibia T may be centered and fixed within the tibial coordinate system TBONE. Prior to the surgical procedure, each tracked bone, such as the femur F, the tibia T, and the surgical instrument 16, may be Preoperative images and / or virtual models of the object are positioned in the object's coordinate system. For example, the coordinate system of the object is mapped according to the The object may be mapped to the object by

[0085] Throughout the early stages of the surgical procedure, the trackers 34, 36 are mounted on the patient's femur F and tibia F, respectively. The coordinate systems FBONE and TBONE can be fixedly attached to the bone T. , can be mapped to the coordinate systems BTRK1 and BTRK2, respectively. No. 6,269,793, filed on Oct. 1, 2004. U.S. Patent No. 6,269,793, filed on Oct. 1, 2004, which is incorporated herein by reference. Pointers with their own tracker PT, as disclosed in Patent No. 7,725,162, etc. Using the instrument P (Figure 1), the femoral coordinate system FBONE and the tibial coordinate system TBONE are The femur coordinates may be aligned to the bone tracker coordinate systems BTRK1 and BTRK2, respectively. The fixed positional relationship between the FBONE coordinate system and the bone tracker coordinate system BTRK1, and the tibia coordinate system TBO The fixed positional relationship between NE and the bone tracker coordinate system BTRK2 is used as the transformation data 83 for navigation. The information can be stored in the application controller 22.

[0086] Given a fixed spatial relationship between the femur F and tibia T and their trackers 34, 36, and via the conversion engine 78, the navigation controller 22 converts the femoral coordinate system The position of femur F in FBONE is calculated by the position of femur F in the bone tracker coordinate system BTRK1. The position of the tibia T in the tibia coordinate system T Bone can be converted to the position by the bone tracker This can be converted to the position of the tibia T in the coordinate system BTRK2. The position of the trackers 34, 36 in the localization coordinate system LCLZ is determined using the colorizer 18. By doing so, the navigation controller 22 respectively defines the position identification coordinate system LCLZ Determining the position of the femoral coordinate system FBONE and the position of the tibial coordinate system TBONE in and correspondingly, the femur F and tibia T in the localization coordinate system LCLZ are The position of the

[0087] Similarly, the treating end of the surgical instrument 16 is fixed and centered within the coordinate system EAPP. The origin of the coordinate system EAPP may be fixed to the center of gravity of the surgical cutting burr, for example. The position of the coordinate system EAPP and, correspondingly, the position of the treatment end of the surgical instrument 16 are determined by the time when the procedure begins. The coordinate system E may be fixed in the instrument tracker coordinate system TLTR of the tracker 38 beforehand. The constant positional relationship between the APP and the instrument tracker coordinate system TLTR is also calculated as transformation data 83. The localizer 1 may be stored in the navigation controller 22. 8 to determine the position of the instrument tracker coordinate system TLTR in the positioning coordinate system LCLZ. By doing so, the navigation controller 22, via the conversion engine 78, etc. Based on the positional relationship between the tool tracker coordinate system TLTR and the coordinate system EAAP, the positioning coordinate system LC The position of the coordinate system EAPP in the LZ can be determined and, correspondingly, the localization coordinates The position of the treating end of the surgical instrument 16 in the reference LCLZ can be determined.

[0088] The vision device 40 may similarly be fixed and centered within a visual coordinate system VIS. The origin of the visual coordinate system VIS may correspond to the center of gravity of the visual device 40. Each real depth map generated by 40 is a reference to the visual device 40, as described above. The position of the exposed surface within the target area can be shown, and the position of the exposed surface in the coordinate system VIS can be synchronized. It can be shown in the following way.

[0089] As shown in FIG. 1, the visual device 40 is integrated with the localizer 18. Sometimes the visual coordinate system VIS and the localizer coordinate system LCLZ can be considered equivalent. In other words, the position of an object or coordinate system in the localizer coordinate system LCLZ is The position of the object or coordinate system in the visual coordinate system VIS is so close to or equal to the Alternatively, the visual coordinate system VIS can be used when the visual device 40 is connected to the localizer 18. When integrated, it can be fixed in the localizer coordinate system LCLZ, and vice versa. Therefore, the positional relationship between the visual coordinate system VIS and the localizer coordinate system LCLZ, and The positional relationship between the visual device 40 and the localizer 18 is determined by the manufacturing process of the surgical system 10. This may be determined during manufacturing and stored in the navigation controller 22 at the factory as conversion data 83. It may be remembered.

[0090] When the vision device 40 is separated from the localizer 18, the vision device 40 The positional relationship between the visual coordinate system VIS and the localizer coordinate system LCLZ, and the corresponding visual To establish the positional relationship between the device 40 and the localizer 18, The navigation system may include a tracker (not shown) rigidly attached to the housing. The motion controller 22 has a transformation data table that stores the positional relationship between the tracker coordinate system and the visual coordinate system VIS. In this way, the localizer 18 may be used to The navigation is performed by determining the position of the tracker's coordinate system in the localization coordinate system LCLZ. The position controller 22 calculates the coordinate system of the tracker and the visual coordinate system VIS. Based on the positional relationship, the position of the visual coordinate system VIS in the localization coordinate system LCLZ is determined. and correspondingly, the visual device 40 in the localizer coordinate system LCLZ. Additionally, the position of the navigation controller can be determined. 22 is the localizer 18 in the localizer coordinate system LCLZ and the visual coordinate system VIS. The position of the vision device 40 relative to the object can be determined.

[0091] Alternatively, the navigation controller 22 may be inserted into the target site and configured to control the localizer 1. based on a common light pattern detectable by both the camera 8 and the vision device 40. The riser common coordinate system LCLZ is configured to determine the positional relationship between the visual coordinate system VIS. For example, the localizer 18 and the vision device 40 may be positioned to view the target area. After the target site is illuminated, a pattern of light, such as invisible light, may be projected onto the target site. may reflect that light pattern back to the localizer 18 and the visual device 40. The navigation controller 22 directs the light source 44 of the vision device 40 to project this light pattern. The light pattern may be projected onto the target site, or a separate light projector (not shown) may be used. As a further example, the localizer 18 and the visual device 40 may be a tracker having a marker configured to emit a pattern of light detectable by the or other physical device such as a pointer PT (FIG. 1) within the target site and localized The optical system may be positioned within the field of view of the optical sensor 18 and the visual device 40.

[0092] The navigation controller 22 receives localization information via a localizer engine 76 or the like. Using the localizer 18, the localizer coordinate system LCLZ specific to the localizer 18 is The navigation system may be configured to generate localization data indicative of the location of the light pattern. Controller 22 also receives a calibration depth map representing the light pattern from vision device 40. The light pattern in the visual coordinate system VIS can be taken through a conversion engine 78, etc. The navigation system may be configured to determine the position of the object based on a calibrated depth map. The application controller 22 uses a regression algorithm or the like via a conversion engine 78 or the like. The projection pattern in the localization coordinate system LCLZ and the visual coordinate system LCLZ is determined. Based on the position, the positional relationship between the position identification coordinate system LCLZ and the visual coordinate system LCLZ is determined. The device may be configured to:

[0093] Figure 4 shows the use of tracker-based localization and machine vision to locate objects within a surgical workspace. for tracking an object and determining whether the object interferes with a surgical plan - Patents.com 100. The method 100 is performed by the surgical navigation system 12, more specifically , may be executed by the navigation controller 22.

[0094] In block 102, the positions of the localizer 18 and the vision device 40 in a common coordinate system are determined. Specifically, the positional relationship can be identified through the conversion engine 78 or the like. The motion controller 22 may use any of the above methods to coordinate the localizer 18 with the visual device. The positional relationship with the vise 40 is identified, and the localizer coordinate system LCLZ and the visual coordinate system LCLZ are correspondingly determined. For example, the visual device 40 may be configured to specify the positional relationship with the reference system VIS. The tracker may be fixed, or may be tracked by both the localizer 18 and the vision device 40. A light pattern may be placed at the detectable target site. Alternatively, the localizer 1 may be configured at the time of manufacture. 8 is integral with or otherwise fixed to the visual device 40. When the navigation controller 22 is connected to the camera, the positional relationship is determined at the time of manufacture, and the conversion data is transmitted to the navigation controller 22. The data may be stored in advance as data 83.

[0095] In block 104, one or more objects within the target region are identified based on the transformation data 83, etc. The transformation data 83 can be used to access a virtual model corresponding to the object. Indicating objects within the target region to which trackers such as 34, 36, and 38 are attached The navigation controller 22 can In some examples, the virtual model may be obtained for each of the One or more of these acquired virtual models may be used to define the target volume to be treated during the surgical procedure. can also be defined.

[0096] 5 shows a target site 200 in a patient undergoing knee replacement surgery. The target site 200 includes: A patient may include a target volume 202 of bone tissue to be removed with a surgical instrument (e.g., surgical instrument 16). The target site 200 may include a portion of the subject's femur F. The target site 200 may include ligaments 204 and epidermal tissue 206. 06. The target region 200 may further include soft tissue adjacent to the target volume 202. is positioned to retract the epidermal tissue 206 and provide access to the patient's femur F. The target site 200 may also include surgical tools such as a retractor 208. The navigation system may include a tracker 209 firmly attached to the femur F. The virtual controller 22 extracts a virtual model corresponding to the femur F from the model data 82. An example of this is shown in Figure 6.

[0097] Referring again to FIG. 4, in block 106, the sensor attached to the tracker within the target region is The location of the object can be detected using a localizer 18. Specifically, , the localizer 18 is used to calculate the coordinates of each tracker in the localizer coordinate system LCLZ as described above. Generates optically based data that indicates the position of a coordinate system and correspondingly indicates the position of each tracker The navigation controller 22 may be configured to For example, the detected positions of the trackers and their coordinate systems, as well as the transformation data, are transmitted via a digital signal 76 or the like. Based on the positional relationship between the tracker and the object shown in the data 83, the localizer Determine the position of the coordinate system of the object attached to the tracker in the coordinate system LCLZ The tracker may be configured to detect the position of the object. The virtual model of each attached object is then converted to the coordinate system of the object in the transformation data 83. The navigation controller 22 can also map the object The detected position of the tracker to which the object is attached, and the positional relationship between the tracker and the object Based on the relationship, the localizer coordinates of the virtual model of each object attached to the tracker are The position in the reference system LCLZ can be specified.

[0098] FIG. 6 shows a continuation of the target site 200 of FIG. 5, with the tracker 209 attached within the target site 200. 2 shows a virtual model 210 that may correspond to a patient's femur F, to which the femur F may be attached. 0 is the position of the tracker 20 in the localizer coordinate system LCLZ determined by the localizer 18. Depending on the position of the femur F and the posterior femur 9, the femur F may be positioned in the localizer coordinate system LCLZ. The virtual model 210 defines a virtual target volume 212 that will be resected from the femur F during treatment. 5. The target volume 202 shown in FIG.

[0099] In block 108, for example, the accessed virtual model, the localizer coordinate system LCL The detected position of the object corresponding to the virtual model in Z and the locus in the common coordinate system A predicted depth map is generated based on the positional relationship between the colorizer 18 and the visual device 40. As mentioned above, the position of the virtual model in the localizer coordinate system LCLZ is The position of the object in the localizer coordinate system LCLZ determined using Isa 18 The navigation controller 22 can respond to the request via the visual engine 80, etc. The positional relationship between the localizer 18 and the visual device 40 in the common coordinate system, and Based on the positional relationship between the localizer coordinate system LCLZ and the visual coordinate system VIS, The position of the virtual model in the LCLZ coordinate system is calculated by It may be configured to convert the position of the

[0100] Then, the navigation controller 22 calculates the coordinates of the virtual model in the visual coordinate system VIS. Based on the position, a predicted depth map can be generated. The depth map generated by 40 provides a visual representation of the exposed object surface within the target region. The position (e.g., depth and location) of the object relative to the device 40 can be shown in the visual coordinate system VIS. The position of the virtual model is determined by the visual device 4 of the object surface represented by the virtual model. The position relative to 0 can be immediately indicated and fixed within the visual coordinate system VIS. Therefore, the navigation controller 22 may move the target area without any other objects in it. , and the virtual model in the visual coordinate system VIS is generated via the visual engine 80 or the like. Based on the determined location, a visual device 40 is generated that focuses on the target area. It may be configured to simulate an imagined depth map.

[0101] 7 illustrates a predicted depth map that may be generated based on the virtual model 210 of FIG. The predicted depth map of the ligament 204, the epidermal tissue 206, the retractor 208, and the tracker 209 In the visual coordinate system VIS, assuming that no other objects are present in the target area, According to the transformed position of the virtual model 210, the femur F of the patient is visualized by the visualization device 40. The expected depth map in Figure 7 can be simulated. The map may also be cropped to a region of interest, as described in more detail below.

[0102] At block 110, a real depth map captured by the vision device 40 is received. Specifically, in block 106, the localizer 18 detects a target region. Generate localizer data that indicates the location of objects attached to the tracker. At the same time, the vision device 40 may generate a depth map of the target area as described above. In this way, the depth map may be interleaved in time with the localization data, It may also be temporally interleaved with estimated depth maps generated based on specific data. In other words, both the actual depth map and the predicted depth map are substantially Target sites can be depicted at the same time point.

[0103] FIG. 8 is a diagram of a visual image generated by the visual device 40 viewing the target site 200 depicted in FIG. 2 shows a depth map that can be obtained by performing the above steps, with tracker 209 removed for clarity. The depth map is arranged in a matrix, similar to pixels, and the image of the depth map is Box 214 can include several image components that form a frame. To highlight one example of an image component, it is artificially placed on the illustrated depth map. The location of each image component in the depth map image frame is determined by the visual device. The horizontal and vertical distances from the central viewpoint of the image element 40 can be represented, and the brightness of each image element can be represented by the image element 40. may correspond to the distance from the visual device 40 of the object surface point represented by the component. In the illustrated example, the bright image components represent surface points of the target region that are closer to the vision device 40. However, dark image components represent surface points of the target region that are far from the vision device 40.

[0104] In block 112, the real depth map is transferred to the virtual model accessed in block 104. The detected position of the object corresponding to the virtual model in the localizer coordinate system LCLZ is calculated. and the positional relationship between the localizer 18 and the visual device 40 in the common coordinate system. Based on this, the region of interest (ROI) for the surgical procedure can be trimmed. As will be described in further detail, the actual depth map is compared to the predicted depth map to determine the depth of the target area. and identifying any such object in the target volume within the target region. The actual depth map and the The larger the dimensions of the expected depth map, the more computational effort is required for the comparison. , the navigation controller 22, via the vision engine 80 or the like, To reduce the image dimensions, the actual image is based on the position of the virtual model in the visual coordinate system VIS. The depth map may be configured to be cropped to the ROI. The position of the virtual model in the VIS system is determined by the object localizer coordinate system LC The position of the LZ, the positional relationship between the localizer 18 and the visual device 40 in the common coordinate system, It may be determined based on the following.

[0105] For example, the virtual model accessed in block 104 may be used to visualize the shape of the object to be treated during a surgical procedure. The target volume can be defined. The position of the virtual model in the visual coordinate system VIS is therefore , the location of the target volume in the visual coordinate system VIS can be indicated, and correspondingly, the visual coordinate system The location of the target volume in the real depth map generated by the device 40 can be indicated. The navigation controller 22, via the visual engine 80 or the like, generates a real depth map. The target volume is then cropped to any location greater than a threshold distance from the target volume's location in the visual coordinate system VIS. Additionally or alternatively, the vision engine 80, etc. Through this, the navigation controller 22 can select a shape or procedure specific to the user's selection. The shape is centered on the location of the target volume in the depth map, and the area outside the shape in the depth map is The navigation controller 22 may be configured to remove the predicted depth map. During or after the calculation of the predicted depth map, the size and shape of the predicted depth map may be adjusted to the cropped actual depth map. It may be configured to fit within the size and shape of the map.

[0106] FIG. 9 shows a virtual model 210 corresponding to the femur F of FIG. 6, cropped to an ROI. 8. The virtual model 210 represents the target volume to be treated during the surgical procedure. 202. The navigation controller 22 may define a virtual target volume 212 corresponding to the , the determination of the target volume in the visual coordinate system VIS, which can indicate the location of the target volume in the real depth map. Based on the determined position, for example, the selected shape is centered at the center of the target volume in the depth map. By placing a shape in the depth map and removing the area of the depth map outside the shape, the real depth map in Figure 8 is obtained. The predicted depth map in Figure 7 is similarly cropped. Fits into the size and shape of the actual depth map.

[0107] In blocks 114 and 116, portions of the depth map that do not match the expected depth map are removed. Specifically, in block 114, the difference between the actual depth map and the predicted depth map is determined. The actual depth map can be compared to the expected depth map, such as by calculating a min. The navigation controller 22, via the vision engine 80 or the like, receives the predicted depth map and By computing the depth difference between each corresponding pair of image components in the depth map and the real depth map, , the difference between the predicted depth map and the actual depth may be calculated. The corresponding image component pairs in the depth map and the real depth map are The horizontal and vertical locations may be included in the same location. Assuming the depth map has been cropped to the ROI, the depths compared in block 114 are The map may be a cropped depth map.

[0108] The difference between the actual depth map and the estimated depth can be calculated appropriately, e.g., using an attached tracker. Objects that are not tracked or cannot be tracked (e.g., soft tissue, surgeon's hand) There are objects within the target area that have not yet been identified and tracked, such as This difference can be represented by a differential depth map, Each image component in the depth map has the same horizontal and vertical position as the image component in the difference depth map. The corresponding image components located in the actual depth map and the predicted depth map are calculated at the same position. 10 shows the difference in calculated depth between the actual depth map of FIG. 9 and the predicted depth map of FIG. 7. FIG. 10 shows a differential depth map calculated in

[0109] Corresponding image components from the actual and predicted depth maps that show the same depth are The difference is zero, and the object is identified and tracked previously using a tracker and localizer18 or similar. A zero depth difference may correspond to an object with maximum brightness in the difference depth map. or have a color and / or hue specific to zero depth. In the differential depth map of FIG. 10, regions 216, 218, and 220 have a depth of zero. 10 shows corresponding image components from actual and predicted depth maps with intensity difference.

[0110] Corresponding image components in the actual and predicted depth maps that do not show the same depth are zeroed out. This results in a depth difference that is not previously determined using a tracker and localizer 18 or the like. A non-zero depth difference may correspond to an untracked object. In the depth map, image components with a brightness less than the maximum brightness, or a color specific to zero depth, and / or hue may be represented using image components of different colors. In this case, the darker areas adjacent to areas 216, 218, and 210 have a non-zero depth difference. 10 shows corresponding image components of the actual and predicted depth maps.

[0111] At block 116, the calculated differences are filtered based on one or more object thresholds. The object threshold can be adjusted to accommodate noise or negligible calibration inaccuracies. Non-zero differences due to the presence of additional objects within the target region The object threshold can be designed to distinguish between differences. This may include a threshold depth and / or a minimum size threshold, each of which may be non-zero.

[0112] In one embodiment, for each of one or more non-zero sections of the differential depth map: Therefore, the navigation controller 22 determines whether the non-zero section is greater than the depth threshold. Specifically, the differential depth map may be configured to determine whether it indicates depth. Contains one or more non-zero sections, each of which is a non-zero The non-zero sections of the difference depth map may include a set of consecutive image components that indicate a depth difference of The section is the magnitude (sign) of the non-zero depth difference indicated by each image component in the non-zero section. If the depth threshold is not referenced, then the object has an absolute depth greater than the depth threshold. It can be considered that a non-zero section of the difference indicates an absolute depth greater than the threshold depth. In response to this, the navigation controller 22 may, for example, The non-zero sections in the differential depth map and the real depth map have the same horizontal position and The parts of the actual depth map that do not match the estimated depth map, for example, due to their vertical position. configured to identify sections of the real depth map that correspond to non-zero sections of the difference. You may do so.

[0113] As a further example, for each non-zero section of the difference, the navigation control The controller 22 determines whether the size (e.g., area) of a non-zero section is greater than a minimum size threshold. The non-zero section size may be configured to determine whether the size of the non-zero section is greater than or equal to a minimum size threshold. In response to determining that the visual encoder is greater than For example, a section of the real depth map can be used to capture non-zero points in the differential depth map. The horizontal and vertical positions of the image are the same in the depth map and the image section. Corresponds to non-zero sections of the difference as parts of the actual depth map that do not match the expected depth map. The method may be configured to identify a section of the real depth map to which the image is to be added.

[0114] In another embodiment, navigation controller 22 may select a non-zero section with a minimum size. If the small size is greater than the threshold and the non-zero sections indicate absolute depths greater than the threshold depth, In response to determining a difference as a portion of the actual depth map that does not match the estimated depth map. The depth map may be configured to identify sections of the real depth map that correspond to non-zero sections of the stomach.

[0115] In block 118, the presence of an object in the target region is determined based on the filtered difference. Specifically, it is possible to determine whether the object exists or not through the visual engine 80 or the like. Therefore, the navigation controller 22 detects portions of the actual depth map that do not match the expected depth map. It may be configured to determine whether the part has been specified. If not specified (block If the "No" branch of block 118 is selected, the method 100 returns to block 106 to can be used to re-locate the object to which the tracker is attached. If so (the "Yes" branch of block 118), method 100 proceeds to block 120. and applying machine vision techniques to portions of the actual depth map that do not match the expected depth map. This allows for the recognition of objects within the target area.

[0116] In block 120, the navigation controller 2, via the visual engine 80, etc. 2 applies machine vision techniques to identified portions of the actual depth map that do not match the expected depth map. The system may be configured to recognize objects within the target region from the identified portions using the For example, but not by way of limitation, the navigation controller 22 may be configured to perform pattern recognition. , edge detection, color recognition, wavelength analysis, image component intensity analysis (e.g., pixel or voxel Metrics generated through metric analysis, depth analysis, and machine learning are used to determine actual depth. Configured to segment between objects represented in a specific part of the map In some embodiments, regions of a particular part separated by edges, different dimensions, Areas with regular patterns, areas with different color palettes, and / or different depths The areas showing the range of focus may correspond to different objects. The surfaces of different objects (e.g., different tissues) within the region are visualized by the visual device 40. Possible to produce different wavelengths and / or different intensities in the reflected and detected signal. The vision device 40 outputs such information for each image component of the real depth map. Based on this information, the navigation controller 22 may be configured to based on the various wavelengths and / or signal intensities occurring over a specified portion of the real depth map. It can be configured to segment different objects in the identified area. The navigation controller 22 may display multiple objects in the area identified using machine vision. If the object cannot be found, navigation controller 22 returns the specified part. The entire image may be configured to be considered as a single object within the target region.

[0117] Additionally or alternatively, the navigation controller 22 may, for example, via a visual engine 80 Then, based on the model data 82 stored in the navigation controller 22, It may be configured to identify objects at a specified portion of the actual depth. The definition may be based on the object, for example, identifying the object as a ligament, a retractor, or epidermal tissue. Identifying a label for each object represented in the depth map that describes the type of object Identification can differ from segmentation in that each object in the target area can be identified. Once the object is identified, the navigation controller 22, via the visual engine 80 or the like, model the entire object as opposed to just the surface of the object, and This may allow for better prediction of object movements and other reactions. This allows the surgical navigator 81 of the navigation controller 22 to This could potentially allow the driver to make more informed navigation decisions.

[0118] As described above, the model data 82 stored in the navigation controller 22 is It is possible to define 3D models corresponding to objects potentially present in the target area. The model data 82 also contains location data for various objects potentially present in the target region. You can also define specific profiles, each of which is used to extract the object from the real depth map. object-specific, assisting navigation controller 22 in identifying the object; For example, a profile for a given object may be Without limitation, color palette, wavelength range, signal strength range, distance or depth range perimeter, area, volume, shape, polarization, and data from learned or statistical models corresponding to the object. The profile of a given object may include one or more of the following deep metric outputs: It also includes 3D models of objects, such as those generated from patient scans mentioned above. It's okay to be.

[0119] Therefore, the navigation controller 22 may select a location based on the identified portion of the real depth map. Based on this, at least a part of the identified part is matched to one of the predefined profiles, i.e. That is, by matching the object with a predefined profile, It may be configured to identify objects that do not match the depth map. The application controller 22 generates a real-time image as a specific object corresponding to the profile. The method may be configured to label at least a portion of the identified portion of the degree map, As a result, this can be considered adjacent to the localizer object.

[0120] In an alternative embodiment, the user may navigate through the segmented navigation menus provided by the navigation controller 22. To manually select the object in the identified part and / or to The user interface allows you to select a predefined profile for the object. The user can also interact with the user interface 24. Interact with the object represented by the real depth map in the identified area, etc. Manually tracing objects and / or predefined settings for the traced objects You can then select the profile you defined. 2. Selected, segmented, or traced object Label the selected predefined profile accordingly. The selected or traced object may be tracked. stomach.

[0121] In block 122, the position of each object recognized from the real depth map is calculated in visual coordinates. Determined in a coordinate system common to the local object, such as the VIS system or the localizer coordinate system LCLZ. For example, the navigation controller can be The 22th layer calculates the position of each object recognized from the depth map and the distance between the object and the object. and determining the position of each local object in a common coordinate system relative to a target volume that may be defined by the The navigation controller 22 may be configured to recognize objects and / or whether any local objects are obstacles to treating the target volume. This allows us to determine whether the

[0122] The navigation controller 22 calculates the coordinates of the localizer coordinate system L using the localizer 18. The detected location of local objects in CLZ, the recognized object in the real depth map The location of the object and the translation data 83 stored in the navigation controller 22 The positional relationship between the localizer 18 and the vision device 40 in a common coordinate system can be defined as and determining the position of each recognized object relative to the local object based on the relationship between the recognized object and the local object. As mentioned above, the position of the recognized object in the real depth map is determined by the visual coordinate system. It is possible to indicate the location of a recognized object in the VIS. Each image component of the real depth map that forms the object is visualized from the central viewpoint of the visual device 40. It can represent a vector to a position in the coordinate system VIS. The position of each image element in the image may indicate the horizontal and vertical components of a vector, and each image The depth indicated by the component may represent the depth component of the vector.

[0123] Therefore, the navigation controller 22 can detect the object in the real depth map. Based on the position of each recognized object, the position of each recognized object in the visual coordinate system VIS may be determined. Next, the positional relationship between the localizer 18 and the visual device 40, The position of each recognized object in the localizer coordinate system LCLZ and / or the position of each local object in the localizer coordinate system LCLZ are calculated. The position of each recognized object is used to map it to a local object in a common coordinate system. The position of the object may be determined.

[0124] In block 124, the recognized object from the real depth map and the localizer 1 are calculated. For each tracked object, including the object localized in 8, Based on the determined position of the object, etc., in a common coordinate system, A corresponding virtual boundary can be generated, specifically via the vision engine 80, etc. The navigation controller 22 generates a virtual boundary in a common coordinate system and controls the surgical instruments. The device may be configured to provide a constraint on the movement of a surgical tool such as tool 16. In addition, the navigation controller 22 also uses the localizer 18 or the like to calculate a common coordinate system. The system may be configured to track the movement of the surgical instrument 16. The virtual boundary created by the roller 22 includes sensitive anatomical structures and other surgical tools. The surgical instrument 16 may be placed in or near a space that may be occupied by other objects. Areas of the common coordinate system that should not be moved close to each other can be defined.

[0125] For example, the navigation controller 22 may According to the determined position, the 3D virtual model stored for each local object is converted to a common coordinate system. The navigation controller 22 may be configured to insert the map into a landmark. The data 82 is used to calculate the distance between a given object recognized from a specified portion of the real depth map. When defining a three-dimensional virtual model for the According to the determined position of a given recognized object in the system, a 3D virtual model is created. Additionally or alternatively, the model data 82 may be configured to be interpolated into a real coordinate system. One or more primitives for a given object recognized from a specified portion of the depth map. It may also show geometric shapes (e.g., spheres, cylinders, boxes). The image controller 22 calculates the surface topography of the object as indicated by the real depth map. Based on the above, the basic geometric shapes are sized and / or positioned, and the sized and / or positioned shapes are and / or the located basic geometry of a given object in a common coordinate system. Additionally or alternatively, the coordinate system may be configured to be interpolated into a common coordinate system according to the position of the coordinate system. For example, a virtual depth map can be generated for a given object recognized from a specified portion of the real depth map. In some cases, such as when a model or basic geometry is not shown, the navigation control La22 generates a mesh based on the surface topography of the object shown in the real depth map. Construct a boundary and map the mesh according to the determined position of a given object in a common coordinate system. The mesh boundaries may be configured to be inserted into a common coordinate system.

[0126] As a further example, navigation may be used in addition to, or instead of, one or more of the techniques described above. The motion controller 22 calculates the determined position of the given object in the common coordinate system. Therefore, by inserting a force particle into the common coordinate system, we can calculate the force of a given object in the common coordinate system. The navigation controller may be configured to approximate the boundaries of the object. 22 selects various points on the surface of the recognition object and calculates the coordinates of the various points in the common coordinate system. The force particles may be arranged in a common coordinate system at the determined positions. Each of these is a function of the force particle, which is used to determine whether other objects moving in the vicinity of the force particle in the common coordinate system are within a certain distance. Therefore, the hand in the common coordinate system During the tracking movement of the surgical instrument 16, the force particles repel the surgical instrument 16, thereby 6 can be prevented from colliding with objects constituted by force particles. Instead of representing the entire surface of the recognized object, the virtual boundary represents various parts of the surface of the recognized object. By inserting the force particles corresponding to the various points into a common coordinate system, we can achieve this with relatively little processing bandwidth and The amount of data may be used to generate a virtual boundary for the object.

[0127] For example, FIGS. 11-13 show the predicted depth map of FIG. 7 based on the difference shown in FIG. 9. The target region 200 (FIG. 5) recognized from the identified portion of the real depth map of FIG. 9 does not match the ) shows the virtual boundaries in the common coordinate system corresponding to the objects in the A virtual model of the retractor corresponding to the retractor 208 in the target region 200 and depicted in the real depth map is also shown. The coordinate system diagram, FIG. 12, corresponds to the ligament 204 of the target region 200 and is shown in the real depth map. 13 shows the ligament virtual model in a common coordinate system, and FIG. 14 shows the epidermal tissue of the target site 200. 206, and shows the virtual model of the epidermis tissue drawn on the real depth map in a common coordinate system. As an alternative embodiment, the imaginary boundaries of the ligament 204 at the target site 200 may be expressed in terms of a common coordinate system A basic geometry such as a cylinder is inserted into the common coordinate system according to the determined position of the ligament 204 in the The target site 200 may be in the form of a geometric object, The imaginary boundary is inserted into the common coordinate system at the determined location of the epidermal tissue 206 in the common coordinate system. The surface may be in the form of a mesh or force particles.

[0128] Figure 14 shows the objects recognized from the real depth map and the objects localized by the localizer 18. The relative position of the femur F of the patient is shown in a common coordinate system. 3 and a virtual model 210 of the patient's femur F, as illustrated in FIG. During a surgical procedure, the navigation controller 22 may include, for example, a surgical navigator 81. The current position of the surgical instrument 16 in the common coordinate system is calculated via the 14 together with the image or virtual model, and tracked by the localizer 18, etc. , may be configured to assist the surgeon in guiding the surgical instrument 16 to the target volume 202.

[0129] In block 126, based on the tracked object and / or the surgical plan 84, a potential It is possible to determine whether an obstacle exists in the target area. The navigation controller 22, via a gater 81 or the like, creates a real depth map. One of the tracked objects, such as a recognized object, is a target object in a common coordinate system. Based on the position of the object relative to the surgical plan 84 and the surgical plan 84, obstacles to the surgical plan 84 are identified. For example, the surgical plan 84 may be configured to determine whether the target volume is A planned trajectory of the surgical instrument 16 through a common coordinate system for treatment can be defined If the planned trajectory causes a collision with one of the virtual boundaries for the tracked object, In this case, the navigation controller 22 is configured to determine that an obstacle is present. Good too.

[0130] In response to determining that an obstacle is present (the "Yes" branch of block 126), At lock 128, remedial action may be triggered. Through the navigation controller 22, one of several available actions can be Executing one or more of the following may be configured to trigger a remedial action. For example, in response to determining that an object is an obstacle to the surgical plan 84 The navigation controller 22 then modifies the surgical plan 84 to avoid the obstacle. For example, the navigation controller 22 may be configured to The trajectory of the surgical instrument 16 is changed to avoid the injury, and the changed surgical plan 84 is input to the manipulator control. In another embodiment, the navigation system may be configured to transmit the information to the navigation controller 50 for execution. The navigation controller 22 notifies the navigation controller 22 that the obstacle has been removed. The surgical navigation system 12 provides a navigation system for the operator to navigate the surgical site. The surgical guidance and the robotic manipulator 14 may be configured to stop movement. The navigation controller 22 controls the user interface of the surgical navigation system 12. The device may be configured to trigger an alarm and / or notification of an obstruction via the interface 24. As a further example, when the obstructive object is identified as soft tissue, The navigation controller 22 controls the soft tissue guide via the user interface 24. For example, the navigation controller 22 may be configured to provide of the soft tissue object causing the harm to other objects in the target area. to illustrate the location of the obstruction and provide suggestions for mobilizing the soft tissue to remove the obstruction. The navigation controller 22 may be configured to provide soft tissue guidance. The position of the soft tissue in the common coordinate system is then monitored and the obstacle threat is removed. It may also be configured to notify the user.

[0131] Following triggering and / or overcoming of remedial action (block 128), or once an obstacle has been identified, In response to not having done so (the "No" branch of block 126), block 130 determines whether the actual depth is The movement of the recognized object may be tracked using the vision device 40. In this case, the navigation controller 22, via the visual engine 80 or the like, The additional real depth map generated by the sensor 40 is then used to recognize the object. Each recognition object is configured to monitor the state of the corresponding portion of the real depth map. For each subsequently generated real depth map, generating a predicted depth map and calculating the difference between the predicted depth map and the subsequent actual depth map; Instead of matching the difference and the saved profile to the difference, In the generated depth map, the real depth map determined to correspond to a previously recognized object is By focusing changes to parts of the map, the navigation controller 22 It may eventually be possible to monitor the movement of recognized objects at increased speed. do.

[0132] More specifically, each portion of the real depth map corresponding to a recognized object is It represents an array of features specific to the object and can be placed at a specific location in the real depth map. Although not specified, the feature array may be an array of vertices with geometric relationships specific to the object, A sequence of edges or lines with geometric relationships specific to an object, or It can be an array of depths with relative and geometric relationships specific to the object. The arrangement of the object's features and their spatial relationships to other parts of the object may be fixed.

[0133] Therefore, the navigation controller 22 can generate object-specific images at real depth. The arrangement of features in the additional depth map differs from the arrangement in the real depth map. By monitoring whether the object moves to a different position, the object is recognized from the real depth map. When the navigation controller 2 moves, the navigation controller 2 2. Based on the new position of the feature array corresponding to the object in the additional depth map Determine the new position of the object in the common coordinate system and then convert it to the common coordinate system accordingly. It may also be configured to update the virtual bounds associated with the objects in a given object. The array of features for monitoring the object's movement is represented in the object's model data 82. or by using the user interface 24 to select objects in the real depth map. It may also be set manually by the user by selecting points on the corresponding part.

[0134] For example, FIG. 15 shows the actual depth map generated by the visual device 4 after the actual depth map illustrated in FIG. 15 illustrates an additional real depth map generated subsequently by adding 0. The location of the portion representing the retractor 208 (FIG. 5) is different from the location of this portion in the depth map of FIG. The navigation controller 22 is the portion representing the retractor 208 and is positionally fixed relative to the remainder of the retractor 208 The changed position of a particular feature sequence in the additional depth map relative to the previous actual depth map. Such movement of the retractor 208 may be tracked by monitoring the position of the retractor. For example, the navigation controller 22 may be configured to move the retractor 208 to a vertex between the head and body of the retractor 208. Additional depth maps may be monitored for changes in the position of the 222 array.

[0135] In response to determining a change in the position of the array of vertices 222, the navigation controller The vertex 222 is updated to show the updated positions of the array of vertices 222 in the additional depth map of FIG. Based on the arrangement of points 222 and their constant positional relationship to the remaining retractors 208, The navigation system may be configured to determine an updated position of the retractor 208. The motion controller 22 calculates the coordinates of the retractor 20 in the common coordinate system based on the updated position. 8. FIG. 16 shows an additional depth map of FIG. Depending on the new position of the array of vertices 222 drawn on the top, the retractor 20 in the common coordinate system 8, i.e., the updated position of the virtual model corresponding to the retractor 208.

[0136] Herein, a combination of machine vision and tracker-based localization is used to A system and method for tracking objects in a workspace is disclosed. Due to the flexible nature of soft tissues such as ligaments, tracker-based localization is usually difficult. Therefore, surgical navigation systems are not suitable for tracking soft tissues. ,locating rigid objects in the surgical workspace using tracker-based localization. In addition to detecting the location, the system is configured to generate a depth map of exposed surfaces within the surgical workspace. The surgical navigation system may include a visual device configured to locate the The detected position of the object in the target area using a virtual model corresponding to the object. and the positional relationship between the localizer and the visual device in the common coordinate system. The surgical navigation system may be further configured to generate a predicted depth map of the device. The application system identifies portions of the actual depth map that do not match the estimated depth map and The target area may be configured to recognize an object including soft tissue based on the portion. The surgical navigation system then detects whether the object is an obstacle to the current surgical plan. The control unit 100 may be configured to determine whether the

[0137] Generally, the routines executed to implement embodiments of the present invention are Implemented as part of a system or as part of a specific application, component, or program be implemented as a program, object, module, or sequence of instructions, or "Computer program code" as used herein, regardless of whether it is a subset thereof , or simply "program code." Program code typically includes: It resides at various times in various memories and storage devices within the computer, and When read and executed by one or more processors in a computer The actions necessary to carry out the acts and / or elements embodying various aspects of embodiments of the invention The present invention relates to a method for performing the operations of an embodiment of the present invention. The computer readable program instructions may be written in, for example, assembly language or one or more program source code or object code written in any combination of programming languages. That's fine.

[0138] Various program code described herein may be implemented in accordance with certain embodiments of the present invention. To the extent that it is implemented, it may be differentiated based on the application. The specific program nomenclature used is merely for convenience and is therefore The invention is not limited to any particular application identified and / or implied by such nomenclature. It should be understood that the present invention should not be limited to use in computer programs. You can organize your programs into routines, procedures, methods, modules, objects, etc. the generally infinite number of ways in which it can be implemented, as well as the various software programs that reside within a typical computer. Layers (e.g., operating system, libraries, APIs, applications, Given the various ways in which program functions can be allocated between ,Embodiments of the present invention relate to the particular organization and allocation of program functions described herein. It will be understood that the present invention is not limited to the above.

[0139] A program embodied in any application / module described in this document The Code may be distributed individually or collectively in various different forms of program products. In particular, the program code may cause a processor to perform aspects of embodiments of the present invention. a computer readable storage medium having computer readable program instructions thereon for causing It may be distributed using a storage medium.

[0140] A computer-readable storage medium that is non-transitory in nature may contain computer-readable instructions, data, Any method or method for storing information such as structures, program modules, or other data. includes volatile and non-volatile, removable and non-tangible media implemented in technology. The computer-readable storage medium may further include RAM, ROM, erasable programmable logic, Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read dedicated memory (EEPROM), flash memory or other solid-state memory technology, portable Compact disk read-only memory (CD-ROM) or other optical storage, magnetic magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or can be used to store desired information and be read by a computer. The computer-readable storage medium may include any other medium capable of being stored in a non-transitory storage medium. of a signal (e.g., radio waves or other propagating electromagnetic waves, propagating through a transmission medium such as a waveguide) should not be interpreted as electromagnetic waves or electrical signals transmitted through wires per se. The computer readable program instructions are transmitted from the computer readable storage medium to the computer, Even if downloaded to another type of programmable data processing device or another device or downloaded to an external computer or external storage device via a network. It may be loaded.

[0141] The computer readable program instructions stored on the computer readable medium are , other types of programmable data processing devices, or other devices that function in a particular way. The instructions stored on the computer-readable medium are used to direct the The functions, acts, and / or operations specified in the process, sequence diagrams, and / or block diagrams The computer program instructions may be used to create an article of manufacture that includes the instructions. , a general-purpose computer, a special-purpose computer, or other programmable data processing device instructions provided to and executed by one or more processors are - The functions, acts, and / or components specified in the charts, sequence diagrams, and / or block diagrams Machines can be constructed to perform a series of calculations to perform an operation.

[0142] Certain alternative embodiments may be illustrated using flowcharts, sequence diagrams, and / or block diagrams. The functions, acts, and / or operations described herein may be rearranged in a manner consistent with an embodiment of the present invention. They may be processed sequentially and / or simultaneously. Any sequence diagrams and / or block diagrams may be used in conjunction with the illustrated embodiments consistent with the present invention. It may contain more or less blocks.

[0143] The terminology used herein is for the purpose of describing particular embodiments only. It is not intended to limit the embodiments of the present invention. The numeral forms "a," "an," and "the" are used in the plural unless the context clearly indicates otherwise. As used herein, the term "comprises" is intended to include forms. "es") and / or "comprising" means the functions described, integers , specifying the presence of steps, operations, elements, and / or components but not one or more other functions , the presence or addition of integers, steps, operations, elements, components, and / or groups thereof It will be further understood that nothing in the detailed description or claims should be construed as excluding The terms "includes" and "having" are used in either of the following cases: "has," "with," "comprised" Wherever "including" or variations thereof are used, such terms shall be construed as including the term "including" or variations thereof. The IFRS 1000 is intended to be as comprehensive as the IFRS 1000 Standards and Regulations (

[0144] While the present invention has been illustrated in its entirety by the description of various embodiments, and these embodiments have been described in considerable detail, it is not the intention of the applicant to limit, or in any way restrict, the scope of the appended claims to such details. Additional advantages and modifications will readily occur to those skilled in the art. Accordingly, the invention in its broader aspects is not limited to the specific details, representative apparatus and methods, and illustrative examples shown and described. Accordingly, departures may be made from such details without departing from the spirit or scope of the applicant's general inventive concept. The technical concepts that can be understood from the above-described embodiments will be described below. [Aspect 1] a localizer configured to detect a position of a first object; a vision device configured to generate a real depth map of a surface near the first object; a controller coupled to the localizer and the vision device, the controller comprising: accessing a virtual model corresponding to the first object; determining a positional relationship between the localizer and the vision device in a common coordinate system; generating a predicted depth map for the vision device based on the detected position of the first object, the virtual model, and the positional relationship; identifying portions of the actual depth map that are inconsistent with the expected depth map; recognizing a second object based on the identified portion; and the controller configured to: A navigation system equipped with. [Aspect 2] The navigation system of aspect 1, wherein the controller is configured to determine the position of the second object relative to the first object in the common coordinate system based on the detected position of the first object, the location of the second object in the real depth map, and the positional relationship. [Aspect 3] The first object defines a target volume of patient tissue to be treated according to a surgical plan, and the controller: determining whether the second object is an obstacle to treating the target volume according to the surgical plan based on the position of the second object relative to the target volume in the common coordinate system and the surgical plan; modifying the surgical plan and / or triggering a notification and / or halting surgical navigation in response to determining that the second object is an obstruction to the surgical plan; 3. The navigation system of embodiment 2, configured to: [Aspect 4] A tracker is rigidly coupled to the first object, and the controller Detecting, via the localizer, a position of the tracker in a first coordinate system specific to the localizer; determining a position of the virtual model in the first coordinate system based on the detected position of the tracker in the first coordinate system and a positional relationship between the tracker and the first object in the first coordinate system; transforming the position of the virtual model in the first coordinate system into a position of the virtual model in the second coordinate system specific to the vision device based on the position of the virtual model in the first coordinate system and a positional relationship between the localizer and the vision device in a second coordinate system; generating the predicted depth map based on the position of the virtual model in the second coordinate system; The navigation system according to any one of aspects 1 to 3, configured to perform the above. [Aspect 5] The controller calculating a difference between the actual depth map and the predicted depth map; determining whether a first section of the difference indicates an absolute depth greater than a threshold depth; In response to determining that the first section of the difference indicates an absolute depth greater than a threshold depth, identifying as the portion a second section of the actual depth map corresponding to the first section of the difference; 5. The navigation system of any one of aspects 1 to 4, configured to identify portions of the actual depth map that are inconsistent with the expected depth map by performing the following steps. [Aspect 6] A navigation system as described in aspect 5, wherein the threshold depth is not zero. [Aspect 7] The controller determining whether the size of the first section is greater than a minimum size threshold; identifying the second section as the portion in response to determining that the size of the first section is greater than the minimum size threshold; The navigation system of aspect 5 or aspect 6, wherein the navigation system is configured to identify the portion of the actual depth map that is inconsistent with the expected depth map by performing the following steps: [Aspect 8] The navigation system of any one of aspects 1 to 7, wherein the controller is configured to recognize the second object based on the identified portion by matching the identified portion with a predetermined profile corresponding to the second object. [Aspect 9] The navigation system of any one of aspects 1 to 8, wherein the portion of the real depth map includes an array of features corresponding to the second object and positioned at a first position in the real depth map, and the controller is configured to track movement of the second object by monitoring whether the array of features moves to a second position different from the first position in an additional real depth map later generated by the visual device. [Aspect 10] A navigation system described in any one of aspects 1 to 9, wherein the controller is configured to generate a virtual boundary corresponding to the second object in the common coordinate system, the virtual boundary providing constraints on movement of a surgical tool. [Aspect 11] The navigation system of any one of aspects 1 to 10, wherein the controller is configured to crop the real depth map to a region of interest based on the virtual model, the detected position of the first object, and the positional relationship between the localizer and the visual device in a common coordinate system, and the controller is configured to compare the cropped real depth maps, thereby comparing the real depth maps. [Aspect 12] The controller projecting a pattern onto a surface within a field of view of the localizer and the vision device; generating location data using the localizer indicating the location of the pattern in a first coordinate system specific to the localizer; receiving a calibration depth map indicative of the projection pattern produced by the vision device; determining a position of the projection pattern in a second coordinate system specific to the vision device based on the calibration depth map; determining the positional relationship between the localizer and the vision device in the common coordinate system based on the position of the pattern in the first coordinate system and the position of the pattern in the second coordinate system; 12. The navigation system according to any one of aspects 1 to 11, wherein the navigation system is configured to perform the above-mentioned operation to identify the positional relationship between the localizer and the visual device in the common coordinate system. [Aspect 13] 13. The navigation system of any one of aspects 1 to 12, wherein the localizer is configured to operate in a first spectral band to detect the position of the first object, and the visual device is configured to operate in a second spectral band to generate the real depth map of the surface in the vicinity of the first object, and the first spectral band is different from the second spectral band. [Aspect 14] A robotic manipulator for use in the navigation system described in any one of aspects 1 to 13, wherein the robotic manipulator supports a surgical tool and comprises a plurality of links and a plurality of actuators configured to move the links to move the surgical tool, and the robotic manipulator is controlled to avoid the second object. [Aspect 15] 1. A method of operating a navigation system including a localizer configured to detect a position of a first object, a vision device configured to generate a real depth map of a surface in the vicinity of the first object, and a controller coupled to the localizer and the vision device, the method comprising: accessing a virtual model corresponding to the first object; determining a positional relationship between the localizer and the vision device in a common coordinate system; generating a predicted depth map for the vision device based on the detected position of the first object, the virtual model, and the positional relationship; identifying portions of the actual depth map that are inconsistent with the expected depth map; recognizing a second object based on the identified portion; and The method comprising: [Aspect 16] The method of aspect 15, further comprising determining a position of the second object relative to the first object in the common coordinate system based on the detected position of the first object, the location of the second object in the real depth map, and the positional relationship. [Aspect 17] the first object defines a target volume of patient tissue to be treated according to a surgical plan; determining whether the second object is an obstacle to treating the target volume according to the surgical plan based on the position of the second object relative to the target volume in the common coordinate system and the surgical plan; modifying the surgical plan and / or triggering a notification and / or halting surgical navigation in response to determining that the second object is an obstruction to the surgical plan; 17. The method of embodiment 16, further comprising: [Aspect 18] a tracker rigidly coupled to the first object; Detecting, via the localizer, a position of the tracker in a first coordinate system specific to the localizer; determining a position of the virtual model in the first coordinate system based on the detected position of the tracker in the first coordinate system and a positional relationship between the tracker and the first object in the first coordinate system; transforming the position of the virtual model in the first coordinate system into a position of the virtual model in the second coordinate system specific to the vision device based on the position of the virtual model in the first coordinate system and a positional relationship between the localizer and the vision device in a second coordinate system; generating the predicted depth map based on the position of the virtual model in the second coordinate system; 18. The method according to any one of aspects 15 to 17, further comprising: [Aspect 19] Identifying portions of the actual depth map that are inconsistent with the expected depth map includes: calculating a difference between the actual depth map and the predicted depth map; determining whether a first section of the difference indicates an absolute depth greater than a threshold depth; in response to determining that the first section of the difference indicates an absolute depth greater than a threshold depth, identifying as the portion a second section of the actual depth map corresponding to the first section of the difference; 19. The method according to any one of aspects 15 to 18, comprising: [Aspect 20] 20. The method of embodiment 19, wherein the threshold depth is not zero. [Aspect 21] Identifying portions of the actual depth map that are inconsistent with the expected depth map includes: determining whether the size of the first section is greater than a minimum size threshold; identifying the second section as the portion in response to determining that the size of the first section is greater than the minimum size threshold; and 21. The method of embodiment 19 or embodiment 20, comprising: [Aspect 22] A method according to any one of aspects 15 to 21, wherein recognizing the second object based on the identified portion includes matching the identified portion with a predetermined profile corresponding to the second object. [Aspect 23] A method according to any one of aspects 15 to 22, wherein the portion of the real depth map includes an array of features corresponding to the second object and positioned at a first position in the real depth map, and the controller is configured to track movement of the second object by monitoring whether the array of features moves to a second position different from the first position in an additional real depth map later generated by the vision device. [Aspect 24] A method according to any one of aspects 15 to 23, further comprising generating a virtual boundary corresponding to the second object in the common coordinate system, the virtual boundary providing a constraint on movement of a surgical tool. [Aspect 25] A method according to any one of aspects 15 to 24, further comprising cropping the real depth map to a region of interest based on the virtual model, the detected position of the first object, and the positional relationship between the localizer and the visual device in a common coordinate system, and comparing the real depth maps comprises comparing the cropped real depth maps. [Aspect 26] Identifying the positional relationship between the localizer and the vision device in the common coordinate system includes: projecting a pattern onto a surface within a field of view of the localizer and the vision device; generating location data using the localizer indicating the location of the pattern in a first coordinate system specific to the localizer; receiving a calibration depth map corresponding to the projection pattern produced by the vision device; determining a position of the projection pattern in a second coordinate system specific to the vision device based on the calibration depth map; determining the positional relationship between the localizer and the vision device in the common coordinate system based on the position of the pattern in the first coordinate system and the position of the pattern in the second coordinate system; 26. The method according to any one of aspects 15 to 25, comprising: [Aspect 27] operating the localizer in a first spectral band to detect the location of the first object; generating the real depth map of the surface near the first object by operating the visual device in a second spectral band, the second spectral band being different from the first spectral band; 27. The method of any one of aspects 15 to 26, further comprising: [Aspect 28] 28. A computer program product comprising: a non-transitory computer-readable medium having stored thereon instructions configured, when executed by one or more processors, to implement the method of any one of aspects 15 to 27.

Claims

1. Surgical tools and a robotic manipulator configured to support and move the surgical tool; a manipulator controller configured to control the robotic manipulator; a localizer configured to detect a position of a first object in a surgical environment; a vision device configured to generate a real depth map of a surface near the first object; a navigation controller coupled to the manipulator controller, the localizer, and the vision device; Equipped with The navigation controller accessing a virtual model corresponding to the first object; determining a positional relationship between the localizer and the vision device in a common coordinate system; generating an expected depth map, which is a depth map of the first object that is expected to be generated by the vision device, based on the position of the first object detected by the localizer, the virtual model, and the positional relationship; performing a comparison between the actual depth map and the expected depth map; identifying portions of the actual depth map that are inconsistent with the expected depth map based on the comparison; and recognizing a second object based on the portion; and communicating information about the second object to the manipulator controller; configured to: The manipulator controller is configured to control the robotic manipulator to avoid the second object based on the information communicated from the navigation controller.

2. The navigation controller the position of the first object detected by the localizer; and the location of the second object in the real depth map generated by the vision device; and the positional relationship between the localizer and the vision device in the common coordinate system; The surgical system of claim 1 , configured to determine a position of the second object relative to the first object in the common coordinate system based on:

3. The first object defines a target volume of patient tissue to be treated according to a surgical plan, and the navigation controller determining, based on the position of the second object relative to the target volume in the common coordinate system and the surgical plan, whether the second object is an obstacle to a planned trajectory of a surgical tool for treating the target volume according to the surgical plan; modifying the surgical plan and / or triggering a notification and / or halting surgical navigation in response to determining that the second object is the obstacle; The surgical system of claim 2 , configured to:

4. a tracker coupled to the first object, and a positional relationship between the first object and the tracker is fixed; and the navigation controller Detecting, via the localizer, a position of the tracker in a first coordinate system specific to the localizer; determining a position of the virtual model in the first coordinate system based on the position of the tracker in the first coordinate system and a positional relationship between the tracker and the first object in the first coordinate system; transforming the position of the virtual model in the first coordinate system into a position of the virtual model in the second coordinate system based on the position of the virtual model in the first coordinate system and a positional relationship between the localizer and the visual device in a second coordinate system specific to the visual device; generating the predicted depth map based on the position of the virtual model in the second coordinate system; The surgical system according to any one of claims 1 to 3, configured to perform the following:

5. The navigation controller generating a difference depth map indicating a difference between the depth of the actual depth map and the depth of the predicted depth map for each pair of corresponding image components in the actual depth map and the predicted depth map; determining a first section of the difference depth map that exhibits an absolute depth greater than a threshold depth; identifying a second section of the actual depth map as the portion, the second section corresponding to the first section of the differential depth map; The surgical system of any one of claims 1 to 4, wherein the surgical system is configured to perform a comparison between the actual depth map and the expected depth map, and to identify the portion based on the comparison.

6. The navigation controller determining whether a size of the first section of the differential depth map is greater than a minimum size threshold; identifying the second section of the real depth map as the portion in response to determining that the size of the first section is greater than the minimum size threshold; The surgical system of claim 5 , further configured to:

7. The surgical system of any one of claims 1 to 6, wherein the navigation controller is configured to identify an object associated with a predetermined profile that matches the portion as the second object, thereby recognizing the second object based on the portion.

8. 8. The surgical system of claim 1, wherein the navigation controller is configured to track movement of the second object by monitoring whether a position of a feature array corresponding to the second object depicted in the real depth map changes in an additional real depth map subsequently generated by the vision device.

9. The navigation controller generating a virtual boundary corresponding to a surface of the second object in the common coordinate system; determining whether a planned trajectory of a surgical tool for treating a target volume of patient tissue to be treated according to a surgical plan collides with the virtual boundary; The surgical system according to any one of claims 1 to 8, configured to perform the following:

10. The navigation controller cropping the real depth map to a region of interest based on the virtual model, the position of the first object detected by the localizer, and the positional relationship between the localizer and the vision device in a common coordinate system; configured to compare the cropped actual depth map with the predicted depth map, thereby comparing the actual depth map with the predicted depth map; The surgical system according to any one of claims 1 to 9, configured to perform the following:

11. The navigation controller projecting a pattern of light onto a surface within a field of view of the localizer and the vision device; generating, using the localizer, location data indicative of a location of the pattern in a first coordinate system specific to the localizer; receiving a calibration depth map indicative of the pattern produced by the vision device; locating the pattern in a second coordinate system specific to the vision device based on the calibration depth map; determining the positional relationship between the localizer and the vision device in the common coordinate system based on the position of the pattern in the first coordinate system and the position of the pattern in the second coordinate system; The surgical system according to any one of claims 1 to 10, wherein the surgical system is configured to perform the following steps to identify the positional relationship between the localizer and the visual device in the common coordinate system.

12. the localizer is configured to detect the location of the first object by detecting light in a first spectral band; the vision device is configured to detect light in a second spectral band to generate the real depth map of the surface near the first object; The surgical system of any one of claims 1 to 11, wherein the first spectral band is different from the second spectral band.

13. The surgical system of any one of claims 1 to 12, wherein the robotic manipulator comprises a plurality of links and a plurality of actuators configured to move the plurality of links to move the surgical tool, and the manipulator controller controls the plurality of actuators to move the plurality of links to avoid the second object.

14. A surgical system described in any one of claims 1 to 13, wherein the visual device is directly attached to the robotic manipulator.

15. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, obtaining a position of a first object in a surgical environment from a localizer; obtaining a real depth map of a surface near the first object from a vision device; accessing a virtual model corresponding to the first object; determining a positional relationship between the localizer and the vision device in a common coordinate system; generating an expected depth map, which is a depth map of the first object that is expected to be generated by the vision device, based on the position of the first object detected by the localizer, the virtual model, and the positional relationship; performing a comparison between the actual depth map and the expected depth map; identifying portions of the actual depth map that are inconsistent with the expected depth map based on the comparison; and recognizing a second object based on the portion; and 10. A computer-readable medium configured to:

Images