Systems and methods for identifying and tracking physical objects during a robotic surgical procedure

Abstract

A system and method for identifying and tracking physical objects during robotic surgery are disclosed. A navigation system and method for tracking physical objects near a target site during surgery are provided. The navigation system includes a robotic device and instruments attached to the robotic device. A vision device is attached to the robotic device or instruments and generates a visual dataset. The visual dataset is captured from multiple perspectives of the physical object. A computational system associates virtual objects with physical objects based on one or more features of identifiable physical objects in the visual dataset. The virtual objects at least partially define the virtual boundaries of the instruments.

Description

[0001] This application is a divisional application of Chinese invention patent application No. 201780031879.8, filed on May 23, 2017, entitled "System and method for identifying and tracking physical objects during robotic surgery".

[0002] Related applications

[0003] This application claims priority and benefit to U.S. Provisional Patent Application No. 62 / 340,176, filed May 23, 2016, the entire disclosure and contents of which are hereby incorporated by reference. Technical Field

[0004] This disclosure generally relates to systems and methods for identifying and tracking physical objects during robotic surgical procedures. Background Technology

[0005] Navigation systems help users precisely locate objects. For example, navigation systems are used in industrial, aerospace, and medical applications. In the medical field, navigation systems help surgeons precisely place surgical instruments relative to a target site on the patient. The target site typically requires some form of treatment (such as tissue resection). Traditional navigation systems use a locator that works in conjunction with a tracker to provide positional and / or orientation data associated with the surgical instruments and the target site, such as the amount of bone to be removed. The locator is typically positioned such that it has the tracker's field of view. The tracker is fixed to the surgical instruments and the patient to move in response to both. Trackers attached to the patient are often attached to the bone being treated, thus maintaining a rigid relationship relative to the target site due to the rigidity of the bone. By using separate trackers on the surgical instruments and the patient, the treatment end of the surgical instruments can be precisely positioned at the target site.

[0006] Typically, retractors or other physical objects are located near target sites that should be avoided during surgery. These retractors or other physical objects can be tracked using a separate tracker in the same way as surgical instruments, but adding trackers to retractors and other physical objects significantly increases the cost and complexity of the navigation system, particularly by increasing the number of objects the locator will be tracking. Furthermore, because these physical objects are often movable relative to the trackers associated with the instruments and the patient, they are not easily referenced by these trackers. It has been proposed to use object recognition technology in images captured by cameras attached to or otherwise fixed relative to the locator to track these additional physical objects. However, this approach can be computationally expensive and difficult.

[0007] During robotic surgery, especially when the robotic device is operating autonomously, it is difficult to avoid physical objects when the navigation system cannot identify the positions of all physical objects near the target site. As a result, the robotic device is currently controlled to monitor for collisions with these physical objects and is shut down if a collision occurs (e.g., relying on feedback from force / torque sensors indicating a collision). However, it is undesirable to wait until a collision occurs before shutting down the robotic device, as this could result in tool damage or potential harm to the patient from debris generated by such a collision (e.g., when a rotary drill or saw hits a retractor). Collisions with physical objects can delay the surgical procedure. This delay can prolong the time the patient receives general anesthesia or otherwise increase the risks associated with the surgical procedure.

[0008] Therefore, there is a need in the art for systems and methods for identifying and tracking physical objects during robotic surgery. Summary of the Invention

[0009] In one embodiment, a navigation system is provided for tracking a physical object near a target site during a surgical procedure. The navigation system includes a robotic device and an instrument attachable to the robotic device. A vision device is attached to either the robotic device or the instrument, such that the vision device can move with the robotic device. The vision device is configured to generate a visual dataset. The visual dataset is captured from multiple perspectives of a physical object that the vision device, moving with multiple degrees of freedom while the robotic device is moving, can realize. A computing system communicates with the vision device. The computing system has at least one processor configured to associate virtual objects with physical objects based on one or more features of identifiable physical objects in the visual dataset. The virtual objects at least partially define virtual boundaries of the instrument.

[0010] In yet another embodiment, a method for tracking a physical object is provided. The method includes generating a visual dataset in response to movement of a vision device caused by movement of a robotic device. Due to this movement, the visual dataset is captured from multiple perspectives of the physical object. The method also includes associating a virtual object with the physical object based on one or more features of the identifiable physical object in the visual dataset. The virtual object defines virtual boundaries of the device.

[0011] This system and method offer several advantages. For example, by moving the vision device relative to a physical object that includes one or more of the aforementioned features, the computational system is able to both identify and locate the physical object and track the movement of the instrument relative to it. In some cases, a single visual dataset (e.g., a single image) captured by the vision device may be sufficient to estimate the position of the physical object, and subsequent visual datasets can be used to improve the tracking results. This is useful for avoiding physical objects while treating a target site with an instrument during a surgical procedure. Furthermore, by attaching the vision device to a moving robotic device or instrument, a lower-cost vision device can be utilized by acquiring multiple visual datasets from different perspectives of the physical object, such as multiple video frames from different perspectives. As a result, these physical objects can be tracked without the need for separate, expensive trackers. Attached Figure Description

[0012] The advantages will be readily apparent, as they become even more readily understood when considered in conjunction with the accompanying drawings and by referring to the following specific embodiments.

[0013] Figure 1 It is a three-dimensional diagram of a robotic surgical system that includes robotic devices, locators, and vision devices.

[0014] Figure 2 This is a schematic diagram of the control system used to control the robotic surgical system.

[0015] Figure 3 It is a three-dimensional diagram of the coordinate system used in robotic surgical systems.

[0016] Figure 4 It is a diagram of the surgical site when a physical object is observed using a visual device.

[0017] Figure 5 It is an example of a characteristic on a physical object.

[0018] Figure 6 It is another example of a characteristic on a physical object.

[0019] Figure 7 It is an example of a feature group that includes encoded features.

[0020] Figure 8 It is an example of a feature set that includes encoded features.

[0021] Figure 9 It is a diagram of a virtual object associated with a physical object.

[0022] Figure 10 It is a flowchart of the steps performed by the method. Detailed Implementation

[0023] like Figure 1 The diagram illustrates a system 20 for treating a patient 22. System 20 is shown in a surgical setting (such as an operating room in a medical facility). In the illustrated embodiment, system 20 includes a processing station 24 and a guiding station 26. The guiding station 26 is configured to track the movement of various objects in the operating room. These objects include, for example, surgical instruments 30, the patient's femur F, and the patient's tibia T. The guiding station 26 tracks these objects to display their relative positions and orientations to the user, and in some cases, to control or constrain the movement of the surgical instruments 30 relative to a target site (such as a femoral target site TS). The surgical instruments 30 are shown as part of the processing station 24.

[0024] The guidance station 26 includes a navigation cart assembly 32 that houses the navigation computer 34. The navigation interface is operatively in communication with the navigation computer 34. The navigation interface includes a first display 36 adapted to be located outside a sterile area and a second display 38 adapted to be located within a sterile area. The displays 36 and 38 are adjustablely mounted to the navigation cart assembly 32. A first input device 40 and a second input device 42 (such as a keyboard and mouse) can be used to input information into the navigation computer 34 or otherwise select / control certain aspects of the navigation computer 34. Other input devices, including touchscreens (not shown) or voice-activated inputs, are contemplated.

[0025] The locator 44 communicates with the navigation computer 34. In the illustrated embodiment, the locator 44 is an optical locator and includes a locator camera unit 46. The locator camera unit 46 has a housing 48 that accommodates one or more optical position sensors 50. In some embodiments, at least two, preferably three or more, optical sensors 50 are employed. The optical sensors 50 may be three separate charge-coupled devices (CCDs). In one embodiment, three one-dimensional CCDs are employed. It should be understood that in other embodiments, separate locator camera units may also be arranged around the operating room, each camera unit having a separate CCD or two or more CCDs. The CCDs detect infrared signals. Additionally, the locator 44 may employ different modes and may be an electromagnetic locator, an RF locator, an ultrasonic locator, or any other conventional locator capable of tracking an object.

[0026] The locator camera unit 46 is mounted on an adjustable arm to position the optical sensor 50 within the ideal, unobstructed field of view of the tracker, as discussed below. In some embodiments, the locator camera unit 46 can be adjusted to at least one degree of freedom by rotating about a rotary joint. In other embodiments, the locator camera unit 46 can be adjusted to approximately two or more degrees of freedom.

[0027] The locator camera unit 46 includes a locator camera controller 52 that communicates with the optical sensor 50 to receive signals from the optical sensor 50. The locator camera controller 52 communicates with the navigation computer 34 via a wired or wireless connection (not shown). One such connection could be an IEEE 1394 interface, a serial bus interface standard for high-speed communication and synchronous real-time data transmission. The connection could also use company-specific protocols. In other embodiments, the optical sensor 50 communicates directly with the navigation computer 34.

[0028] Position and orientation signals and / or data are transmitted to navigation computer 34 for tracking the object. The navigation cart assembly 32, displays 36 and 38, and locator camera unit 46 may be as described in U.S. Patent No. 7,725,162, entitled "Surgery System," issued May 25, 2010, to Malackowski et al., which is hereby incorporated by reference.

[0029] The navigation computer 34 has displays 36 and 38, a central processing unit (CPU) and / or other processor 62, memory (not shown), and storage devices (internal and external, not shown) necessary for performing the functions described herein. The navigation computer 34 is loaded with software as described below. This software converts signals received from the locator camera unit 46 into locator data, which represents the position and orientation of the object being tracked by the locator. The navigation computer 34 is capable of wired or wireless communication with a computer network (such as a local area network (LAN) and / or the Internet). One or more data interfaces, such as a universal serial bus (USB) interface or means for reading data carriers such as CD-ROMs or SD cards, may be provided for the navigation computer 34. Internal or external storage devices, or both, may be configured to store image data of patient images captured by the imaging device. Alternatively, or additionally, this image data may also be received (e.g., downloaded) via a computer network. Furthermore, internal or external storage devices, or both, may be configured to store various calibration data / information entries described herein. This calibration data / information constitutes the prior knowledge of system 20, and various examples of calibration data will be described in more detail below. As will be understood, alternatively, or additionally, the prior knowledge of system 20 may include other information entries.

[0030] The guiding station 26 can operate in conjunction with a plurality of tracking devices 54, 56, 58, also referred to herein as trackers. In the illustrated embodiment, one tracker 54 is securely attached to the patient's femur F and another tracker 56 is securely attached to the patient's tibia T. Trackers 54 and 56 are securely attached to segments of bone. Trackers 54 and 56 can be attached to the femur F and tibia T as shown in U.S. Patent No. 7,725,162, which is hereby incorporated by reference. Trackers 54 and 56 can also be mounted as shown in U.S. Patent Application Publication No. 2014 / 0200621, filed January 16, 2014, entitled "Navigation Systems and Methods for Indicating and Reducing Line-of-Sight Errors," which is hereby incorporated by reference. In other embodiments, trackers 54 and 56 can be mounted to other tissues of the anatomical structure.

[0031] The instrument tracker 58 is securely attached to the surgical instrument 30. The instrument tracker 58 may be integrated into the surgical instrument 30 during manufacturing or may be installed separately on the surgical instrument 30 during preparation for the surgical procedure. The treatment end of the surgical instrument 30 tracked by the instrument tracker 58 may include a rotary drill, an electroablation tip, an ultrasonic tip, a sagittal saw blade, or other types of treatment elements.

[0032] Trackers 54, 56, and 58 may be powered by internal batteries or may have leads that receive power through a navigation computer 34, which, like the locator camera unit 46, preferably receives external power.

[0033] In the illustrated embodiment, the surgical instrument 30 is attached to a manipulator 66 of the processing station 24. The manipulator 66 may also be referred to as a robotic device or robotic arm. This arrangement is illustrated in U.S. Patent No. 9,119,655, entitled “Surgical Manipulator Capable of Controlling a Surgical Instrument in Multiple Modes,” the disclosure of which is hereby incorporated by reference. The surgical instrument 30 can be any surgical instrument (also referred to as a tool) that can be used to perform medical / surgical procedures. The surgical instrument 30 can be a lamellar instrument, an electrosurgical instrument, an ultrasonic instrument, a reamer, an impactor, a sagittal saw, or other instrument. In some embodiments, multiple surgical instruments are used to treat a patient, wherein each surgical instrument is individually tracked by a positioner 44.

[0034] The optical sensor 50 of the locator 44 receives optical signals from trackers 54, 56, and 58. In the illustrated embodiment, trackers 54, 56, and 58 are active trackers. In this embodiment, each tracker 54, 56, and 58 has at least three active tracking elements or markers for transmitting optical signals to the optical sensor 50. Active markers may be, for example, light-emitting diodes or LEDs 60 that transmit light (such as infrared light). The sampling rate of the optical sensor 50 is preferably 100 Hz or higher, more preferably 300 Hz or higher, and most preferably 500 Hz or higher. In some embodiments, the sampling rate of the optical sensor 50 is 8000 Hz. The sampling rate is the rate at which the optical sensor 50 receives optical signals from the sequentially emitting LEDs 60. In some embodiments, the optical signals from the LEDs 60 are emitted at different rates for each tracker 54, 56, and 58.

[0035] refer to Figure 2 Each of the LEDs 60 is connected to a tracker controller 61, which is housed within a housing of associated trackers 54, 56, and 58 that transmit data to / receive data from the navigation computer 34. In one embodiment, the tracker controller 61 transmits approximately several megabytes per second of data via a wired connection to the navigation computer 34. In other embodiments, a wireless connection may be used. In these embodiments, the navigation computer 34 has a transceiver (not shown) for receiving data from the tracker controller 61.

[0036] In other embodiments, trackers 54, 56, and 58 may have passive markers (not shown), such as reflectors that reflect light emitted from the locator camera unit 46. The optical sensor 50 then receives the reflected light. Active and passive arrangements are well known in the art.

[0037] In some embodiments, trackers 54, 56, 58 further include a gyroscope sensor and an accelerometer, such as the tracker shown in U.S. Patent No. 9,008,757, entitled "Navigation System Including Optical and Non-Optical Sensor," issued to Wu on April 14, 2015, which is hereby incorporated by reference.

[0038] The navigation computer 34 includes a navigation processor 62. It should be understood that the navigation processor 62 may include one or more processors for controlling the operation of the navigation computer 34. The processor can be any type of microprocessor or multiprocessor system. The term "processor" is not intended to limit the scope of any embodiment to a single processor.

[0039] The locator camera unit 46 receives optical signals from the LEDs 60 of trackers 54, 56, and 58, and outputs signals relating to the position of the LEDs 60 of trackers 54, 56, and 58 relative to the locator 44 to the navigation processor 62. Based on the received optical signals (and in some embodiments, non-optical signals), the navigation processor 62 generates data indicating the relative position and orientation of trackers 54, 56, and 58 relative to the locator 44, such as through known triangulation methods. In some embodiments, the data is generated by the locator camera controller 52 and then transmitted to the navigation computer 34.

[0040] Prior to initiating the surgical procedure, additional data is loaded into the navigation processor 62. Based on the position and orientation of trackers 54, 56, and 58, and the previously loaded data, the navigation processor 62 determines the position of the treatment end of the surgical instrument 30 (e.g., the centroid of the surgical drill) and the orientation of the surgical instrument 30 relative to the target site (such as the femoral target site TS) against which its treatment end will be applied. In some embodiments, the navigation processor 62 forwards this data to the manipulator controller 64. The manipulator controller 64 can then use this data to control the manipulator 66 as described in U.S. Patent No. 9,119,655, entitled “Surgical Manipulator Capable of Controlling a Surgical Instrument in Multiple Modes,” which is hereby incorporated by reference. In one embodiment, the manipulator 66 is controlled to remain within a virtual boundary set by the surgeon. In the embodiments described herein, such a virtual boundary defines the amount of material of the femur F to be removed by the surgical instrument 30. Thus, the virtual boundary is the boundary within which the treatment end of the surgical instrument 30 remains. The manipulator 66 can be controlled to operate in a manual mode in which the user grasps and manipulates the instrument 30 to move it, or autonomously, as described in U.S. Patent No. 9,119,655 entitled “Surgical Manipulator Capable of Controlling a Surgical Instrument in Multiple Modes”, which is hereby incorporated by reference.

[0041] The navigation processor 62 also generates image signals indicating the relative position of the treatment end to the target site. These image signals are applied to displays 36, 38. Based on these signals, displays 36, 38 generate images that allow surgeons and staff to substantially view the relative position of the treatment end to the target site. In most cases, the image illustrates the treatment end relative to one target site at a time. For example, during surgery where the femoral F and tibial T are treated simultaneously, the relative position of the femoral target site TS and the treatment end of the surgical instrument 30 to the femoral target site TS can be visually illustrated while material is being removed from the femoral F. Similarly, when the user has finished removing material from the femoral F and is preparing to remove material from the tibial T, displays 36, 38 may only illustrate the placement of the treatment end of the surgical instrument 30 relative to the target site associated with the tibial T.

[0042] refer to Figure 3 Generally, object tracking is performed using the reference locator coordinate system LCLZ. The locator coordinate system LCLZ has an origin and orientation (a set of x, y, and z axes). During this process, one objective is to maintain the locator coordinate system LCLZ in a known position. An accelerometer (not shown) mounted on the locator camera unit 46 can be used to track sudden or accidental movement of the locator coordinate system LCLZ, such as if the locator camera unit 46 is accidentally bumped by a surgical personnel.

[0043] Each tracker 54, 56, 58 and the object being tracked also has its own coordinate system separate from the locator coordinate system LCLZ. For example, trackers 54, 56, 58 have a bone tracker coordinate system BTRK1, a bone tracker coordinate system BTRK2, and an instrument tracker coordinate system TLTR.

[0044] In the illustrated embodiment, the guidance station 26 monitors the position of the patient's femur (F) and tibia (T) by monitoring the position of bone trackers 54 and 56, which are securely attached to the bone. The femoral coordinate system is FBONE and the tibial coordinate system is TBONE; these are the coordinate systems of the bones to which the bone trackers 54 and 56 are securely attached.

[0045] Virtual objects define the target sites to be treated by surgical instruments 30. In the illustrated embodiment, the femoral target site TS is associated with the femoral F. Of course, several other target sites, such as the tibial target site T, are also possible, each defined by its own individual virtual object. The virtual object representing the target site is pre-operated by the user to define the amount of material to be treated, the trajectory of the surgical instruments 30, the plane to be cut by the surgical instruments 30, the hole to be drilled, etc. In the illustrated embodiment, the virtual object VB (see Figure 9This limits the amount of material to be removed from the femur. In some cases, virtual objects are set up or reset intraoperatively, i.e., during the surgical procedure. It should be understood that although the descriptions set forth herein pertain to orthopedic surgical procedures, the systems and methods described herein are equally applicable to any type of surgical procedure.

[0046] Before the start of the surgical procedure, preoperative images of the anatomical structures of interest, such as the femoral F and tibial T (or, in other embodiments, other tissues or structures), are generated. These images can be MRI, radiographic, or computed tomography (CT) scans based on the patient's anatomy. These images are used to develop virtual models of the anatomical structures of interest, such as the femoral F and tibial T and / or other anatomical structures to be treated by the surgical instruments 30. These virtual models can also be generated intraoperatively, such as by capturing points on the surfaces of the femoral F and tibial T and / or other anatomical structures to be treated using a navigation guide or other suitable device. These points are then compiled and the gaps between the points are filled to generate the virtual model. This set of points can also be combined with a gross bone model to allow the gross bone model to deform and better match the anatomical structures of interest.

[0047] Typically, a virtual model is a 3D model that includes data representing the entire anatomical structure being treated or at least a portion of the anatomical structure to be treated, as well as data representing a virtual object defining the target site. In the illustrated embodiment, the virtual model VM of the femur is a 3D model that includes model data representing a portion of the femur F and a virtual object VB (see [link to documentation]). Figure 9 The virtual object VB defines the target site TS and the amount of material removed from the femur F during the surgical procedure. Virtual objects can be defined within the virtual model and can be represented as mesh surfaces, constructed solid geometry (CSG), voxels, etc., using other virtual object representation techniques.

[0048] Preoperative images and / or virtual models are mapped to the femoral coordinate system (FBONE) and the tibial coordinate system (TBONE) using methods known in the art. These preoperative images and / or virtual models are then fixed in the femoral coordinate system (FBONE) and the tibial coordinate system (TBONE). Alternatively, treatment plans can be developed in the operating room using kinematic studies, bone tracking, and other methods, as an alternative to taking preoperative images. These methods can also be used to generate the previously described 3D virtual models.

[0049] During the initial phase of the surgical procedure described herein, bone trackers 54 and 56 are securely fixed to the patient's bones. The pose (position and orientation) of coordinate systems FBONE and TBONE are mapped to coordinate systems BTRK1 and BTRK2, respectively. In one embodiment, a tracker PT with its own tracker (see [link to document]) can be used. Figure 1The guide device P disclosed in U.S. Patent No. 7,725,162 to Malackowski et al., which is hereby incorporated by reference (see also...) Figure 1 The femoral coordinate system FBONE and the tibial coordinate system TBONE are registered to the bone tracker coordinate systems BTRK1 and BTRK2, respectively. Given a fixed relationship between a bone and its trackers 54 and 56, the position and orientation of the femur F and tibia T in the femoral coordinate system FBONE and the tibial coordinate system TBONE can be transformed into the bone tracker coordinate systems BTRK1 and BTRK2. Therefore, the locator camera unit 46 can track the femur F and tibia T by tracking trackers 54 and 56. This posture description data is stored in a memory integrated with both the manipulator controller 64 and the navigation processor 62.

[0050] The treatment end of the surgical instrument 30 (also referred to as the distal end of the energy applicator) has its own coordinate system, EAPP. For example, the origin of the EAPP coordinate system can represent the center of mass of the surgical cutting drill. Before the start of the surgical procedure, the orientation of the EAPP coordinate system is fixed to the orientation of the instrument tracker coordinate system, TLTR. Therefore, the orientations of these coordinate systems, EAPP and TLTR, relative to each other are determined. The orientation description data is stored in a memory integrated with the manipulator controller 64 and the navigation processor 62.

[0051] refer to Figure 2 The positioning engine 100 is a software module that can be considered part of the navigation computer 34. The components of the positioning engine 100 run on the navigation processor 62. The positioning engine 100 can run on the manipulator controller 64 and / or the navigation processor 62.

[0052] The positioning engine 100 receives optical-based signals from the locator camera controller 52 and, in some embodiments, non-optical-based signals from the tracker controller 61 as input. Based on these signals, the positioning engine 100 determines the poses of the bone tracker coordinate systems BTRK1 and BTRK2 in the locator coordinate system LCLZ. Based on the same signals received for the device tracker 58, the positioning engine 100 determines the pose of the device tracker coordinate system TLTR in the locator coordinate system LCLZ.

[0053] The positioning engine 100 forwards signals representing the posture of trackers 54, 56, and 58 to the coordinate transformer 102. The coordinate transformer 102 is a software module running on the navigation processor 62. The coordinate transformer 102 references data defining the relationship between preoperative images and / or virtual models of the patient and the bone trackers 54 and 56. The coordinate transformer 102 also stores data indicating the posture of the treatment end of the surgical instrument 30 relative to the instrument tracker 58. If separate from the virtual model, the coordinate transformer 102 also references data defining the virtual object.

[0054] During the surgical procedure, coordinate transformer 102 receives data on the relative pose of the indicator trackers 54, 56, and 58 with the locator 44. Based on this data and previously loaded data, coordinate transformer 102 generates data on the relative position and orientation of the indicator coordinate system EAPP, the bone coordinate systems FBONE and TBONE, and the locator coordinate system LCLZ.

[0055] As a result, coordinate transformer 102 generates data indicating the position and orientation of the treatment end of surgical instrument 30 relative to the target site against which the treatment end is applied. Image signals representing this data are forwarded to displays 36, 38, allowing surgeons and staff to view the information. In some embodiments, other signals representing this data may be forwarded to manipulator controller 64 to guide corresponding movements of manipulator 66 and surgical instrument 30. Thus, the data also indicates the virtual position of the treatment end of surgical instrument 30, which may also be modeled as a separate virtual object (e.g., virtual tool object VI) relative to other virtual objects.

[0056] Refer back Figure 1 The guidance station 26 also includes a vision device 72. In the illustrated embodiment, the vision device is mounted on the surgical instrument 30. In other embodiments, the vision device 72 may be mounted on a robotic arm, such as the distal end of the robotic arm. The vision device 72 is preferably positioned such that it has an unobstructed field of view of the target area. The vision device 72 has a vision controller 73 operably communicatively communicating with the navigation computer 34 (see...). Figure 2 The vision device 72 may also be referred to as an imaging device or a digital imaging device. The vision device 72 may include a camera 160, which has a housing 76 and supports one or more image sensors 78 (see [link to image sensor 78]). Figure 2 The image sensor 78 can be a CMOS sensor or other suitable sensor.

[0057] The navigation computer 34 communicates with the vision controller 73 to receive visual data sets from the camera 160. The visual data sets are provided in the visual coordinate system (VIS) (see [link]). Figure 3As the camera 160 moves relative to the patient, the visual dataset can be a set of data points in the visual coordinate system (VIS) captured by the camera 160. These data points are defined by x, y, and z coordinates. These data points can be saved or stored as a visual data file.

[0058] like Figure 4 As shown, in the vicinity of the target site, there are physical objects other than the femur F, tibia T, and surgical instrument 30. These objects may include retractors, irrigation / suction tools, surgical guides, skin around the incision, or other types of physical objects. In the embodiments described herein, navigation computer 34 and / or manipulator controller 64 identify and track these physical objects so that system 20 can determine the relative positions of surgical instrument 30, femur F, tibia T, and all physical objects near the target site, for example, so that surgical instrument 30 can avoid physical objects during the surgical procedure. In other embodiments, it may be desirable to identify and track physical objects so that surgical instrument 30 can engage one or more of the physical objects in certain situations, such as when the physical object is a tool guide. For illustrative purposes, the physical object shown is retractor 162.

[0059] refer to Figure 4 Each physical object includes multiple features defined in a feature set 170, enabling the physical objects to be identified and tracked by the navigation computer 34 and / or manipulator controller 64 via camera 160 during the surgical procedure. Feature set 170 includes multiple features identifiable in the visual dataset captured by camera 160. For identification purposes, pattern recognition capabilities may be provided by the navigation computer 34 and / or manipulator controller 64. System 20 has prior knowledge of the arrangement, encoding, or other characteristics of the features to be detected.

[0060] One or more of these features may be active markings (e.g., emitting radiation to be detected by camera 160). Alternatively, one or more features may be passive markings. Passive markings may have reflective or non-reflective properties. Passive markings can be implemented by printing, stickers, etc., on any rigid (e.g., flat) or flexible substrate of a physical object (e.g., on or near the patient's skin around an incision or other location). Features are also defined by a surface roughness that may be generated by a coating on the surface of the physical object or by the physical object itself. System 20 has prior knowledge of the features (e.g., in the form of calibration information). For several different types of physical objects, prior knowledge may involve one or more of the feature encoding scheme and the position of the features relative to each other. For example, previously known features may be passive markings stamped or printed on a substrate of the physical object, or alternatively, active markings. In the case of using printed stickers or other passive markings around an incision, previously known features may be understood as being placed in a ring or suitably understood as being in other locations associated with the incision opening, such that instrument 30 can be controlled to avoid the skin and other tissue surrounding the opening.

[0061] Camera 160 is configured to acquire visual datasets from two or more different perspectives, such that each visual dataset includes at least some features from feature set 170. Movement of camera 160 is caused by movement of the robotic device and / or instrument 30 relative to the patient during the acquisition of the visual datasets. This movement can be caused by manual manipulation of the robotic device and / or instrument 30 or by autonomous movement of the robotic device and / or instrument 30. For example, camera 160 can be implemented as a video camera capable of providing visual datasets in the form of a continuous video data stream (e.g., video frames). In one variation, camera 160 is rigidly mounted to instrument 30 such that during autonomous movement of the robotic device, camera 160 can move with instrument 30 under the action of the robotic device. In other variations, camera 160 is rigidly mounted to instrument 160 to move with instrument 30 by means of manual manipulation of the robotic device and / or instrument 30. When mounted on surgical instrument 30, camera 160 has a field of view including the physical object and the patient surface targeted by surgical instrument 30. For example, when the surgical instrument 30 has a longitudinal axis pointing towards the patient, the field of vision can extend along the longitudinal axis of the surgical instrument 30.

[0062] It should be understood that by integrating camera 160 into surgical instrument 30, the visual coordinate system (VIS) of camera 160 can be easily calibrated to the instrument tracker coordinate system (TLTR). This calibration can occur during manufacturing, for example, using calibration data determined during manufacturing, or it can be calibrated using conventional calibration methods before the start of the surgical procedure. Therefore, the pose of the visual coordinate system (VIS) relative to the locator coordinate system (LCLZ) can be determined based on the aforementioned transformation method and the use of the instrument tracker 58 associated with surgical instrument 30. Consequently, the visual coordinate system (VIS) can also be transformed to the locator coordinate system (LCLZ) and vice versa. The pose description data is stored in a memory integrated with the manipulator controller 64 and the navigation processor 62.

[0063] In other embodiments, such as where camera 160 is mounted on a robotic device rather than surgical instrument 30, camera 160 may have a vision device tracker (not shown) rigidly mounted on housing 76 to establish a relationship between the visual coordinate system (VIS) and the locator coordinate system (LCLZ). For example, using preloaded data defining the relationship between the coordinate system of the vision device tracker and the visual coordinate system (VIS), coordinate transformer 102 can transform the visual coordinate system (VIS) to the locator coordinate system (LCLZ) based on the position and orientation of the vision device tracker in the locator coordinate system (LCLZ). Alternatively, if the robotic device is tracked separately in the locator coordinate system (LCLZ) by means of a connector encoder and a robot base tracker (not shown) attached to the base of the robotic device, camera 160 can be associated with the robot base tracker (e.g., by means of a calibration step), thus eliminating the need for a separate tracker on camera 160. As long as the base of the robotic device does not move and the robot base tracker is visible, locator 44 can determine the position and orientation of camera 160.

[0064] The navigation computer 34 can be configured to store the visual dataset received from the camera 160 in an external or internal storage device. As mentioned above, these video datasets can be received as a video data stream that is at least temporarily stored for processing by the navigation processor 62. For example, this processing may include pattern recognition for identifying (e.g., locating and decoding) one or more features in the received visual dataset.

[0065] In one embodiment, navigation processor 62 uses pattern recognition technology to first identify multiple features in the visual dataset and determine their coordinates in the visual coordinate system (VIS) (e.g., in the form of keypoint coordinates). A projection model of camera 160, stored as calibration data, can be used to determine the position of camera 160 relative to one or more features identified in the visual dataset provided by camera 160 (see, for example, U.S. Patent Application Publication No. 2008 / 0208041A1, which is hereby incorporated by reference). Transformation parameters under a specific projection model can be provided by the respective camera manufacturer or by the distributor of system 20. These parameters can also be estimated using a field calibration fixture or normalized for a specific camera type. In some implementations, camera 160 itself (e.g., in real-time, depending on the currently selected zoom level) can provide the transformation parameters via a suitable interface.

[0066] Additionally, information relating to feature set 170 is provided as calibration data, for example, to the internal storage device of navigation computer 34. This information may include the relative positions of features and / or any feature encoding scheme used in the application. Based on the known relative positions of features and (i.e., projected) relative positions of features in the visual dataset (e.g., images) captured by camera 160 (in the associated visual coordinate system VIS), the feature coordinates can be determined from the corresponding visual coordinate system VIS toward any reference system in which the feature coordinates are provided (such as the physical object coordinate system POCS of the physical object of interest, see [reference]). Figure 3 The stereo back projection of the camera 160 is used to determine the transformation parameters of other transformations (e.g., in real time). This is indicated by the transformation T1 of the camera 160. The transformation parameters for the transformation T1 of the camera 160 are calculated by solving for each independent feature j in the system as follows:

[0067] M j,160 =T2·T1 -1 ·M j,cal ,

[0068] M j,160 Let J be an imaging feature j in the visual dataset (e.g., video frames) of camera 160, having coordinates relative to the visual coordinate system VIS. M is provided as calibration data and indicates the coordinates of feature j (e.g., its keypoints) having coordinates relative to the physical object coordinate system POCS. j,cal Furthermore, the second transformation T2 specifies the transformation parameters between camera 160 and its associated visual coordinate system VIS.

[0069] It should be noted that the stereo back projection described above is sometimes referred to as camera pose estimation, or performed in conjunction with camera pose estimation. Figure 4In the embodiment shown, each feature is symbolized by black dots. Multiple coded features can also be used. Each feature can have a two-dimensional extension including a unique, extended QR-type code (which allows features to overlap). Figure 5 In the diagram, each extended feature 170A is illustrated graphically as being located within a white ring, where the center of the ring defines the feature keypoint. Generally, each extended feature can define this specific keypoint or center that indicates the location or coordinates of the feature. In computer vision, the term "feature" is also referred to as a description around the keypoint (i.e., feature extension). Figure 6 The key point concept of the extended feature in the form of an exemplary ring is illustrated schematically. It should be understood that the key point concept can be readily extended to... Figure 5 The feature type. It should be understood that, in alternative embodiments, the individual features can be defined and encoded in other ways. For example, loops can be formed using different numbers of dashes or combinations of dashes and dots, such as... Figure 7 As shown in feature 170B. Alternatively, color-coded circles or dots can be used. In some implementations, individual features can be grouped on physical objects within small areas forming a flat (i.e., planar) surface. The relative positions of the individual features and their encoding scheme (which allows for differentiation between features) can be used as calibration data storage.

[0070] In this embodiment, the physical object's coordinate system POCS is defined or spans by a feature 170C (in the form of a combination of black and white areas) provided on the two-dimensional surface of the physical object, such as... Figure 8 As shown in the diagram. This means that once the surface-defined features of the physical object (also known as tracker features) are identified and decoded in the visual dataset received from camera 160, the coordinates of those features within the physical object coordinate system POCS, along with the coordinates of the features within the physical object coordinate system POCS, can be determined. System 20 will typically also have prior knowledge (in the form of calibration data) about the relative positions and encodings of the features provided on the physical object.

[0071] Tracker features can be unique and / or encoded, enabling the navigation computer 34 and / or manipulator controller 64 to identify a physical object or information related to the physical object based on the features, such as pattern recognition of the features or information encoded in the features. This information may include one or more of the following: physical object identifier, physical object type, physical object size, physical object dimensions, physical object serial number, or physical object manufacturer.

[0072] refer to Figure 9Once a physical object is identified, the navigation computer 34 and / or the manipulator controller 64 can retrieve a virtual object 164 associated with the physical object in the physical object's coordinate system (POCS) from memory, such as a database of physical objects. Alternatively, features on the physical object can be encoded with information associated with the virtual object, such as virtual object identification, virtual object type, virtual object size, or virtual object dimensions. For example, the database of physical objects stored in the navigation computer 34 may include physical objects made by several different manufacturers, of several different types, and of several different sizes and configurations. The information encoded in the features enables the navigation computer 34 to identify specific details about the physical object viewed in a visual dataset based on the features of that particular physical object, and also to retrieve the specific virtual object associated with that physical object. Thus, for example, if the physical object is identified as a puller of a certain size, the associated virtual object can be of a considerable size.

[0073] By transforming the feature coordinates in the physical object coordinate system (POCS) to the visual coordinate system (VIS), the virtual object 164 can also be transformed to the visual coordinate system (VIS). The virtual object 164 can then be further transformed to the locator coordinate system (LCLZ). Figure 9 (As shown in the diagram), this allows the movement of the surgical instrument 30 to be tracked relative to the virtual object 164. In this respect, the virtual object 164 can be predefined relative to feature coordinates. For example, in the case of a retractor, the virtual object is predefined during manufacturing as a three-dimensional model of the retractor having model data associated with features on the retractor, such that the position and orientation of the virtual object are known in the physical object coordinate system (POCS).

[0074] In an alternative embodiment, prior knowledge of the virtual object 164 relative to a physical object is not required because the coordinates of the surgical instrument 30 are known, allowing the user to define the virtual object 164 using either a navigation guide or the surgical instrument 30. As a result, the user will be able to define a "no-fly zone" by simply outlining the area with a navigation guide, such as a point, line, volume, area, etc. The tracked / outlined area will establish the virtual object 164 relative to one or more features used to track it.

[0075] Virtual object 164 can be defined as a volume or area to be avoided during surgery (e.g., the space occupied by retractor 162). Figure 9As shown, the virtual object 164 can be confined outside the target site TS in the locator coordinate system LCLZ. In other embodiments, portions of the virtual object may exist within the target site TS. The virtual object 164 can be processed by the navigation processor 62 to be displayed to a user on displays 36, 38, allowing the user to see the position and orientation of the surgical instrument 30 relative to, for example, the target site TS of the retractor 162 and the virtual object 164. In some cases, the virtual object 164 includes one or more of a three-dimensional model, a two-dimensional surface, a point cloud, a voxelized volume, a surface mesh, a plane, a line, or a single point.

[0076] It should be noted that due to the elastic and / or flexible properties of some tissues near the target site TS, physical objects (such as retractor 162) can move relative to the target site TS during the surgical procedure. As a result, system 20 periodically refreshes the determined pose of virtual objects, such as the pose of virtual object 164 associated with retractor 162, in the locator coordinate system LCLZ or other coordinate systems of interest. This rate of update of the virtual object pose can be the same as the rate at which the locator updates the pose of trackers 54, 56, 58, the same rate at which the manipulator controller 64 calculates each new commanded position of the surgical instrument along the tool path, or any other suitable rate. In some cases, updating the pose of certain physical objects may be slower than updating the pose of others. For example, because knowing the position of surgical instrument 30 is more critical in certain situations, updating the position of retractor 162 may occur less frequently than updating the position of surgical instrument 30 relative to the target site TS.

[0077] In some embodiments, virtual objects can be generated based on the locations of multiple physical objects. For example, virtual cutout 168 (see...) Figure 9 The virtual incision 168 can be defined by the positions of two or more retractors 162, as shown in the figure. In this case, the virtual incision 168 can be a virtual opening with the boundary outlined by the retractors 162. Thus, the virtual incision 168 can change its configuration when the retractors 162 are adjusted. In other words, the virtual incision 168 can be dynamic and change shape, but using the navigation techniques described herein, the virtual incision 168 can be continuously updated with each new visual dataset, enabling the instrument 30 to avoid physical objects or boundaries defined by virtual objects associated with these physical objects during the surgical procedure, regardless of changes in the vicinity of the target site TS.

[0078] Virtual objects can define volumes or boundaries to be avoided, thus creating a "no-fly zone" to prevent the treatment end of surgical instrument 30 from entering. This "no-fly zone" can be associated with sensitive anatomical structures, rigid structures, soft tissues and bones to be protected, other tools, etc., located near the target site. Knowing the position of the virtual object in the locator coordinate system LCLZ or other common coordinate system, when manipulator 66 operates in autonomous mode, it can control the positioning of surgical instrument 30 to avoid the virtual object, thereby avoiding sensitive anatomical structures, rigid structures, soft tissues and bones to be protected, other tools, etc., located near the target site. During manual manipulation of the robotic device and / or instrument 30, the "no-fly zone" acts as a tactile boundary providing tactile feedback to the user to prevent the user from allowing instrument 30 to enter the "no-fly zone." For example, when the physical object is a retractor, each physical object has a virtual object associated with it in the form of a virtual 3D model of the retractor. Physical objects tracked by camera 160 can be avoided by tracking the movement of the treatment end of surgical instrument 30 relative to the retractor. Then, if one of the retractors is within the toolpath, the manipulator controller 64 can adjust its toolpath or stop the movement of the surgical instrument 30. Alternatively, in manual operation mode, the manipulator controller 64 can limit / stop movement that would otherwise cause the robotic device and / or instrument 30 to collide with one of the retractors, such as by actively actuating one or more joint motors, brakes, etc., thereby generating tactile feedback to the user if the instrument 30 enters a "no-fly zone" during manual operation. Similarly, the navigation system can warn the user that a physical object (such as one of the retractors) is obstructing the planned toolpath and suggest that the user move the physical object.

[0079] refer to Figure 10 One embodiment of a method for tracking a physical object relative to a target site utilizes a robotic device in autonomous mode. In a first step 300, a camera 160 is moved relative to a physical object near the target site by means of a surgical instrument 30 moving autonomously to treat the target site. The camera 160 also moves autonomously relative to the physical object, and in step 302, a visual dataset is generated from multiple perspectives of the physical object (although a single perspective may be sufficient). The visual dataset is defined in a visual coordinate system (VIS).

[0080] In step 304, one or more features, preferably at least three or four features, are identified in the visual dataset. These features may be grouped into feature set 170, enabling camera 160 to determine the feature coordinates of each feature in the visual coordinate system VIS in step 306. Pattern recognition techniques are used to identify the encoded features or other visual identifiers in some embodiments. Based on this recognition, in step 310, navigation computer 34 and / or manipulator controller 64 may capture information about the physical object. This information may include, for example, the position and orientation of the physical object coordinate system POCS relative to the feature coordinates.

[0081] In step 312, the physical object coordinate system POCS can then be transformed to the visual coordinate system VIS and the locator coordinate system LCLZ using coordinate transformer 102. In step 314, the virtual object is associated with the physical object. The virtual object defined in the physical object coordinate system POCS is retrieved from memory and, due to this transformation, is defined relative to the visual coordinate system VIS and subsequently relative to the locator coordinate system LCLZ. In step 316, the manipulator controller 64 controls the autonomous movement of the manipulator 66 and the surgical instrument 30 attached thereto, such that the surgical instrument 30 avoids physical objects or other constraints defined by physical objects.

[0082] It should be understood that although the virtual object associated with the physical object is transformed to the locator coordinate system LCLZ to be tracked relative to the treatment end and target site of the surgical instrument 30, any common coordinate system (such as the manipulator coordinate system MNPL or other coordinate systems) can be used to track the relative movement and posture of the surgical instrument 30, the target site, and the physical object.

[0083] As those skilled in the art will understand, aspects of this embodiment may take the form of a computer program product implemented on one or more computer-readable media having computer-readable program code embodied thereon. Computer software including instructions or code for performing the methods described herein may be stored in one or more associated memory devices (e.g., ROM, fixed or removable memory) and, when ready to be used, may be partially or wholly loaded into a CPU (e.g., RAM) and implemented by the CPU. This software may include, but is not limited to, firmware, resident software, microcode, etc.

[0084] In other embodiments, one or more of the following techniques—SfM (Structure of Motion) technology, Simultaneous Localization and Mapping (SLAM) technology, and pose estimation technology—can be used to determine the feature coordinates and model of a physical object. For example, SLAM can be applied to feature groups. As another example, SfM can construct feature trajectories from different perspectives for individual features (not necessarily feature groups) identifiable in a visual dataset. Triangulation based on different (camera) perspectives can be applied to each feature trajectory. Triangulation can help reconstruct and optionally optimize the feature coordinates in three dimensions (e.g., in the physical object's coordinate system).

[0085] Several embodiments have been discussed in the foregoing description. However, the embodiments discussed herein are not intended to be exclusive or to limit the invention to any particular form. The terminology used is intended to be descriptive rather than restrictive. In view of the above teachings, many modifications and variations are possible, and the invention may be practiced in ways other than those specifically described.

Claims

1. A system for tracking a puller, comprising: Robotic devices; An instrument, attached to a robotic device, configured to treat tissue; A vision device, fixedly attached to one of a robotic device and a mechanism, such that the vision device can move with the robotic device, is configured to generate a visual dataset, wherein the visual dataset is captured from multiple perspectives of a retractor by moving the vision device in multiple degrees of freedom during the movement of the robotic device; and A computing system having at least one processor and communicating with a vision device, said at least one processor being configured to: Identify a set of features applied to a retractor in a visual dataset, wherein at least one of the features encodes information related to the retractor; Based on encoded information, a predetermined size of a virtual object associated with a retractor is retrieved, wherein the virtual object at least partially defines a virtual boundary that defines constraints on the movement of the robot device relative to the retractor, and wherein the virtual object also at least partially defines a virtual boundary of the instrument. The position of the feature relative to the vision device is determined so as to track the pose of the virtual object associated with the puller relative to the vision device based on the vision dataset; The orientation of the tracking device relative to a known coordinate system; Based on the pose of a tracking robot device and apparatus relative to the known coordinate system, the pose of the tracked virtual object associated with the retractor relative to the vision device, and the fixed relationship, the pose of the tracked virtual object associated with the retractor relative to the known coordinate system is tracked; and Based on the tracked pose of the virtual object associated with the retractor relative to the known coordinate system and the pose of the instrument relative to the known coordinate system, the movement of the robotic device is constrained such that the robotic device and the instrument do not exceed the virtual boundaries during the movement of the robotic device to treat tissue. Among them, the posture of the retractor is updated less frequently compared to the posture of the retractor.

2. The system of claim 1, wherein, The visual device includes a video camera.

3. The system of claim 1, wherein, The vision device is attached to the instrument according to the fixed relationship, and the computing system is configured to track the posture of the vision device relative to the known coordinate system based on the posture of the instrument relative to the known coordinate system and the fixed relationship, which is tracked during the autonomous movement of the instrument along a planned tool path to treat tissue by the robotic device.

4. The system of claim 1, wherein, The computing system is configured to determine the identity of the puller based on the encoded information.

5. The system of claim 1, wherein, The robotic device is configured to operate in an autonomous mode, enabling it to autonomously move instruments along a planned tool path to treat tissue, and to capture visual datasets from multiple perspectives of the retractor by moving a vision device in multiple degrees of freedom during the autonomous movement of instruments along the planned tool path to treat tissue.

6. The system according to claim 1, wherein, The at least one processor is further configured to: Determine whether the virtual boundary obstructs the planned machine path; and In response to the determination that virtual boundaries hinder the planned instrument path, an adjusted path for the instrument is provided.

7. The system according to claim 1, wherein, The at least one processor is further configured to: The virtual object associated with the retractor is displayed graphically on the monitor.

8. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement a method for tracking a puller using a system comprising: Robotic devices; An instrument, attached to a robotic device configured to treat tissue; a visual device, attached in a fixed relationship to one of the robotic device and the instrument, such that the visual device can move with the robotic device. And a computing system having at least one processor and communicating with a vision device, the method includes the following steps: A visual dataset is generated by a vision device in response to the movement of the vision device caused by the movement of the robot device, wherein the visual dataset is captured from multiple perspectives of the puller; The at least one processor identifies a set of features applied to the retractor in a visual dataset, wherein at least one of the features encodes information related to the retractor; The processor retrieves a predetermined size of a virtual object associated with a retractor based on encoded information, wherein the virtual object defines a virtual boundary that defines constraints on the movement of the robot device relative to the retractor, and wherein the virtual object also at least partially defines the virtual boundary of the instrument. The location of the feature in the visual dataset is determined by the at least one processor; Position tracking based on the determined features is used to determine the pose of the virtual object associated with the puller relative to the vision device. The at least one processor tracks the orientation of the instrument relative to a known coordinate system. Using the at least one processor, the pose of the virtual object associated with the retractor relative to the known coordinate system is tracked based on the pose of one of the tracked robotic devices and instruments relative to the known coordinate system, the pose of the tracked virtual object associated with the retractor relative to the vision device, and the fixation relationship; and The at least one processor constrains the movement of the robotic device based on the tracked pose of the virtual object associated with the retractor relative to the known coordinate system and the pose of the instrument relative to the known coordinate system, such that the robotic device and the instrument do not exceed the virtual boundaries during the movement of the robotic device to treat tissue. Among them, the posture of the retractor is updated less frequently compared to the posture of the retractor.

9. The storage medium according to claim 8, wherein, The vision device is attached to the instrument according to the fixation relationship, and the method further includes: tracking the posture of the vision device relative to the known coordinate system based on the posture of the instrument relative to the known coordinate system tracked during the autonomous movement of the instrument along a planned tool path to treat tissue by the robotic device, and the fixation relationship.

10. The storage medium according to claim 8, wherein, The method also includes: The identity of the puller is determined based on the encoded information.

11. The storage medium according to claim 8, wherein, The method further includes: an autonomously moving robotic device that autonomously moves the instrument along a planned tool path to treat the tissue, and during the autonomous movement of the instrument along the planned tool path to treat the tissue, a visual dataset is captured from multiple perspectives of the retractor by moving a vision device in multiple degrees of freedom.

12. The storage medium according to claim 8, wherein, The method also includes: The at least one processor determines whether a virtual boundary obstructs a planned machine path; and The at least one processor provides an adjusted path for the instrument in response to determining that a virtual boundary is hindering the planned instrument path.

13. The storage medium according to claim 8, wherein, The method also includes: The virtual object associated with the retractor is graphically presented on the display using the at least one processor.

Images