Systems and methods related to registration for medical procedures

Abstract

A medical system includes one or more processors. The processor is configured to perform operations including receiving image data associated with an anatomical feature from a first imaging device. The image data is associated with the first imaging device space. Probe data associated with the anatomical feature is obtained from a second imaging device. The probe data is associated with the second imaging device space. The probe data is registered to the image data to generate a first registration between the second imaging device space and the first imaging device space. An anatomical model associated with the anatomical feature is registered to the probe data to generate a second registration between a model space of the anatomical model and the second imaging device space. A third registration between the model space and the first imaging device space is generated based on the first registration and the second registration.

Description

[0001] Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 62 / 866,209, filed on June 25, 2019, which is incorporated herein by reference. Technical Field

[0003] The present disclosure relates to systems for performing medical procedures, and more particularly to systems and methods for registering anatomical models to real-time image data during a medical procedure. Background Art

[0004] Medical robotic systems, such as teleoperations used to perform minimally invasive surgical procedures, offer numerous benefits over traditional open surgical techniques, including less pain, shorter hospital stays, faster return to normal activities, minimal scarring, reduced recovery time, and less damage to tissue. As a result, demand for such medical teleoperations is strong and growing.

[0005] Examples of medical teleoperation systems include the da Surgical System and da Each of these systems includes an operator's console, a patient cart, a high-performance three-dimensional ("3-D") vision system, and Intuitive Surgical's proprietary Articulated instruments, Articulated instruments are modeled after the human wrist. When added to the motion of the manipulator holding the surgical instrument, these articulated instruments allow for at least six degrees of freedom of motion of their end effectors, which is comparable to or greater than the natural motion of open surgery.

[0006] Minimally invasive medical technology aims to reduce the amount of damaged tissue during medical procedures, thereby reducing patient recovery time, discomfort and harmful side effects. Such minimally invasive techniques can be performed through natural orifices in the patient's anatomical structure or through one or more surgical incisions. Through these natural orifices or incisions, the operator can insert minimally invasive medical instruments (including surgical, diagnostic, therapeutic or biopsy instruments) to reach the target tissue position. In order to assist in reaching the target tissue position, the position and movement of the medical instrument can be associated with the preoperative or intraoperative image of the patient's anatomical structure. With image-guided instruments associated with images, instruments can navigate in natural or surgically created channels in anatomical systems (such as lungs, colon, intestines, kidneys, heart, circulatory system, etc.). However, it may be computationally expensive and time-consuming to align the anatomical model generated from the preoperative or intraoperative image to the real-time image data provided by the imaging device, and the alignment quality (e.g., accuracy, completeness, validity, consistency) may be unsatisfactory, which may cause uncertainty in image-guided procedures.

[0007] Therefore, it would be advantageous to provide improved registration for performing image-guided procedures. Summary of the invention

[0008] Embodiments of the present invention are best summarized by the claims that follow the description.

[0009] In some embodiments, the medical system includes one or more processors. The processors are configured to perform operations including: receiving image data associated with an anatomical feature of a patient from a first imaging device, the image data being associated with a first imaging device space; obtaining probe data associated with the anatomical feature from a second imaging device, the probe data being associated with a second imaging device space; registering the probe data to the image data to generate a first registration between the second imaging device space and the first imaging device space; registering an anatomical model associated with the anatomical feature to the probe data to generate a second registration between a model space of the anatomical model and the second imaging device space; and generating a third registration between the model space and the first imaging device space based on the first registration and the second registration.

[0010] In some embodiments, a method includes receiving image data associated with an anatomical feature of a patient from a first imaging device, the image data being associated with the first imaging device space; obtaining probe data associated with the anatomical feature from a second imaging device, the probe data being associated with a second imaging device space; registering the probe data to the image data to generate a first registration between the second imaging device space and the first imaging device space; registering an anatomical model associated with the anatomical feature to the probe data to generate a second registration between a model space of the anatomical model and the second imaging device space; and generating a third registration between the model space and the first imaging device space based on the first registration and the second registration.

[0011] In some embodiments, a non-transitory machine-readable medium includes a plurality of machine-readable instructions which, when executed by one or more processors associated with a medical device, are suitable for causing the one or more processors to perform any of the operations or methods described herein.

[0012] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1A is a schematic diagram of a remotely operated medical system in a surgical reference frame according to an embodiment of the present disclosure; Figure 1B is a perspective view of a patient-side system according to one example of the principles described herein.

[0014] Figure 2A , Figure 2B and Figure 2C is a simplified diagram of a system for registering an anatomical model to image data provided by an imaging device during a medical procedure, in accordance with some embodiments.

[0015] Figure 3A , Figure 3B and Figure 3C According to some embodiments, Figure 2A-2C A simplified diagram of image data captured by an imaging device in the depicted configuration.

[0016] Figure 4A is a flow chart illustrating a method for performing registration of an anatomical model to a patient's anatomy in an image-guided surgical procedure or a portion thereof, according to some embodiments; Figure 4B showing an anatomical model with an identified target probe region according to some embodiments; Figure 4C illustrates an estimated target probe region corresponding to a target probe region identified in the image data according to some embodiments; and Figure 4D is a display showing a registered anatomical model superimposed on image data in accordance with some embodiments.

[0017] Figure 5 is a flow chart illustrating a method for automatically determining an activation state of a probe device according to some embodiments.

[0018] Fig. 6A is a flow chart illustrating a method for registering probe data to image data in an image-guided surgical procedure in accordance with some embodiments.

[0019] Figure 6B is a display showing a projection of a probe device onto image data during registration of the probe data to the image data in accordance with some embodiments.

[0020] Figure 7 is a flow chart illustrating a method for performing registration of an anatomical model to probe data in an image-guided surgical procedure in accordance with some embodiments. DETAILED DESCRIPTION

[0021] For the purpose of promoting understanding of the principles of the present disclosure, reference will now be made to the embodiments shown in the accompanying drawings, and specific language will be used to describe them. However, it should be understood that it is not intended to limit the scope of the present disclosure. In the following detailed description of aspects of the present invention, many specific details are set forth to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to those skilled in the art that embodiments of the present disclosure may be practiced without these specific details. In other cases, well-known methods, procedures, components and circuits are not described in detail to avoid unnecessary confusion of aspects of embodiments of the present invention.

[0022] As would normally occur to a technician in the field to which the present disclosure relates, any changes and further modifications to any further application of the described equipment, apparatus, method, and principles of the present disclosure are fully contemplated. Specifically, it is fully contemplated that the features, components, and / or steps described with respect to one embodiment may be combined with the features, components, and / or steps described with respect to other embodiments of the present disclosure. In addition, the dimensions provided herein are used for specific examples, and it is contemplated that the concepts of the present disclosure may be implemented using different sizes, dimensions, and / or ratios. To avoid unnecessary description repetition, one or more components or actions described according to an illustrative embodiment may be used or omitted when applicable to other illustrative embodiments. For the sake of brevity, multiple iterations of these combinations will no longer be described separately. For simplicity, in some cases, the same reference numerals are used throughout the accompanying drawings to refer to the same or similar parts.

[0023] The following embodiments will describe various apparatuses and parts of apparatuses according to their states in three-dimensional space. As used herein, the term "positioning" refers to the position of an object or a portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or a portion of an object (three rotational degrees of freedom - e.g., roll, pitch, and yaw). As used herein, the term "attitude" refers to the positioning of an object or a portion of an object in at least one translational degree of freedom and the orientation of the object or a portion of the object in at least one rotational degree of freedom (up to six total degrees of freedom). As used herein, the term "shape" refers to a set of attitudes, positions, or orientations measured along an object.

[0024] Aspects of the present invention are based primarily on the use of computer-assisted medical systems such as those commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Surgical System). However, those skilled in the art will appreciate that the inventive aspects disclosed herein may be embodied and implemented in a variety of ways, including robotic and (if applicable) non-robotic embodiments and implementations. Surgical System) are merely exemplary and should not be considered to limit the scope of the invention disclosed herein. For example, any reference to surgical instruments and surgical methods is non-restrictive, because the tools, systems and methods described herein can be used for animals, human corpses, animal corpses, parts of human or animal anatomical structures, tissues removed from human or animal anatomical structures (with or without returning to human or animal anatomical structures), non-surgical diagnosis, industrial systems and general robots or remote operation systems. As a further example, the instruments, systems and methods described herein can be used for non-medical purposes, including industrial uses, general robotic uses, sensing or manipulating non-tissue artifacts, cosmetic improvements, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, setting up or removing systems, training medical or non-medical personnel, etc. Additional example applications include surgery on tissue removed from human or animal anatomical structures (without returning to human or animal anatomical structures) and surgery on human or animal corpses. In addition, these technologies can also be used for medical treatment or diagnostic procedures that include or do not include surgical aspects.

[0025] Refer to the attached figure Figure 1A A teleoperated medical system for use in medical procedures, such as those including diagnostic, therapeutic, or surgical procedures, is generally indicated by reference numeral 10 and operates within a surgical environment having a surgical reference coordinate system XS, YS, ZS. As will be described, the teleoperated medical system of the present disclosure is under remote operational control of an operator. In alternative embodiments, the teleoperated medical system may be under partial control of a computer programmed to execute a procedure or subroutine. In other alternative embodiments, a fully automated medical system under full control of a computer programmed to execute a procedure or subroutine may be used to execute the procedure or subroutine. Figure 1AAs shown, the teleoperated medical system 10 generally includes a teleoperated assembly 12, which is mounted to or near an operating table T, on which a patient P is located. The teleoperated assembly 12 may be referred to as a patient-side cart. A medical device system 14 and an endoscopic imaging system 15 are operably coupled to the teleoperated assembly 12. An operator console 16 allows an operator O (e.g., a surgeon or other type of clinician) to view an image of the surgical site or an image representing the surgical site and control the operation of the medical device system 14 and / or the endoscopic imaging system 15. As will be described below, a supplementary imaging system 17 including a minimally invasive image capture probe 19 may be used with the medical device system 14 and the endoscopic imaging system 15. Together with the patient-side cart, the supplementary imaging system 17 may be referred to as a patient-side system. The endoscopic imaging system 15 provides an image of the outer surface of an anatomical structure within a surgical environment. The supplementary imaging system 17 may be an internal imaging system, such as an ultrasound, x-ray, or gamma imaging system capable of imaging outside the outer surface of an anatomical structure.

[0026] The operator console 16 may be located at the operator's console, which is typically located in the same room as the operating table T. However, it should be understood that the operator O may be located in a different room or a completely different building from the patient P. The operator console 16 includes a left eye display and a right eye display for presenting a coordinated stereoscopic view of the surgical site to the operator O that enables depth perception. The console 16 also includes one or more input control devices that cause the remote operation assembly 12 to manipulate one or more instruments or endoscopic imaging systems. The input control device may provide the same degrees of freedom as the instrument 14 with which it is associated to provide the operator O with a sense that the remote presentation or input control device is integrated with the instrument 14, so that the operator has a strong sense of directly controlling the instrument 14. To this end, positioning, force and tactile feedback sensors (not shown) may be employed to transmit positioning, force and tactile sensations from the instrument 14 back to the operator's hand through the input control device. The one or more control devices may include one or more of any number of various input devices, such as handles, joysticks, trackballs, data gloves, trigger guns, manually operated controllers, voice recognition devices, touch screens, body motion or presence sensors, etc. In some embodiments, the one or more control devices will be provided with the same degrees of freedom as the medical device of the remotely operated component to provide the operator with telepresence, the perception that the one or more control devices are one with the device, so that the operator has a strong sense of directly controlling the device, as if he were right at the surgical site. In other embodiments, the one or more control devices may have more or fewer degrees of freedom than the associated medical device and still provide telepresence to the operator. In some embodiments, the one or more control devices are manual input devices that move with six degrees of freedom and may also include an actuatable handle for actuating the device (e.g., for closing the grasper, applying an electric potential to an electrode, delivering drug therapy, etc.).

[0027] The teleoperation assembly 12 supports and manipulates the medical device system 14 while the operator O views the surgical site through the console 16. Images of the surgical site can be obtained by an endoscopic imaging system 15 (e.g., a stereo endoscope), which can be manipulated by the teleoperation assembly 12 to orient the endoscope 15. An electronic cart 18 can be used to process images of the surgical site for subsequent display to the operator O through the operator console 16. The number of medical device systems 14 used at one time generally depends on factors such as diagnostic or surgical procedures and space limitations in the operating room. The teleoperation assembly 12 may include one or more non-servo-controlled linkages (e.g., one or more linkages that can be manually positioned and locked in place, generally referred to as setup structures) and a kinematic structure of a teleoperated manipulator. The teleoperation assembly 12 includes a plurality of motors that drive inputs on the medical device system 14. These motors move in response to commands from a control system (e.g., control system 20). The motor includes a drive system that, when coupled to the medical device system 14, can advance the medical device into a natural or surgically created anatomical orifice. Other motorized drive systems can move the distal end of a medical instrument with multiple degrees of freedom, which can include three degrees of freedom for linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of freedom for rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). In addition, a motor can be used to actuate an articulated end effector of the instrument to grasp tissue in the jaws of a biopsy device, etc.

[0028] The teleoperated medical system 10 also includes a control system 20. The control system 20 includes at least one memory and at least one processor (not shown), and typically includes multiple processors for implementing control between the medical instrument system 14, the endoscope system 15, the operator console 16, and the monitor on the electronics cart 18. The control system 20 can also receive and process images from the supplemental imaging system 17. The electronics cart 18 can house components of the endoscopic imaging system 15, the supplemental imaging system 17, the control system 20, and the monitor and processor for processing and displaying captured images.

[0029] The control system 20 also includes programming instructions (e.g., a computer-readable medium storing instructions) to implement some or all of the methods described in accordance with the aspects disclosed herein. Figure 1A1 is shown as a single box in the simplified schematic diagram of , but the system may include two or more data processing circuits, where optionally part of the processing is performed on or adjacent to the remote operation component 12, another part of the processing is performed at the operator console 16, and so on. Any of a variety of centralized or distributed data processing architectures can be used. Similarly, the programming instructions can be implemented as multiple separate programs or subroutines, or they can be integrated into multiple other aspects of the remote operation system described herein. In one embodiment, the control system 20 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE802.11, DECT, and wireless telemetry.

[0030] In some embodiments, the control system 20 may include one or more servo controllers that receive force and / or torque feedback from the medical device system 14. In response to the feedback, the servo controller transmits a signal to the operator console 16. The one or more servo controllers may also transmit a signal instructing the remote operation assembly 12 to move one or more medical device systems 14 and / or endoscopic imaging systems 15, which extend to an internal surgical site in the patient's body via an opening in the body. Any suitable conventional or dedicated servo controller may be used. The servo controller may be separate from or integrated with the remote operation assembly 12. In some embodiments, the servo controller and the remote operation assembly are provided as part of a remote operation arm cart positioned adjacent to the patient's body.

[0031] The teleoperated medical system 10 may also include optional operating and support systems (not shown), such as lighting systems, steering control systems, irrigation systems, and / or suction systems. In alternative embodiments, the teleoperated system may include more than one teleoperated assembly and / or more than one operator console. The exact number of manipulator assemblies will depend on factors such as the surgical procedure and space limitations in the operating room. The operator consoles may be collocated, or they may be located in separate locations.

[0032] Figure 1Bis a perspective view of one embodiment of a teleoperation assembly 12 and an accessory system 17, which may be collectively referred to as a patient-side system. The teleoperation assembly 12 shown provides manipulation of three surgical tools 26 (e.g., an instrument system 14) and an imaging device 28 (e.g., an endoscopic imaging system 15 (e.g., a stereo endoscope for capturing images of a surgical site)). The imaging device may transmit signals to an electronics cart 18 via a cable 56. Manipulation is provided by a teleoperation mechanism having multiple joints. The imaging device 28 and the surgical tool 26 may be positioned and manipulated through an incision or natural orifice in the patient's body such that a kinematic telecenter is maintained at the incision to minimize the size of the incision. The image of the surgical site may include an image of the distal end of the surgical tool 26 when the distal end of the surgical tool 26 is positioned within the field of view of the imaging device 28.

[0033] The remote operation assembly 12 includes a drivable base 58. The drivable base 58 is connected to a telescopic column 57, which allows adjustment of the height of the arm 54. The arm 54 may include a rotary joint 55 that both rotates and moves up and down. Each of the arms 54 can be connected to an orientation platform 53. The orientation platform 53 can be capable of 360-degree rotation. The remote operation assembly 12 may also include a telescopic horizontal boom 52 for moving the orientation platform 53 in a horizontal direction.

[0034] In this example, each of the arms 54 is connected to a manipulator arm 51. The manipulator arm 51 can be directly connected to the surgical tool 26. The manipulator arm 51 can be remotely operable. In some examples, the arm 54 connected to the directional platform is not remotely operable. Instead, such an arm 54 is positioned as needed before the operator begins operation with the remote operation component.

[0035] The accessory system 17 may provide one or more functions that enhance and / or supplement the functions provided by the surgical tool 26. The accessory system 17 includes an accessory 19 that is used to provide one or more additional functions. The accessory 19 may be a minimally invasive instrument that is sized and designed for insertion into a surgical environment. Examples of accessories and their functions are described below with reference to Figure 2A-Figure 7 Discussed in more detail. Accessory 19 may include an endoscopic imaging system.

[0036] Endoscopic imaging systems (e.g., systems 15, 19, and / or 28) can be provided in various configurations including rigid or flexible endoscopes. Rigid endoscopes include a rigid tube that houses a relay lens system for transmitting images from the distal end of the endoscope to the proximal end. Flexible endoscopes use one or more flexible optical fibers to transmit images. Endoscopes based on digital images have a "chip-on-the-tip" camera design in which a distal digital sensor (e.g., one or more charge-coupled devices (CCDs) or complementary metal oxide semiconductor (CMOS) devices) acquires image data. Endoscopic imaging systems can provide an observer with a two-dimensional or three-dimensional image of the endoscope's field of view (i.e., the imaging area). Two-dimensional images may provide limited depth perception. Three-dimensional stereoscopic endoscopic images can provide an observer with more accurate depth perception. Stereoscopic endoscopic instruments use stereo cameras to capture stereoscopic images of the patient's anatomical field of view. Endoscopic instruments can be fully sterilizable assemblies in which the endoscope cable, handle, and shaft are rigidly coupled and tightly sealed.

[0037] Figure 2A-2C is a simplified diagram illustrating a system 200 for registering an anatomical model to image data captured by an imaging device, according to some embodiments. Figure 3A-3C are respectively shown according to some embodiments from Figure 2A-2C A simplified diagram of image data 300 of imaging device 210 of system 200 is depicted. In some embodiments, image data 300 may be displayed to an operator via a display of console 16 during a medical procedure.

[0038] refer to Figure 2A and Figure 3A In an example of a system 200, an imaging device 210 is associated with an imaging region or field of view 215. According to some embodiments, the imaging device 210 may generally correspond to an endoscope, such as endoscope 15. The system 200 further includes a probe device 220. According to some embodiments, the probe device 220 may generally correspond to an accessory, such as accessory 19, a plug-in accessory, etc. In some embodiments, the probe device 220 includes an imaging device (e.g., an ultrasound probe) that uses a different imaging modality than the imaging device 210.

[0039] System 200 may also include a medical instrument 230 having an end effector 235. According to some embodiments, medical instrument 230 may generally correspond to any of instruments 14. In some embodiments, end effector 235 may provide mechanical, magnetic, or other types of gripping functionality (e.g., jaws) capable of gripping probe device 220.

[0040] like Figure 2A and Figure 3AAs shown in the example of , anatomical feature 240 is located within field of view 215 of imaging device 210. Anatomical feature 240 may include or correspond to a target of a medical procedure. For example, anatomical feature 240 may correspond to an organ (e.g., a kidney) that is operated on during a medical procedure.

[0041] like Figure 2A and Figure 3A As shown in the example of , probe device 220 is located within field of view 215 of imaging device 210. Probe device 220 may correspond to an insertable device that may be used during a medical procedure.

[0042] refer to Figure 2B and Figure 3B For example, in some embodiments, the operator can control the medical instrument 230 to grasp and manipulate the probe device 220.

[0043] refer to Figure 2C and Figure 3C For example, the operator may use the probe device 220 to scan the anatomical feature 240 or a portion thereof by controlling the medical instrument 230. The probe device 220 may generate probe data by scanning the anatomical feature 240.

[0044] refer to Figure 4A-Figure 7 In some embodiments, the system 200 may use probe data generated by the probe device 220 to register the anatomical model with the image data 300 generated by the imaging device. Figure 4A is a simplified diagram of a method 400 for registering an anatomical model to image data using probe data from probe device 220 during a medical procedure. Figure 4B An anatomical model with target probe areas identified is shown. Figure 4C is a display showing an estimated target probe area corresponding to a target probe area identified in the image data. Figure 4D is a display showing a registered anatomical model superimposed on image data in accordance with some embodiments. Figure 4D A display is shown in which the image data 300 is enhanced to include an anatomical model superimposed on the anatomical feature 240 of the image data 300 using registration. Figure 5 A method 500 for automatically detecting activation of a probe device for providing probe data is shown in accordance with some embodiments. Figure 6 shows a method 600 for registering probe data to image data in accordance with some embodiments. Figure 7 A method for registering an anatomical model to probe data according to some embodiments is shown.

[0045] Figure 4Ais a flow chart illustrating a method 400 for registering an anatomical model with image data 300 using probe data generated by a probe device that scans anatomical features 240 during a medical procedure. The method 400 Figure 4A 400 as a set of operations or processes 410 to 490. Not all of the processes 402 to 490 shown may be performed in all embodiments of the method 400. Figure 4A One or more processes not explicitly shown in the drawings may be included before, after, between, or as part of processes 402 through 490. In some embodiments, one or more of the processes may be implemented at least in part in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when executed by one or more processors (e.g., a processor of the control system 20), may cause the one or more processors to perform one or more of the processes.

[0046] At process 402, an anatomical model 245 associated with an anatomical feature 240 is received. The anatomical model 245 is associated with a model space. In some embodiments, the anatomical model 245 is determined based on preoperative data (e.g., using a preoperative CT scan) and / or an intraoperative data model of the anatomical feature 240. In various embodiments, the anatomical model 245 may be a 2D or 3D model and may include various anatomical feature information of the anatomical feature 240. The anatomical feature information may include, for example, the vasculature of the anatomical feature 240, one or more specific vessel bifurcations in the vasculature, blood flow velocity and direction of the vessels, and the like.

[0047] At process 410, image data is obtained using an imaging device of system 200. The image data is spatially associated with the imaging device. Figure 2A-Figure 3C In the example of FIG. 3 , the imaging device 210 can be used to obtain image data 300 of a surgical site of a patient. The imaging device 210 records a concurrent or real-time image of the surgical site including anatomical features 240 and provides the image data to a processor (e.g., a processor of the control system 20). The image data can be provided to an operator via one or more displays of the medical system 10 (e.g., one or more displays of the console 16). The concurrent image can be, for example, a two-dimensional or three-dimensional image captured by an endoscope (e.g., endoscope 15) positioned within the surgical site.

[0048] At process 420, the activation state of the probe device is determined. Figure 2A-Figure 3CIn an example, the activation state of a probe device 220 (e.g., an ultrasound probe) can be determined. In an example, the probe device 220 can be a plug-in accessory that lacks independent movement capabilities. The probe device 220 is referred to as being in an activated state when it is controllable by an operator, and is referred to as being in an inactivated state when it is not controlled by the operator. In an example, the operator can control a medical device 230 having an end effector 235 to grasp the probe device 220, and then control the probe device 220 by controlling the medical device 230. In this example, the probe device 220 is in an activated state. In another example, the probe device 220 is not controlled by any instrument controllable by the operator. In this example, the probe device 220 is in an inactivated state. In some embodiments, the operator can manually indicate the activation state of the probe device 220 (e.g., an activated state or an inactivated state) (e.g., using the operator console 16). In some alternative embodiments, as described below with reference to Figure 5 As discussed in detail, the activation state of the probe device 220 can be automatically determined. In some embodiments, at process 420, if it is determined that the probe device 220 has an activated state, the method 400 can proceed to processes 430 to 490 to receive probe data from the probe device 220 and perform registration from the anatomical model to the imaging data based on the probe data. In some embodiments, at process 420, if it is determined that the probe device 220 has an inactive state, the method 400 can wait until it is detected that the probe device 220 is in an activated state.

[0049] At process 430, the processor of the control system 20 obtains probe data from the probe device. In some embodiments, the probe device 220 records concurrent or real-time images of the surgical site including anatomical features 240 (e.g., by scanning anatomical features 240), and provides the probe data to the processor (e.g., the processor of the control system 20). The probe data is associated with the probe device space. The probe data can be provided to the operator through one or more displays of the medical system 10 (e.g., one or more displays of the console 16). In the example where the probe device 220 is an ultrasound device, the probe data includes ultrasound data. In various embodiments, the field of view of the probe device 220 can have a three-dimensional frustum shape, a frustum shape, a slice-of-pie shape, or some other shape. In some embodiments, the probe device 220 provides a field of view different from the field of view of the imaging device 210.

[0050] In some embodiments, at process 430, probe device 220 may scan the entire surface of anatomical feature 240. Alternatively, in some embodiments, process 430 may use processes 440 and 450 to obtain local probe data of anatomical feature 240 at a specific local probe area, which may reduce scanning time without affecting registration accuracy. Figure 4B At process 440, target probe regions 246 and 248 are determined based on the anatomical feature elements of the anatomical feature 240 (e.g., one or more specific vascular bifurcations in the vascular system, blood flow velocity and direction of the blood vessels, etc.). The probe data from these probe regions 246 and 248 can be used for more accurate registration between the probe data and the anatomical model 245 (e.g., using feature-based registration in the image processing algorithm to map those anatomical feature elements in the probe data and the anatomical model 245). In some embodiments, the processor can determine whether a 2D model (e.g., a cross section) or a 3D model of the probe data from a specific probe region is used for improved registration between the probe data and the anatomical model 245 based on the anatomical model 245 and its anatomical feature element information. In an example, the processor determines that the probe region 246 includes a single vascular bifurcation, and the 2D model from the probe data is suitable for registration. In another example, the processor determines that the probe region 248A includes multiple vascular bifurcations, and the 3D model from the probe data is suitable for registration.

[0051] refer to Figure 4C , at process 450, the processor may provide instructions to the operator to move the instrument 230 toward the estimated probe areas 246 and 248 shown in the image data, so that the probe device 220 scans the estimated probe areas 246 and 248 in the image data 300. The estimated probe areas 246 and 248 correspond to the target probe areas 246 and 246 of the anatomical model 245, respectively, and the estimated probe areas 246 and 248 may be generated based on the registration between the anatomical model 245 and the image data 300 (e.g., the less accurate registration without using any probe data). In some embodiments, the instructions may indicate whether the probe data is collected for a 2D model or a 3D model. In the example of a 2D model (e.g., for estimating the probe area 246B), the instructions may include a probe device direction / orientation aligned with a 2D cross-sectional direction, which is aligned with the probe device direction / orientation. In the example of a 3D model (e.g., for estimating the probe area 248B), the instructions may include multiple probe device directions / orientations so that the collected probe data is sufficient for 3D reconstruction of the probe area.

[0052] At process 460, the probe data is registered to the image data to generate a first registration. Various image registration algorithms may be used, including, for example, intensity-based registration, feature-based registration, registration with a linear transformation model, registration with a non-rigid transformation model, etc. Process 460 may generate a first registration (e.g., a first transformation model) between the probe device space and the imaging device space. Fig. 6A and Figure 6BAn example registration process of process 460 is described in detail. In some embodiments, after completing the registration process of registering the probe data to the image data, a computer-aided design (CAD) model of the probe device 220 can be superimposed on the image data in one or more displays of the medical system 10 (e.g., one or more displays of the console 16 based on the first registration). For example, the position of the CAD model of the probe device 220 on the image data can be determined based on the first registration. In some embodiments, the probe data (e.g., ultrasound image / video) can be superimposed (with or without the CAD model of the probe device 220) on the image data in the display based on the first registration.

[0053] At process 470, the anatomical model is registered to the probe data to generate a second registration. Various image registration algorithms can be used, including, for example, intensity-based registration, feature-based registration, registration with a linear transformation model, registration with a non-rigid transformation model, etc. Process 460 can generate a second registration (e.g., a second transformation model) between the probe device space and the imaging device space. In some embodiments, a full scan of the anatomical feature 240 is performed by the probe device. In such an embodiment, a reconstruction (2D or 3D) of the entire anatomical feature is performed based on the full scan. For example, 3D reconstruction can be performed using both the probe data and the depth map from the image data 300. Such a full scan may be time-consuming and computationally expensive.

[0054] Alternatively, refer to Figure 7 Described in detail, in some embodiments, only need to carry out partial scanning (for example, probe area 246,248) to anatomical feature 240 by probe device 220 for registration.The probe data from each of probe area can be used for 2D or 3D registration.In an example, the 2D image of the probe data from probe area 246 can be used for probe data and 3D anatomical model registration (for example, using the cross section of 3D anatomical model).In another example, the 3D model of the probe data from probe area 248 can be used for probe data and 3D anatomical model registration.

[0055] In various embodiments, various types of anatomical feature information can be determined using probe data from the probe device 220, and the probe data from the probe device 220 can be used to improve the second registration. In some examples, those types of information from the probe data may not be obtained from the image data from the imaging device 210. For example, the probe device 220 provides an ultrasound image of the anatomical feature 240 with color Doppler, which can provide vascular blood flow information of the anatomical feature 240 (e.g., by superimposing colors on the image to indicate the speed and direction of blood flow in the blood vessel). In some embodiments, the anatomical model (e.g., a 3D model including blood flow information of a blood vessel) can be registered to the probe data by matching those types of information (e.g., blood flow information of a blood vessel) in the anatomical model and the probe data. In some embodiments, the registration is performed by registering the 3D anatomical model with a 2D image of the probe data (e.g., a cross section including one or more blood vessels).

[0056] At process 480, the anatomical model is registered to the image data to generate a third registration. The third registration can be generated using the first registration and the second registration. By combining the rich information provided by the probe device, the accuracy of the third registration between the anatomical model and the image data is improved. In an example, at process 480, the third registration (e.g., the third transformation model) between the model space associated with the anatomical model and the imaging device space associated with the image data is determined by combining the first registration (e.g., the first transformation model) and the second registration (e.g., the second transformation model). In an example, the anatomical model is registered to the image data by transforming the anatomical model using the second registration. Then the anatomical model transformed in the probe device space is transformed to the imaging device space using the first registration, and the anatomical model transformed in the probe device space is registered to the image data.

[0057] At process 490, the registered anatomical model using the third registration is displayed along with the image data on one or more displays of the medical system 10 (eg, one or more displays of the console 16). Figure 4D In an example of an anatomical model 245, the anatomical model 245 is superimposed on the image data 300 in the display based on a third registration between the model space and the imaging device space. In some embodiments, one or more procedures (e.g., one or more surgical, diagnostic, therapeutic, or biopsy procedures) are performed (e.g., by an operator using medical instructions) on the anatomical features based on the third registration (e.g., based on the registered anatomical structure).

[0058] Figure 5 is a simplified diagram of a method 500 for automatically detecting activation of a probe device according to some embodiments. Figure 4A Consistent with some embodiments, method 500 may generally correspond to process 420 of method 400 .

[0059] At process 510, one or more probe activity signals associated with the activity of the probe device are received. In some embodiments, the probe activity signals may include various signals related to the activity or inactivity of the probe device. For example, the probe activity signals may include one or more of the following: the posture of the medical device; the gripping position of the end effector of the medical device (for example, certain postures and gripping positions may be related to the operator gripping the probe device with the medical device); probe data (for example, the presence or absence of reflections in the ultrasonic probe data can be used to distinguish whether the probe device interacts with air, tissue, or another material); instrument kinematics; the orientation of the probe device relative to the standard or effective orientation of the probe device when gripped by the medical device, etc.

[0060] At process 520, the activation state of the probe device is automatically determined based on the probe activity signal. Various activation state determination algorithms can be used to determine the activation state of the probe device. In an example, at process 530, the activation state is determined by comparing the probe activity signal with a predetermined activation threshold. In this example, if the probe activity signal exceeds the predetermined threshold, the processor can determine that the probe device is in an activated state, and if the probe activity signal does not exceed the predetermined threshold, the processor determines that the probe device is in an inactive state. In another example, at process 540, a trained decision tree is used to determine the activation state of the probe device. In yet another example, at process 550, a neural network model (e.g., a machine learning model) can be used to determine the activation state of the probe device. In some embodiments, the processor can further determine that the probe device is in an activated state after determining that the probe device (e.g., an ultrasonic probe) contacts the target anatomical feature.

[0061] At process 560, the activation state of the probe device automatically determined is provided by a processor (e.g., a processor of the control system 20). In some embodiments, the detection of the activation state (e.g., when switching between the activation state and the inactive state) can cause the probe device activation state message to be displayed to the operator through the display system. In some embodiments, if it is determined that the probe device 220 has an activation state, the method 400 can proceed to processes 430 to 490 to receive probe data from the probe device 220 and perform registration from the anatomical model to the imaging data based on the probe data. In some embodiments, if it is determined that the probe device 220 has an inactive state, the method 400 can wait until it is detected that the probe device 220 is in an activated state.

[0062] Referring to the example of FIG. 6 , a simplified diagram of a method 600 for registering probe data to image data is shown according to some embodiments. Figure 4AConsistent with some embodiments, method 600 may generally correspond to process 440 of method 400. As described in detail below, an initial registration between probe device space and imaging device space may be determined based on the kinematics of the imaging device and a medical instrument controlling the probe device, and the initial registration may be fine-tuned based on probe device information in the image data.

[0063] At process 610, an initial registration is estimated from the probe device space to the imaging device space based on the kinematics of the imaging device and / or the medical device, and a projection of the probe device relative to the image data is predicted based on the initial registration. In some embodiments, the initial registration includes an initial transformation with one or more initial transformation parameters (e.g., translation parameters, scale parameters, etc.). The initial registration including its registration parameters can be determined based on the kinematics of the imaging device (e.g., endoscope), the kinematics of the medical device, and the positioning and orientation of the probe device relative to the medical device (e.g., determined based on the posture of the end effector of the medical device that grasps the probe device).

[0064] refer to Figure 6B , shows a predicted projection 650 of a probe device based on an initial registration from the probe device space to the imaging device space. In an example, the predicted projection 650 of the probe device corresponds to an expected contour that the probe device is expected to occupy in the image data (eg, an endoscopic image) based on the initial registration.

[0065] At process 620, the predicted projection 650 of the probe device is compared with the actual projection of the probe device in the image data. An image alignment algorithm (e.g., a 2D or 3D image alignment algorithm) can be used to align the predicted projection 650 with the actual projection of the probe device 220 in the image data (e.g., the actual outline of the probe device).

[0066] At process 630, the initial alignment (including its alignment parameters) is updated to match the actual projection of the probe device based on the comparison of the predicted projection of the probe device. In an example, the alignment update algorithm iteratively updates one or more alignment parameters so that the predicted projection of the probe device based on the updated registration matches the actual projection of the probe device. In an example, an optimizer can be used to generate an alignment that minimizes a similarity measure (e.g., a scalar value describing similarity) between the predicted projection 650 and the actual projection of the probe device 220.

[0067] At process 640, the probe data is registered to the image data based on the updated registration (including updated registration parameters for the probe data).

[0068] Figure 7 is a simplified diagram of a method 700 for registering an anatomical model to probe data according to some embodiments. Figure 4AConsistent with some embodiments, method 700 may generally correspond to process 460 of method 400. As discussed in detail below, in method 700, a partial scan of the entire anatomical feature may be performed rather than a full scan to register the anatomical model to the probe data, which reduces computation time, provides faster feedback to the operator, and conserves computational power.

[0069] At process 710, one or more target attributes of the anatomical feature (e.g., probe regions 246 and / or 248) are identified based on the anatomical model. In an example, the target attributes may include one or more segmented features (also referred to as anatomical feature elements (e.g., the vasculature of a kidney, one or more specific vessel bifurcations in the vasculature, etc.) of the anatomical feature based on the anatomical model.

[0070] At process 720, probe data of a particular portion of an anatomical feature (e.g., probe regions 246 and / or 248) is obtained using a probe device (e.g., by instructing an operator to scan the probe region). A 2D or 3D image of the probe region from the probe data is used to identify corresponding target attributes in the probe data and the anatomical model. Various image processing algorithms may be used to map features in the probe data and the anatomical model.

[0071] At process 730 , the anatomical model is registered to the probe data based on one or more target attributes identified in the probe data and corresponding target attributes in the anatomical model.

[0072] Some examples of processors may include non-transitory, tangible, machine-readable media that include executable code that, when executed by one or more processors, may cause the one or more processors to perform the processes of methods 400-700. Some common forms of machine-readable media that may include the processes of methods 400-700 are, for example, floppy disks, diskettes, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with patterns of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cassettes, and / or any other media that a processor or computer is suitable for reading.

[0073] Although illustrative embodiments have been shown and described, a wide range of modifications, changes and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without the corresponding use of other features. Those of ordinary skill in the art will recognize many variations, substitutions and modifications. Therefore, the scope of the present invention should be limited only by the appended claims, and the claims should be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein, as appropriate.

Claims

1. A medical system, comprising: One or more processors configured to perform operations comprising: receiving image data associated with an anatomical feature of a patient from a first imaging device, the image data being spatially associated with the first imaging device; obtaining probe data associated with the anatomical feature from a second imaging device, the probe data being spatially associated with the second imaging device; registering the probe data to the image data to generate a first registration between the second imaging device space and the first imaging device space; registering an anatomical model associated with the anatomical feature to the probe data to generate a second registration between a model space of the anatomical model and the second imaging device space; and generating a third registration between the model space and the first imaging device space based on the first registration and the second registration; Wherein registering the probe data to the image data to generate the first registration between the second imaging device space and the first imaging device space comprises: determining an initial registration between the second imaging device space and the first imaging device space based on the kinematics of the first imaging device and a medical device controlling the second imaging device; generating a predicted projection of the second imaging device from the second imaging device space and the first imaging device space based on the initial registration; comparing the predicted projection of the second imaging device with an actual projection of the second imaging device in the image data; and The first registration is generated based on the initial registration and the comparison of the predicted projection of the second imaging device with the actual projection of the second imaging device.

2. The medical system of claim 1, wherein the operations further comprise: transforming the anatomical model in the model space to the first imaging device space based on the third registration; as well as A visual representation of the anatomical model transformed to the first imaging device space is provided on a display.

3. The medical system of claim 2, wherein the visual representation of the anatomical model transformed to the first imaging device space is superimposed on the image data.

4. The medical system of claim 1, wherein the second imaging device is an accessory device lacking independent movement capability.

5. The medical system according to claim 1, further comprising: A manipulator is configured to control the second imaging device by controlling an end effector to grasp and then manipulate the second imaging device to obtain the probe data using the second imaging device.

6. The medical system of claim 5, wherein obtaining the probe data associated with the anatomical feature from the second imaging device comprises: The second imaging device is controlled by the manipulator to scan at least a portion of the anatomical feature.

7. The medical system of claim 5, wherein the operations include: determining that the second imaging device is in an activated state; as well as In response to determining that the second imaging device is in an active state, the probe data is registered to the image data to generate the first registration.

8. The medical system of claim 7, wherein determining that the second imaging device is in the activated state comprises: The second imaging device is automatically determined to be in the activated state based on a probe device activity signal associated with the second imaging device.

9. The medical system of claim 7, wherein determining that the second imaging device is in the activated state comprises: The second imaging device is automatically determined to be in the activation state based on a trained decision tree associated with the activation state of the second imaging device.

10. The medical system of claim 7, wherein determining that the second imaging device is in the activated state comprises: The second imaging device is automatically determined to be in the activation state based on a neural network model associated with the activation state of the second imaging device.

11. The medical system of claim 1, wherein the predicted projection of the second imaging device corresponds to an expected contour that the second imaging device is expected to occupy in the image data.

12. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which, when executed by one or more processors associated with a medical device, are adapted to cause the one or more processors to perform a method comprising: receiving image data associated with an anatomical feature of a patient from a first imaging device, the image data being spatially associated with the first imaging device; obtaining probe data associated with the anatomical feature from a second imaging device, the probe data being spatially associated with the second imaging device; registering the probe data to the image data to generate a first registration between the second imaging device space and the first imaging device space; registering an anatomical model associated with the anatomical feature to the probe data to generate a second registration between a model space of the anatomical model and the second imaging device space; as well as generating a third registration between the model space and the first imaging device space based on the first registration and the second registration; Wherein registering the probe data to the image data to generate the first registration between the second imaging device space and the first imaging device space comprises: determining an initial registration between the second imaging device space and the first imaging device space based on the kinematics of the first imaging device and a medical device controlling the second imaging device; generating a predicted projection of the second imaging device from the second imaging device space and the first imaging device space based on the initial registration; comparing the predicted projection of the second imaging device with an actual projection of the second imaging device in the image data; and The first registration is generated based on the initial registration and the comparison of the predicted projection of the second imaging device with the actual projection of the second imaging device.

13. The non-transitory machine-readable medium of claim 12, wherein the method further comprises: transforming the anatomical model in the model space to the first imaging device space based on the third registration; as well as A visual representation of the anatomical model transformed to the first imaging device space is provided on a display.

14. The non-transitory machine-readable medium of claim 13, wherein the visual representation of the anatomical model transformed to the first imaging device space is superimposed on the image data.

15. The non-transitory machine-readable medium of claim 12, wherein the second imaging device is an accessory device lacking independent movement capability.

16. The non-transitory machine-readable medium of claim 12, wherein the method further comprises: The second imaging device is controlled by a manipulator by controlling an end effector to grasp and then manipulate the second imaging device to obtain the probe data using the second imaging device.

17. The non-transitory machine-readable medium of claim 16, wherein obtaining the probe data associated with the anatomical feature from the second imaging device comprises: The second imaging device is controlled by the manipulator to scan at least a portion of the anatomical feature.

18. The non-transitory machine-readable medium of claim 16, wherein the method further comprises: determining that the second imaging device is in an activated state; as well as In response to determining that the second imaging device is in an active state, the probe data is registered to the image data to generate the first registration.

19. The non-transitory machine-readable medium of claim 18, wherein determining that the second imaging device is in the activated state comprises: Based on a probe device activity signal associated with the second imaging device, it is automatically determined that the second imaging device is in the activated state.

20. The non-transitory machine-readable medium of claim 18, wherein determining that the second imaging device is in the activated state comprises: Based on a trained decision tree associated with an activation state of the second imaging device, it is automatically determined that the second imaging device is in the activation state.

21. The non-transitory machine-readable medium of claim 18, wherein determining that the second imaging device is in the activated state comprises: Based on a neural network model associated with the activation state of the second imaging device, it is automatically determined that the second imaging device is in the activation state.

22. The non-transitory machine-readable medium of claim 16, wherein the predicted projection of the second imaging device corresponds to an expected contour that the second imaging device is expected to occupy in the image data.

Images