System and method for intelligent seed registration

Abstract

The present application relates to systems and methods for intelligent seed registration. The present application provides a method of registering an anatomical dataset used during a medical procedure. The method can include accessing a first model point set of a patient anatomy of interest, and intraoperatively acquiring a second model point set by visualizing a portion of the patient's anatomical surface with a visual probe. The method can further include extracting system information including kinematic information and / or setup information from a robotic arm of a medical system, and generating an initial seed transform based on the extracted system information. Thereafter, the method can include applying the initial seed transform to the first model point set, and generating a first registration between the first model point set and the second model point set to allow an operator to view and use both model and actual information.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 2018800125751 (PCT / US2018 / 023791), filed on March 22, 2018, entitled "System and Method for Smart Seed Registration".

[0002] Cross-references to related applications

[0003] This application claims the benefit of U.S. Provisional Application 62 / 474,866, filed March 22, 2017, which is incorporated herein by reference in its entirety. Technical Field

[0004] This disclosure relates to systems and methods for performing image-guided procedures, and more specifically to systems and methods for displaying pathological data of tissues sampled during image-guided procedures. Background Technology

[0005] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. These techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Clinicians can insert minimally invasive medical instruments (including surgical instruments, diagnostic instruments, therapeutic instruments, or biopsy instruments) through these natural openings or incisions to reach target tissue locations. To assist in reaching target tissue locations, the position and movement of the medical instruments can be correlated with preoperative or intraoperative images of the patient's anatomy. By correlating image-guided instruments with images, instruments can navigate natural or surgically created pathways in anatomical systems such as the lungs, colon, intestines, kidneys, heart, circulatory system, or the like. Traditional instrument tracking and reference systems may require the use of a patient pad during preoperative and surgical imaging and can interfere with the clinical environment or workflow. Systems and methods for performing image-guided surgery with minimal clinical disruption are needed. Summary of the Invention

[0006] Embodiments of the invention are best summarized by the claims appended to the specification.

[0007] However, this paper provides an exemplary method for registering anatomical datasets used in medical procedures. This method may include accessing a first set of model points representing patient anatomy of interest, acquiring a second set of model points intraoperatively using sensor data representing the anatomy of interest, and extracting system information including kinematic and / or setup information from a robotic arm of the medical system. A first registration between the model point set and the surface point set is then generated by a control system processor based on the extracted system information.

[0008] This paper provides another exemplary method for registering anatomical datasets used in medical procedures. This method may include accessing a first set of model points representing a patient anatomy of interest, and obtaining a second set of model points by visualizing a portion of the patient anatomy of interest using a visual probe. The method may further include extracting system information, including kinematic and / or setup information from a robotic arm of the medical system, and generating an initial seed transformation based on the extracted system information before generating the first registration. Subsequently, the method may include applying the initial seed transformation to the first set of model points, and generating a first registration between the first and second set of model points, allowing both model and real-world information to be viewed and used.

[0009] An exemplary medical system may include a robotic arm, a vision probe, and a control system. The robotic arm has multiple joints to allow movement of the robotic arm, the vision probe is coupled to the robotic arm such that the vision probe can move with the robotic arm, and the control system communicates with the robotic arm and the vision probe. The control system may be configured to extract system information including kinematic and / or setup information of the robotic arm from the medical system, and to generate a first registration between a first set of model points of the patient anatomy of interest and a second set of model points collected intraoperatively of a portion of the patient anatomy of interest based on the extracted system information.

[0010] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the disclosure rather than to limit its scope. In this regard, additional aspects, features, and advantages of the disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0011] The various aspects of this disclosure can be best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, in accordance with standard industry practice, the various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or decreased for clarity. Furthermore, reference numerals and / or letters may be repeated in various examples of this disclosure. Such repetition is for simplicity and clarity and does not, in itself, define relationships between the various embodiments and / or the configurations discussed.

[0012] Figure 1A This is a remote-operated medical system according to embodiments of the present disclosure.

[0013] Figure 1B A front view of various aspects of an exemplary remote-operated assembly according to an embodiment of the present disclosure is shown.

[0014] Figure 2AA medical device system utilizing various aspects of this disclosure is shown.

[0015] Figure 2B Another medical device system utilizing aspects of this disclosure is shown.

[0016] Figure 3A and Figure 3B This is a side view in patient coordinate space of a medical device mounted on an insert assembly according to an embodiment of the present disclosure.

[0017] Figure 3C This is a side view of the patient coordinate space including multiple instruments positioned for an exemplary procedure, according to embodiments of the present disclosure.

[0018] Figure 4 This is a flowchart illustrating a general method for registering a model to a corresponding anatomical structure to provide guidance in an image-guided medical procedure, according to embodiments of the present disclosure.

[0019] Figure 5 This is another flowchart illustrating a method for registering a model to a corresponding anatomical structure to provide guidance in an image-guided medical procedure, according to embodiments of the present disclosure.

[0020] Figure 6A This illustrates imaging via a visual probe according to various aspects of this disclosure. Figure 3C The organs.

[0021] Figure 6B Depicting various aspects of this disclosure Figure 6A The three-dimensional point set of the organ.

[0022] Figure 7 Showing the representation Figure 6A and Figure 6B Anatomical models of organs.

[0023] Figure 8A Describing various aspects according to this disclosure Figure 7 The relative orientation and orientation of the model and Figure 6B The point represents an organ.

[0024] Figure 8B Describing various aspects according to this disclosure Figure 7 The model and the rendering in the monitor Figure 6B Registration between organs.

[0025] These figures can be better understood by referring to the following detailed description. Detailed Implementation

[0026] In the following description, specific details of some embodiments consistent with this disclosure are set forth. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are illustrative and not restrictive. Other elements within the scope and spirit of this disclosure (although not specifically described herein) can be implemented by those skilled in the art. Furthermore, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments unless otherwise specifically described or if such one or more features render the embodiment inoperable.

[0027] In some cases, well-known methods, procedures, components, and circuits are not described in detail to avoid unnecessarily obscuring aspects of the embodiments.

[0028] This disclosure describes various instruments and parts thereof in the state of three-dimensional space. As used herein, the term "orientation" refers to the position of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along the Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or part of an object (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the orientation of an object or part of an object in at least one translational degree of freedom and the orientation of an object or part of an object in at least one rotational degree of freedom (up to six total degrees of freedom). As used herein, the term "shape" refers to a set of poses, orientations, or orientations measured along the object.

[0029] Figure 1A This is a simplified diagram of a remotely operated medical system 100 according to some embodiments. In some embodiments, the remotely operated medical system 100 may be adapted for, for example, surgical procedures, diagnostic procedures, treatment procedures, or biopsy procedures. As shown in FIG1, the medical system 100 typically includes a remote operating manipulator assembly 102 for operating a medical device 104 to perform various procedures on a patient P. The remote operating manipulator assembly 102 is mounted to or near the operating table T. A master assembly 106 allows an operator (e.g., a surgeon, clinician, or operator O as shown in FIG1) to observe the intervention site and control the remote operating manipulator assembly 102.

[0030] The master control assembly 106 can be located at the operator's console, which is typically located in the same room as the operating table T, such as to the side of the surgical table where the patient P is located. However, it should be understood that the operator O can be located in a different room from the patient P or in a completely different building. The master control assembly 106 typically includes one or more control devices for controlling the remote operating manipulator assembly 102. The control devices can include any number of various input devices, such as joysticks, trackballs, data gloves, trigger guns, manual controllers, voice recognition devices, body motion or presence sensors and / or the like. To provide the operator O with a strong sense of direct control of the instrument 104, the control devices can be provided with the same degrees of freedom as the associated medical instrument 104. In this way, the control devices provide the operator O with a telepresence or perception that the control devices and the medical instrument 104 are integrated.

[0031] In some embodiments, the control device may have more or fewer degrees of freedom than the associated medical device 104, and still provide remote presentation to the operator O. In some embodiments, the control device may optionally be a manual input device that moves in six degrees of freedom and may also include an actuable handle for actuating the device (e.g., for closing a gripping clamp, applying a potential to an electrode, delivering medication, and / or the like).

[0032] The remote-operated manipulator assembly 102 supports the medical device 104 and may include a kinematic structure of one or more non-servo-controlled linkages (e.g., one or more linkages that are manually positioned and locked in place, typically referred to as a setup structure) and a remote-operated manipulator. The remote-operated manipulator assembly 102 may optionally include a plurality of actuators or motors that drive input devices on the medical device 104 in response to commands from a control system (e.g., control system 112). The actuators may optionally include a drive system that, when coupled to the medical device 104, can advance the medical device 104 into a natural or surgically created anatomical orifice. Other drive systems may move the distal end of the medical device 104 with multiple degrees of freedom, which may include three linear degrees of motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three rotational degrees of motion (e.g., rotation about the X, Y, Z Cartesian coordinate axes). Additionally, the actuator can be used to actuate the articulated end effector of the medical device 104 for grasping tissue in the gripper of a biopsy apparatus and / or similar objects. Actuator orientation sensors, such as solvers, encoders, potentiometers, and other mechanisms, can provide the medical system 100 with sensor data describing the rotation and orientation of the motor shaft. This orientation sensor data can be used to determine the motion of the object manipulated by the actuator.

[0033] The remote-operated medical system 100 may include a sensor system 108 having one or more subsystems for receiving information about the instrument of the remote-operated manipulator assembly 102. Such subsystems may include: an orientation / position sensor system (e.g., an electromagnetic (EM) sensor system); a shape sensor system for determining orientation, orientation, rate, velocity, posture, and / or shape along a distal end and / or one or more segments that may constitute a flexible body of the medical device 104; and / or a visualization system for capturing images from the distal end of the medical device 104.

[0034] The remote-operated medical system 100 also includes a display system 110 for displaying images or representations of anatomical sites and medical devices 104 generated by a subsystem of the sensor system 108. The display system 110 and the master control assembly 106 can be configured such that an operator O can control the medical devices 104 and the master control assembly 106 using remotely presented perception.

[0035] In some embodiments, the medical device 104 may have a visualization system (discussed in more detail below) that may include an endoscope assembly that records concurrent or real-time images of the anatomical site and provides the images to an operator or operator O via one or more displays of the medical system 100 (such as one or more displays of the display system 110). The concurrent images may be, for example, two-dimensional or three-dimensional images captured by an endoscope positioned within the anatomical site. In some embodiments, the visualization system includes an endoscope component that may be integrally or removably coupled to the medical device 104. However, in some embodiments, a separate endoscope attached to a separate manipulator assembly may be used with the medical device 104 to image the anatomical site. In some examples, the endoscope may include one or more mechanisms for cleaning one or more lenses of the endoscope when one or more lenses are partially and / or completely blocked by fluid and / or other materials encountered by the endoscope. In some examples, the one or more cleaning mechanisms may optionally include an air and / or other gas delivery system that can be used to fire a stream of air and / or other gas to clean one or more lenses. Examples of one or more cleaning facilities are discussed in more detail in International Publication No. WO / 2016 / 025465 (filed August 11, 2016) ("Systems and Methods for Cleaning an Endoscopic Instrument"), which is incorporated herein by reference in its entirety. The visualization system can be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which may include the processor of the control system 112.

[0036] Display system 110 can also display images of anatomical sites and medical instruments captured by a visualization system. In some examples, remote-operated medical system 100 can configure controls for medical instrument 104 and master assembly 106 such that the relative orientation of the medical instrument is analogous to the relative orientation of the operator O's eyes and hands. In this way, operator O can manipulate medical instrument 104 and hand controls as if observing a workspace in a substantially real-world setting. In terms of real-world presence, this means that the image presentation is a realistic perspective image simulating the viewpoint of an operator physically manipulating medical instrument 104.

[0037] In some examples, the display system 110 may use image data from imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescence microscopy, temperature recording, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar techniques to present images of anatomical sites recorded preoperatively or intraoperatively. The preoperative or intraoperative image data may be presented as two-dimensional, three-dimensional, or four-dimensional images (including, for example, time-based or velocity-based information) and / or as images from models created from the preoperative or intraoperative image dataset.

[0038] In some embodiments, typically for the purpose of image-guided medical procedures, the display system 110 may display a virtual navigation image in which the actual position of the medical device 104 is registered with preoperative or real-time images / models (i.e., dynamic reference). This allows a virtual image of the internal anatomical location to be presented to the operator O from the perspective of the medical device 104. In some examples, the perspective may be from the tip of the medical device 104. Images and / or other graphic indicators or alphanumeric indicators of the tip of the medical device 104 may be overlaid on the virtual image to assist the operator O in controlling the medical device 104. In some examples, the medical device 104 may be invisible in the virtual image.

[0039] In some embodiments, display system 110 may display a virtual navigation image in which the actual position of medical device 104 is registered with preoperative or live images to present a virtual image of medical device 104 within the anatomical site to operator O from an external perspective. Images of portions of medical device 104 or other graphic indicators or alphanumeric indicators may be overlaid on the virtual image to assist operator O in controlling medical device 104. As described herein, a visual representation of data points may be rendered to display system 110. For example, measurement data points, movement data points, registration data points, and other data points described herein may be displayed on display system 110 as a visual representation. Data points may be visually represented on the user interface as multiple points or dots on display system 110 or as a rendering model (such as a grid or line model created based on the data point set). In some examples, data points may be color-coded according to the data they represent. In some embodiments, the visual representation may be refreshed in display system 110 after each processing operation has been implemented to change the data points.

[0040] The remote-operated medical system 100 may also include a control system 112. The control system 112 includes at least one memory and at least one computer processor (not shown) for implementing control between the medical device 104, the main control assembly 106, the sensor system 108, and the display system 110. The control system 112 also includes programming instructions (e.g., a non-transitory machine-readable medium storing these instructions) for implementing some or all of the methods described according to aspects disclosed herein, including instructions for providing information to the display system 110. Although the control system 112 is shown as a single block in the simplified schematic of FIG1, the system may include two or more data processing circuits, with some processing optionally performed on or near the remote-operated manipulator assembly 102, and other processing performed at the main control assembly 106 and / or the like. The processor of the control system 112 may execute instructions including those corresponding to the processing disclosed herein and described in more detail below. Any of a variety of centralized or distributed data processing architectures may be employed. Similarly, programming instructions can be implemented as multiple separate programs or subroutines, or they can be integrated into many other aspects of the remote operating system described herein. In one embodiment, the control system 112 supports wireless communication protocols such as Bluetooth, IrDA (Infrared Data Communication), HomeRF (Home Radio Frequency), IEEE 802.11, DECT (Digital Enhanced Wireless Communication), and wireless telemetry.

[0041] In some embodiments, the control system 112 may receive force and / or torque feedback from the medical device 104. In response to the feedback, the control system 112 may transmit signals to the master assembly 106. In some examples, the control system 112 may transmit signals instructing one or more actuators of the remote-operated manipulator assembly 102 to move the medical device 104. The medical device 104 may extend through an opening in the patient P's body to an internal anatomical location within the patient P's body. Any suitable conventional and / or specialized actuators may be used. In some examples, one or more actuators may be separate from or integrated with the remote-operated manipulator assembly 102. In some embodiments, one or more actuators and the remote-operated manipulator assembly 102 are provided as part of a remote-operated trolley positioned adjacent to the patient P and the operating table T.

[0042] The control system 112 may optionally further include a virtual visualization system to provide navigational assistance to the operator O when controlling the medical device 104 during an image-guided medical procedure. Virtual navigation using the virtual visualization system may be based on a reference to a preoperative or intraoperative dataset of the acquired anatomical passage. The virtual visualization system processes images of the anatomical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescence microscopy, temperature recording, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar techniques. Software, which can be used in conjunction with manual input, is used to convert the recorded images into segmented two-dimensional or three-dimensional composite representations of parts or whole anatomical organs or regions. The image dataset is associated with the composite representation. The composite representation and the image dataset describe the various locations and shapes of the passage and their connectivity. The images used to generate the composite representation may be recorded preoperatively or intraoperatively during the clinical procedure. In some embodiments, the virtual visualization system may use a standard representation (i.e., not patient-specific) or a mixture of a standard representation and patient-specific data. Composite representations and any virtual images generated from composite representations can represent the static posture of deformable anatomical regions during one or more phases of motion (e.g., during the inspiratory / expiratory cycle of the lungs).

[0043] During the virtual navigation procedure, sensor system 108 can be used to calculate the approximate position of medical device 104 relative to the anatomy of patient P. This position can be used to generate both a macroscopic (external) tracking image of the anatomy of patient P and a virtual internal image of the anatomy of patient P. The system can implement one or more electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors to register and display anatomical images of the medical device and preoperative records, such as those from virtual visualization systems. For example, U.S. Patent Application No. 13 / 107,562 (filed May 13, 2011) (disclosing “Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery”) discloses such a system, which is incorporated herein by reference in its entirety. The remote-operated medical system 100 may further include optional operating and support systems (not shown), such as lighting systems, steering control systems, flushing systems, and / or suction systems. In some embodiments, the remote-operated medical system 100 may include more than one remote-operated manipulator assembly and / or more than one master control assembly. The exact number of remote-controlled manipulator assemblies will depend on the medical procedure, space constraints within the operating room, and other factors. Master control assemblies 106 can be juxtaposed or positioned in different locations. Multiple master control assemblies allow more than one operator to control one or more remote-controlled manipulator assemblies in various combinations.

[0044] Figure 1B yes Figure 1A The diagram shows a front view of some embodiments of the remote-operated assembly 102. Assembly 102 includes a base 120 resting on a floor, a support tower 122 mounted on the base 120, and several arms supporting medical instruments. Figure 1B As shown, arms 124A, 124B, and 124C are instrument arms that support and move medical instruments used to manipulate tissues, and arm 124D is a camera arm that supports and moves an endoscope. Figure 1B Interchangeable medical devices 104A, 104B, and 104C are further shown mounted on instrument arms 124A, 124B, and 124C, respectively, and an imaging system 126 is shown mounted on a camera arm 124D. The imaging system 126 can be used to capture stereoscopic images of anatomical sites and provide individual stereoscopic images to... Figure 1AThe display system 110 is a stereoscopic endoscope. Other embodiments of the imaging system 126 are discussed herein. Those skilled in the art will understand that the arms supporting the instruments and cameras may also be supported by the base 120 or by another base platform (fixed or movable) mounted to the ceiling or wall or, in some cases, to another device (e.g., operating table T) in the operating room. Similarly, they will understand that two or more separate bases may be used (e.g., one base supporting each arm).

[0045] like Figure 1B As further shown, instruments 104A, 104B, 104C, and imaging system 126 each include instrument interfaces 130A, 130B, 130C, and 130D, and instrument axes 132A, 132B, 132C, and 132D, respectively. In some embodiments, the remote operation assembly 102 may include a support for the cannula, which secures instruments 104A, 104B, 104C, and imaging system 126 relative to the cannula. In some embodiments, portions of each of the instrument arms 124A-124D may be adjustable by personnel in the operating room to position instruments 104A-104C and imaging system 126 relative to the patient. Other portions of arms 124A, 124B, 124C, and 124D may be actuated and controlled by an operator at an operator input system. Medical devices 104A, 104B, 104C and imaging system 126 can also be controlled by operator O at operator input system (such as master control assembly 106).

[0046] Figure 2A The illustration shows a medical device system 200, which may be an embodiment of medical device system 104 and / or manipulator assembly 102 in an image-guided medical procedure performed using some embodiments of a remotely operated medical system 100. Alternatively, the medical device system 200 may be used for non-remotely operated exploratory procedures or procedures involving conventionally manually operated medical devices (such as endoscopes). Additionally or alternatively, the medical device system 200 may be used to collect (i.e., measure) a set of data points corresponding to the location of the patient's anatomical access.

[0047] Device system 200 includes a catheter system 202 coupled to a device body or housing 204. Catheter system 202 includes an elongated, flexible catheter body 216 having a proximal end 217 and a distal end 218 or tip portion. In one embodiment, the flexible body 216 has an outer diameter of approximately 3 mm. Other flexible bodies may have larger or smaller outer diameters. Catheter system 202 may optionally include a shape sensor 222 for determining the orientation, rate, velocity, posture, and / or shape of the catheter tip at the distal end 218, and / or determining the orientation, orientation, rate, velocity, posture, and / or shape along one or more segments 224 of the body 216. The entire length of the body 216 between the distal end 218 and the proximal end 217 can be effectively divided into segments 224. If device system 200 is a medical device system 104 of a remotely operated medical system 100, the shape sensor 222 may be a component of sensor system 108. If the instrument system 200 is manually operated or otherwise used for a non-remote operation procedure, the shape sensor 222 can be coupled to the tracking system 230, which queries the shape sensor and processes the received shape data.

[0048] The shape sensor 222 may include an optical fiber aligned with the flexible conduit body 216 (e.g., provided within an internal channel (not shown) or mounted externally). In one embodiment, the optical fiber has a diameter of approximately 200 μm. In other embodiments, the size may be larger or smaller. The optical fiber of the shape sensor system 222 forms an optical fiber bending sensor for determining the shape of the conduit system 202. In an alternative, an optical fiber including a fiber Bragg grating (FBG) is used to provide strain measurements in one or more dimensions of the structure. Various systems and methods for three-dimensional monitoring of the shape and relative orientation of optical fibers are described in U.S. Patent Application 11 / 180,389 (filed July 13, 2005) (disclosing "Fiber optic position and shape sensing device and method relating thereto"), U.S. Patent Application 12 / 047,056 (filed July 16, 2004) (disclosing "Fiber-optic shape and relative position sensing"), and U.S. Patent No. 6,389,187 (filed June 17, 1998) (disclosing "Optical Fibre Bend Sensor"), all of which are incorporated herein by reference in their entirety. In alternative embodiments, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In other alternative embodiments, the shape of the catheter may be determined using other techniques. For example, the history of the distal tip posture of the catheter may be used to reconstruct the shape of the device over time intervals. As another example, historical posture, orientation, or orientation data for known points of the instrument system along alternating cycles of motion (such as breathing) may be stored. The stored data can be used to develop information about the shape of the catheter. Alternatively, a series of orientation sensors (such as electromagnetic (EM) sensors) positioned along the catheter can be used for shape sensing. Alternatively, particularly if the anatomical passage is generally static, the data history from orientation sensors (such as EM sensors) on the instrument system during the procedure can be used to represent the shape of the instrument. Alternatively, a wireless device with orientation or orientation controlled by an external magnetic field can be used for shape sensing. The orientation history of the wireless device can be used to determine the shape of the navigation passage.

[0049] Optionally, the medical device system may include an orientation sensor system 220. The orientation sensor system 220 may be a component of an EM sensor system, wherein the sensor 220 includes one or more conductive coils capable of withstanding an externally generated electromagnetic field. Each coil of the EM sensor system 220 then generates an induced electrical signal having characteristics dependent on the orientation and orientation of the coil relative to the externally generated electromagnetic field. In one embodiment, the EM sensor system may be configured and positioned to measure six degrees of freedom (e.g., three orientation angles of pitch, yaw, and roll for a reference point) or five degrees of freedom (e.g., three orientation angles of pitch and yaw for a reference point). Further description of an EM sensor system is provided in U.S. Patent No. 6,380,732 (filed August 11, 1999) (disclosing "Six-Degree of FreedomTracking System Having a Passive Transponder on the Object Being Tracked"), which is incorporated herein by reference in its entirety. In some embodiments, the shape sensor can also be used as an orientation sensor because the shape of the sensor, together with the positional information of the base of the shape sensor (in the patient’s fixed coordinate system), allows for position calculations along the various points of the shape sensor (including the distal tip).

[0050] Tracking system 230 may include orientation sensor system 220 and shape sensor system 222 for determining the orientation, orientation, velocity, posture, and / or shape of distal end 218 and / or one or more segments 224 along instrument 200. Tracking system 230 may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which may include the processor of control system 116.

[0051] The flexible catheter body 216 includes a channel 221, which is sized and shaped to receive a medical device 226. The medical device may include, for example, an image capture probe, a biopsy instrument, a laser ablation fiber or other surgical tool, a diagnostic tool, a surgical diagnostic tool, and / or a therapeutic tool. The medical tool may include an end effector having a single working component, such as a scalpel, a blunt blade, an optical fiber, or an electrode. Other end effectors may include, for example, forceps, graspers, scissors, or applicators. Examples of electrically activated end effectors include electrosurgical electrodes, transducers, sensors, and the like. In various embodiments, the medical device 226 may be an image capture probe including a distal portion of the flexible catheter body 216 having a stereo or single-field-of-view camera at or near its distal end 218 for capturing images (including video images) processed by the visualization system 231 for display. The image capture probe may include a cable coupled to the camera for transmitting the captured image data. Alternatively, the image capture device may be a bundle of optical fibers (such as a fiber endoscope) coupled to the visualization system. Image capture devices can be single-spectral or multi-spectral, for example, capturing image data in one or more of the visible, infrared, or ultraviolet spectra.

[0052] Medical device 226 may accommodate a cable, linkage, or other actuation control (not shown) extending between the proximal and distal ends of the device to controllably bend the distal end of the device. Steering devices are described in detail in U.S. Patent No. 7,316,681 (filed October 4, 2005) (disclosing “Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity”) and U.S. Patent Application No. 12 / 286,644 (filed September 30, 2008) (disclosing “Passive Preload and Capstan Drive for Surgical Instruments”), both of which are incorporated herein by reference in their entirety.

[0053] The flexible catheter body 216 may also accommodate a cable, linkage, or other steering control (not shown) extending between the housing 204 and the distal end 218 to controllably bend the distal end 218, as illustrated by the dashed line 219 depicted by the distal end. Steering catheters are described in detail in U.S. Patent Application No. 13 / 274,208 (filed October 14, 2011) (disclosing “Catheter with Removable Vision Probe”), which is incorporated herein by reference in its entirety. In embodiments where the instrument system 200 is actuated by a remotely operated assembly, the housing 204 may include a drive input device removably coupled to and receiving power from a motor drive element of the remotely operated assembly. In embodiments where the instrument system 200 is manually operated, the housing 204 may include clamping features, a manual actuator, or other components for manually controlling the movement of the instrument system. The catheter system may be steerable, or alternatively, the system may be non-steerable without an integrated mechanism for operator-controlled bending of the instrument. Alternatively, one or more lumens may be confined within the wall of the flexible body 216, allowing the medical device to be deployed and used at a target anatomical location via one or more lumens.

[0054] In various embodiments, the medical device system 200 may include flexible bronchial instruments, such as bronchoscopes or bronchial tubes for use in the examination, diagnosis, biopsy, or treatment of the lungs. System 200 is also suitable for navigating and treating other tissues via naturally or surgically created connecting channels in any of any of the various anatomical systems, including the colon, intestine, kidneys and renal calyces, brain, heart, circulatory system including the vascular system, and / or the like.

[0055] Information from tracking system 230 can be sent to navigation system 232, where it is combined with information from visualization system 231 and / or a preoperatively acquired model to provide real-time orientation information to the surgeon or other operator on display system 100 for controlling instrument 200. Control system 116 can utilize the orientation information as feedback for locating instrument 200. Various systems for registering and displaying images of medical devices and surgical procedures using fiber optic sensors are provided in U.S. Patent Application No. 13 / 107,562, filed May 13, 2011 (disclosing “Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery”), which is incorporated herein by reference in its entirety.

[0056] exist Figure 2A In one embodiment, the device 200 is remotely operated within the remote-operated medical system 100. In an alternative embodiment, the remote-operation assembly 102 may be replaced by a direct operator control. In direct-operation alternatives, various handles and operator interfaces may be included for handheld operation of the device.

[0057] In alternative embodiments, the remote operating system may include one or more slave manipulator assemblies and / or one or more master manipulator assemblies. The exact number of manipulator assemblies will depend on the medical procedure and space constraints within the operating room, as well as other factors. Master manipulator assemblies may be juxtaposed or positioned in different locations. Multiple master manipulator assemblies allow more than one operator to control one or more slave manipulator assemblies in various combinations.

[0058] Figure 2B A medical device system 250 is shown, which can be used with, for example Figure 1B The illustrated embodiment uses the remotely operated medical system 100 as a medical device system 104 in medical procedures. Alternatively or additionally, the medical device system 250 may be used to collect (i.e., measure) a set of data points corresponding to the locations of patient anatomical passages and surfaces of patient anatomical features (such as organs or body cavities). Figure 2BThis is a perspective view of the manipulator 252, which can be mounted onto assembly 102. For clarity, the sterile drape and associated mechanisms typically used during surgical procedures are omitted. The manipulator 252 includes a deflection servo actuator 254, a pitch servo actuator 256, and an insertion and withdrawal (“I / O”) actuator 258. The medical device 259 is shown mounted on an instrument spar 260, which includes a mounting bracket 261. An exemplary straight cannula 262 is shown mounted to a cannula mount 264. Other types of cannulas may be used, as discussed in more detail below. The axis 266 of the instrument 259 extends through the cannula 262. The manipulator 252 is mechanically constrained such that it moves the instrument 259 about a stationary remote center of motion 268 positioned along the instrument axis. Deflection actuator 254 provides deflection motion 270 about remote center 268, pitch actuator 256 provides pitch motion 272 about remote center 268, and I / O actuator 258 provides insertion and withdrawal motion 274 through remote center 268. Manipulator 252 may include encoders for tracking azimuth and velocity associated with servo azimuth along the insertion axis of I / O actuator 258, and additional encoders for tracking the azimuth and velocity of deflection servo actuator 254 and pitch servo actuator 256. During surgery, remote center 268 may be locked at the incision site in the patient's body wall and allows sufficient deflection and pitch motion to be used to perform the intended medical task. Alternatively, the remote motion center may be located externally to allow a greater range of motion without contact with the patient. Those skilled in the art will understand that motion about the remote motion center may be limited by software use or by physical constraints defined by mechanical assembly.

[0059] The matching force transmission disc in mounting bracket 261 and instrument force transmission assembly 276 is coupled with actuation force from actuators in manipulator 252 to move various parts of instrument 259 to position and orient probe 278 mounted at the distal end of arcuate shaft 266. This actuation force typically causes instrument shaft 266 to roll (thus providing another degree of freedom (DOF) via remote center 268). The amount of rolling can be tracked via encoder. Embodiments of the force transmission assembly are provided in U.S. Patent No. 6,331,191 (filed October 15, 1999; disclosing “Surgical Robotic Tools, Data Architecture, and Use”) and U.S. Patent No. 6,491,701 (filed January 12, 2001; disclosing “Mechanical Actuator Interface System for Robotic Surgical Tools”), which are incorporated herein by reference in their entirety. In an alternative embodiment, instrument 259 may include a wrist at the distal end of the shaft, which provides additional deflection and pitch degrees of freedom. The probe 278 may be, for example, a visual probe that can be introduced and positioned via the manipulator 252, such as a stereo imaging conduit with a stereo camera or a three-dimensional structured light scanner.

[0060] Some embodiments of the instrument systems within the scope of this disclosure include combinations. Figure 2A The instrument system 200 in all aspects and Figure 2B The instrument system 250 includes instruments in various aspects. For example, some instrument systems that can be used in medical system 100 may include flexible catheters, such as those made of... Figure 1A and Figure 2B One or more of the arms 124A-124D supported Figure 2A The flexible catheter body 216. As a more specific example, an arm such as arm 124A can provide insertion and withdrawal movements 274 for a flexible device such as the flexible catheter body 216. In such an embodiment, the control system 112 can use sensor information from sensors disposed on, in, or along the arm or arms, and sensor information from sensors disposed on, in, or along the catheter to determine the orientation and orientation of any portion of the catheter body 216 (such as the distal end of the catheter body 216, which may include an imaging component or another end effector).

[0061] Figure 3A and Figure 3B This is a simplified side view of a medical device mounted on an insert assembly, according to some embodiments, in patient coordinate space. Figure 3A and Figure 3BAs shown, the surgical environment 300 includes a patient P positioned on the operating table T of FIG1. ​​Patient P may be stationary within the surgical environment in the sense of limiting overall patient movement through sedation, restraint, and / or other means. Periodic anatomical movements, including the breathing and cardiac movements of patient P, may continue unless the patient is instructed to hold their breath to temporarily cease these movements. Therefore, in some embodiments, data may be collected at specific phases of breathing and used for tagging and identification. In some embodiments, the phase in which data is collected may be inferred from physiological information collected from patient P. Within the surgical environment 300, a point collection device 304 is coupled to an instrument holder 306. In some embodiments, the point collection device 304 may use an EM sensor modality, a shape sensor modality, and / or other sensor modalities. The instrument holder 306 is mounted to an insertion stage 308 fixed within the surgical environment 300. Alternatively, the insertion stage 308 may be movable but has a known location within the surgical environment 300 (e.g., via a tracking sensor or other tracking device). The instrument holder 306 may be a component of a remotely operated manipulator assembly (e.g., remotely operated manipulator assembly 102) coupled to a point-collecting instrument 304 to control insertion motion (i.e., movement along axis A) and optionally control the movement of the distal end 318 of the elongated device 310 in multiple directions, including yaw, pitch, and roll. The instrument holder 306 or insertion stage 308 may include an actuator (not shown), such as a servo motor, that controls the movement of the instrument holder 306 along the insertion stage 308.

[0062] An elongated device 310 is coupled to an instrument body 312. The instrument body 312 is coupled and fixed relative to an instrument holder 306. In some embodiments, a fiber optic shape sensor 314 is fixed at a proximal point 316 on the instrument body 312. In some embodiments, the proximal point 316 of the fiber optic shape sensor 314 may move with the instrument body 312, but the position of the proximal point 316 may be known (e.g., known via a tracking sensor or other tracking device). The shape sensor 314 measures the shape from the proximal point 316 to another point (such as the distal end 318 of the elongated device 310). The point collection device 304 may be substantially similar to Figure 2A The medical device system 200. Alternatively, the point collection device itself may be a rigid device coupled to a proximal rigid device body or a flexible catheter that can be actuated into a rigid state.

[0063] As the orientation measuring device 320 moves along the insertion axis A on the insertion stage 308, it provides information about the orientation of the instrument body 312. The orientation measuring device 320 may include a solver, encoder, potentiometer, and / or other sensors that determine the rotation and / or orientation of actuators that control the movement of the instrument holder 306 and thus the movement of the instrument body 312. In some embodiments, the insertion stage 308 is linear. In some embodiments, the insertion stage 308 may be arcuate or have a combination of arcuate and linear segments.

[0064] Figure 3A The instrument body 312 and instrument holder 306 are shown in their retracted position along the insertion stage 308. In this retracted position, the proximal point 316 is located at orientation L0 on axis A. In this position along the insertion stage 308, the positional component of the proximal point 316 can be set to zero and / or another reference value to provide a reference describing the orientation of the instrument holder 306 and thus the orientation of the proximal point 316 on the insertion stage 308. With this retracted position of the instrument body 312 and instrument holder 306, the distal end 318 of the elongated device 310 can be precisely positioned within the access port of the patient P. Also in this position, the orientation measuring device 320 can be set to zero and / or another reference value (e.g., I = 0). Figure 3B In this configuration, the instrument body 312 and instrument holder 306 have been advanced along the linear track of the insertion stage 308, and the distal end 318 of the elongated device 310 has been advanced into the patient P. In this advancement orientation, the proximal point 316 is located at orientation L1 on axis A. In some examples, encoders and / or one or more actuators controlling the movement of the instrument holder 306 along the insertion stage 308 and / or other orientation data from one or more orientation sensors associated with the instrument holder 306 and / or the insertion stage 308 are used to determine the orientation L of the proximal point 316 relative to orientation L0. x In some examples, the azimuth L x It can be further used as an indicator of the distance or insertion depth of the distal end 318 of the elongated device 310 inserted into the anatomical body of the patient P.

[0065] Figure 3C This is a perspective view of a patient P undergoing a medical procedure according to some aspects of this disclosure. As shown, the medical procedure is liver surgery. The patient P's right arm RA and abdomen A are shown transparently so that the liver L can be observed more clearly. Figure 3C Further show Figure 1BArms 124A-124D, wherein the instrument shafts 132A-132C of medical devices 130A-130C protrude into the abdomen A. To better access the medical devices 130A-130C within the abdomen A, one of the devices 130A-130C may provide injection into the abdomen A. Figure 3C An imaging probe 136 is depicted inserted into an imaging system 126 within the abdomen A to provide visualization and / or registration of organs and tissues within the abdomen A. Specifically, the imaging probe 136 has been positioned and oriented by an arm 124D to image at least a portion of the liver L. The imaging probe 136 may include one or more sensors providing one or more imaging modalities. For example, the imaging probe 136 may include a stereo imaging camera, a structured light emitter, a LiDAR emitter / detector system, or a combination thereof.

[0066] Figure 4 This is a flowchart illustrating a general method 400 for image-guided medical procedures. In process 402, preoperative or intraoperative image data is acquired from imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescence microscopy, temperature recording, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, or nanotube X-ray imaging. The preoperative or intraoperative image data can correspond to two-dimensional, three-dimensional, or four-dimensional images (including, for example, time-based or velocity-based information). For example, the image data can represent any anatomical structure of patient P. For example, the image data can represent the internal passages of the lungs or the outer surface of the liver.

[0067] In process 404, computer software, alone or in combination with manual input, is used to transform recorded images into segmented two-dimensional or three-dimensional composite representations or models of parts or entire anatomical organs or regions. The composite representations and image datasets describe the various locations and shapes of the channels and their connectivity. More specifically, during the segmentation process, the image is divided into segments or elements (e.g., pixels or voxels) sharing certain features or computational properties (such as color, density, intensity, and texture). This segmentation process results in two-dimensional or three-dimensional reconstruction, which forms a model of the target anatomy based on the acquired image. To represent the model, the segmentation process may delineate a set of voxels representing the target anatomy and then apply a function (such as a marching cube function) to generate a 3D surface surrounding the voxels. The model can be created by generating a mesh, volume, or voxel map. Additionally or alternatively, the model may include a centerline model comprising a set of interconnected line segments or point sets extending through the center of the modeled channel. In cases where the model includes a centerline model (comprising a set of interconnected line segments), these line segments can be converted into a point cloud or point set. By converting line segments, you can manually or automatically select the desired number of points corresponding to the interconnected line segments.

[0068] In process 406, before and / or during the image-guided medical procedure on the patient, anatomical model data is registered to the patient's anatomy. Typically, registration involves matching measurement points to points on the model using rigid and / or non-rigid transformations. Measurement points can be generated using the anatomy, electromagnetic coils scanned and tracked during the procedure, or landmarks in a shape sensor system. Measurement points can be generated for use in… Figure 5 Or the Iterative Closest Point (ICP) technique described in detail elsewhere in this disclosure. Within the scope of this disclosure, other point set registration methods may also be used in the registration process.

[0069] Other registration methods for image-guided surgery typically involve the use of techniques based on electromagnetic or impedance sensing. Metallic objects or certain electronic devices used in the surgical environment can introduce interference that impairs the quality of the sensing data. Other registration methods may hinder clinical workflows. The systems and methods described below perform registration based on ICP or other point set registration algorithms and the calibration movement of point-collecting instruments with fiber optic shape sensors, thereby eliminating or minimizing interference in the surgical environment. Other registration techniques can be used to register measurement point sets to a preoperative model or a model obtained using another modality. In the embodiments described below, EM sensors on the patient and instruments, as well as the optical tracking system of the instruments, can be eliminated.

[0070] Figure 5This is a flowchart illustrating a method 500 for providing guidance to a clinician in an image-guided medical procedure for a patient P in the surgical setting shown in FIG1, according to some embodiments of the present disclosure. Method 500 in Figure 5 The diagram shows a set of boxes, steps, operations, or procedures. Not all of the listed operations shown are performed in all embodiments of method 500. Additionally, Figure 5 Some additional operations not explicitly shown may be included before, after, between, or as part of the listed processes. Some embodiments of method 500 include instructions corresponding to the processes of method 500 stored in memory. These instructions may be executed by a processor (such as the processor of control system 112).

[0071] Method 500 further includes providing operations for generating and applying an initial seed transformation based on dynamic information extracted from manipulator 252 or other arms with kinematic chains and / or based on workflow information associated with the procedure to be performed. Method 500 can be implemented as part of a workflow managed by control system 112 to enable operator O to treat patients such as patient P more effectively and efficiently.

[0072] Some embodiments of method 500 may begin at process 501, wherein a calibration procedure is performed to utilize an orientation measurement system (such as...) Figure 1B The system of encoders or shape sensors in arms 124A-124D is used to calibrate the relative orientation and / or orientation of an imaging system (such as imaging system 126). For example, measurements from the tip of imaging system 126 placed at one or more known locations in a surgical setting and from another orientation measurement system of the encoder or arm 124D can be observed to ensure the correspondence between the measured orientation or kinematic information and the actual orientation and posture of the arm 124D.

[0073] In operation 502, the control system 112 can extract and receive orientation and posture information associated with system 100. For example, the control system 112 can query encoders, potentiometers, and shape sensors located within one or more arms 124A-124D. For example, the control system 112 can receive kinematic information from arm 124D supporting the imaging system 126. This information can be received simultaneously with the operator O controlling arm 124D to provide visualization of the patient P, particularly visualization of the interventional site within the patient. Furthermore, the control system 112 can receive program settings and / or workflow information associated with the procedure to be performed. For example, the operator O can interact with a user interface to select a specific procedure such as a lung biopsy, prostatectomy, or liver surgery. These procedures, and many others, can have corresponding workflows stored in the control system 112. Each workflow can contain information including instructions for the procedure to be performed, the ideal proximity of the target anatomy, and a list of steps to be performed during the normal execution of such a procedure. The control system 112 can communicate the steps in the workflow to the operator O to help ensure the correct execution of the procedure and to provide the physician O with appropriate information and options at the appropriate time. Therefore, the control system 112 can receive information describing the posture of the arm 124D relative to the patient P, and information describing the type of procedure being performed and the steps of the procedure being performed at any given time.

[0074] In some embodiments, the kinematic information received at operation 502 can be received from a fiber optic shape sensor that extends through arm 124D and enters or passes through imaging probe 136. Therefore, the kinematic information can include a series of three-dimensional orientations of arm 124D or a model generated from measured angles and known or measured lengths of arm 124D. Furthermore, some embodiments of operation 502 can receive information from a flexible catheter inserted into the patient's anatomical passage to deliver a medical device to the distal tip of the flexible catheter.

[0075] As mentioned above, Figure 3C Patient P, who underwent liver surgery, is shown. Figure 3C Arms 124A-124D are shown, wherein the instrument shafts 132A-132C of medical devices 130A-130C protrude into the abdomen A. Figure 3C and Figure 6A An imaging probe 136 is depicted in an imaging system 126 inserted into the abdomen A to provide visualization of organs and tissues within the abdomen A. In particular, the imaging probe 136 is aligned by an arm 124D to image a portion of the liver L.

[0076] Returning to method 500, in operation 504, imaging probe 136 can be activated by control system 112 to capture surface data points from the visualized portion of the target tissue. (As...) Figure 6A As shown, the imaging probe 136 may be a structured light probe that emits light in a manner that can be interpreted as determining depth information of the illuminated portion 600. In some embodiments, the illuminated portion 600 may be only a limited portion of the field of view visualized by the imaging probe 136 or another component of the imaging system 126. For example, the emitter of the imaging probe 136 may project an array of dots or a grid of dots 602. These dots can be read by the detector of the imaging probe 136 and interpreted as providing three-dimensional information describing the surface portion of the liver L included in the illuminated portion 600. For example, Figure 6B Multiple three-dimensional points 604 characterizing the surface or volume of the irradiated portion 600 are depicted. The control system 112 can process the three-dimensional information to generate multiple data points 604 to facilitate model registration to the liver L. In other embodiments, other systems such as LIDAR systems can be used to generate a set of data points describing the surface of the liver L.

[0077] In operation 506, the three-dimensional topology of the liver L can be received by the control system 112. This model can be a surface model, such as... Figure 7 The mesh surface model shown. Figure 7 A model 700 depicts the liver L of patient P. Although shown as a surface model in this example, it should be understood that the modal point set can be represented as any number of two-dimensional or three-dimensional anatomical topologies, including wireframe models, volumetric models, etc. As shown, the exemplary model 700 is a mesh model, but other embodiments of operation 508 may include receiving a volumetric model representing the liver L as a set of voxels. This voxel set can be processed to produce a surface or point set characterizing a surface or topology of anatomical structures, such as the internal channels of the liver or lung. In some embodiments, model 700 may be received as a three-dimensional model point set describing the surface of the liver L. In other embodiments, control system 112 receives a volumetric model or a surface model, or another model of the liver L, and processes the model to obtain a model point set. These points can be obtained by collecting and processing preoperative or intraoperative image data from imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescence microscopy, temperature recording, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, or nanotube X-ray imaging. For example, the control system 112 can receive model 700 and find intersections in surface mesh 702. The control system 112 can extract 3D points 704 from the intersections and / or from the center of each triangle or cell shape in mesh 702. These extracted model points can be generated as part of operation 506. Other 3D models can be used in other embodiments.

[0078] In operation 508, an initial seed transformation is determined based on information from the manipulator assembly 102 and program setup information associated with the specific procedure to be performed. Therefore, the initial seed transformation can act on extracted system information, including extracted kinetic information from the manipulator 252 or the other arm having a kinematic chain and / or based on setup information associated with the procedure to be performed. This setup information associated with the procedure includes, for example, the type of procedure to be performed, whether the procedure is performed on the left or right side of the patient, how the patient is oriented relative to the manipulator assembly 102 (e.g., the orientation of the patient's head and feet relative to the manipulator assembly 102), the point of entry (e.g., entry into the target anatomy via the stomach, back, mouth, or leg (e.g., via the femoral artery), the angle of entry, the orientation of the visual probe 278 relative to other medical devices, etc. Some system information can be determined from the instrument being used; however, other information (such as the patient-side point of entry) is provided to the control system 112 by the operator O during surgical setup. In order to register the captured surface point 604 or other modal points with model information (such as model point 704), an initial seed transformation is applied to one or two point sets to place the model in the same reference frame as liver L.

[0079] In order to register and use Figure 2B The surface points collected by the probe 278 of the manipulator 252 can be used to obtain kinematic information. For example, the encoder can indicate a specific orientation associated with the deflection servo actuator 254, the pitch servo actuator 256, and the I / O actuator 258. This information can be used to identify the visualized portion or irradiated portion 600 of the liver L being observed by the probe 278 (e.g., Figure 6A The angle (as shown). The orientation or roll of the distal end of probe 278 can be obtained from the encoder, thereby the orientation of probe 278 relative to the operating environment (including the table) can be used in point registration. For example, if a vision sensor located at the end of probe 278 is oriented at 35° relative to the horizon defined by the table, the initial seed transformation can make ( Figure 7 The point set 704 is more closely oriented to the orientation defined by the table or any other defined orientation in the surgical environment. Additionally, the stereoscopic properties of probe 278 can be used to detect the distance between the tip of probe 278 and the surface of liver L. The viewing angle and the distance between probe 278 and the irradiated portion 600 can be used to generate an initial seed transformation to begin registering the surface model 700 with the captured data points 604 of liver L. For example, a transformation matrix can be generated and applied to point 704 to register point 704 to a three-dimensional point 604 obtained by visualizing liver L (e.g., ...). Figure 6A and Figure 6B(As shown). The transformation matrix may include factors providing rotation of point 704 and factors providing transformations of these points 704 in three dimensions toward points on the surface corresponding to the irradiated portion 600 of the liver L. In some embodiments, the transformations provided by the transformation matrix may be based on user input.

[0080] Additionally, system 100 can be configured to guide an operator (such as operator O) via settings associated with a specific procedure (including procedure setup steps). The surgeon can interact with system 100 to select a specific procedure from multiple options. The selected specific procedure, the target anatomy, the anatomical proximity to the patient's body (e.g., left or right proximity to the anatomy, incision angle based on the target anatomy, etc.), and steps in the associated workflow can also be used to determine the initial seed transformation to be applied to the two point sets (e.g., points 1502 and 1516) to be registered. Operator O can interact with system 100 to select a specific procedure from multiple options. For example, operator O can select liver surgery from multiple medical procedure options.

[0081] The steps in the selected procedure and associated settings can also be used to determine the initial seed transformation to be applied to one of the two point sets (such as points 604 and 704) to be registered together. For example, if the procedural step in the workflow is an incision step to make an incision in the tissue, system 100 can instruct that subsequently collected 3D surface information (new point 704) is not used for registration because an incision can alter the surface of an organ or tissue in a way that does not correspond to preoperative image data, thus resulting in poor-quality registration or degradation of previously performed registration. Alternatively, control system 112 can require registration to be performed using different portions of the organ or tissue. This might require the moving arm 124D such that the probe tip 278 is oriented to collect surface data in an unaltered region of the organ or tissue. This could result in overriding a default setting that updates registration periodically or on an event-driven basis. This setting can instruct the organ or tissue to be visualized from a specific angle at a particular step in the workflow, and can specify approximate angle, orientation, and entry point. For example, the workflow might indicate that a port will be used near the patient's navel, or that one or more instruments will be inserted through a natural orifice of the body, allowing for an estimate of the angle of the instrument relative to the target organ or tissue. During this step, this angle can be used in an initial seed transformation, which can be applied to model point 704 to place model point 704 in the patient coordinate space defined by the surgical environment. Alternatively, the initial seed transformation can be applied to collected surface points 604 to place point 604 in model space. Regardless of which point the transformation is applied to, the initial transformation provides a first step in placing the two sets of points in a common space, allowing both practical and modeling information to be used jointly by the operator O.

[0082] In addition, the relative positions of multiple instruments can also be included in the setup information. Such information can be obtained from the current procedure step in the workflow, as well as immediately when selecting settings from the menu of potential procedure setup information.

[0083] In some embodiments, the force sensor can be used for physical contact with an organ or tissue. Kinematic information from the encoder of the manipulator 252 can be used in conjunction with information from the force sensor to indicate the orientation of a target surface to be registered with the model 700. The force sensor data can provide surface points, such as... Figure 6B Surface point 604. In some embodiments, a vision sensor can be used to determine whether contact has been made. For example, contact between the vision sensor and the target tissue is indicated when the vision sensor approaches the target tissue and the overall light signal obtained from the vision sensor decreases toward zero. Knowledge of kinematic information can be used to determine the orientation of the force sensor or vision sensor at the time of contact. In some embodiments, points can be collected based on the detection of contact. For example, the distal end 218 of the flexible catheter body 216 may include a contact sensor. When contact is made between the distal end 218 and the anatomical surface, the orientation of the distal end 218 can be extracted from a shape sensor or position sensor included on the catheter body 216. Contact with the anatomical surface can be repeated to obtain a set of points for registration.

[0084] In operation 510, a defined initial seed transformation can be applied. For example, the initial seed transformation can be used to place model 700 in... Figure 8A The directions shown. Figure 8A The relative orientation and orientation of model 700 and liver L were depicted after the application of the initial seed. It can be seen that even after the initial seed transformation, the orientation and orientation of the actual liver L and model 700 of liver L may differ so much that information from one source (i.e., actual observation versus preoperative or intraoperative observation obtained from model 700) cannot be reliably applied to other sources.

[0085] Iteration operation 512 can provide a usable registration between liver L and model 700. For example... Figure 5As shown, operation 512 includes a set of sub-operations that can be repeated to produce satisfactory registration. Operation 512 may include operation 514 matching collected data points with model points, operation 516 calculating motion based on the interval distance between matched points, and operation 518 applying a transformation to the captured data points. These operations should bring model 700 closer to the alignment of liver L. In operation 520, the convergence between model 700 and liver L can be evaluated based on two sets of three-dimensional points associated with model 700 and liver L. More detailed information about iterative operation 512 and related sub-operations is provided in application PCT / US16 / 46633, filed August 11, 2016, entitled “SYSTEMS AND METHODS OF REGISTRATION FOR IMAGE-GUIDED SURGERY,” the disclosure of which is incorporated herein by reference in its entirety.

[0086] Figure 8B An exemplary satisfactory convergence between the model 700, as shown in the display system (display system 110 of Figure 1), and the actual liver L of the patient P located within the surgical environment is depicted. The control system 112 can synchronize the display of the model and actual information (e.g., real-time video of the liver L). Additionally, as described above, with the patient surgical space registered to the model space, the current shape of any instruments and the position of their distal ends can be positioned and displayed simultaneously with the rendering of the model 700. Alternatively, the model 700 can be overlaid on the real-time video feed.

[0087] Operator O can rely on the information contained in model 700 to perform procedures on liver L. For example, model 700 may include instructions for a portion of liver L to be surgically removed. Model 700 may be included in a user interface displayed on display system 110 of Figure 1, along with a representation of liver L (such as a real-time video feed). The feed may be obtained via probe 278 or other systems. The user interface may provide operator O with instructions on the location and extent of liver tissue to be surgically removed or observed for biopsy.

[0088] Furthermore, in operation 522, the control system 112 can determine the current position of any or all medical devices 130A-130C. In operation 524, because the model 700 is registered to a common reference frame along with the surgical environment, one or more of the devices 130A-130C can be positioned relative to the model 700. For example, device 130A may include an ablation probe or surgical scissors for cutting tissue from the liver L. The extent of the tissue to be removed can be defined within the model 700, and the model 700, already positioned in the same space as the liver L, can be used by the control system 112 to limit the operation of the ablation probe or surgical scissors to a defined volume within the model 700.

[0089] As noted, when operator O causes a change in the surface used in registration (i.e., the operator makes a cut on the surface at the collection point to perform an initial seed transformation or registration), control system 112 may indicate that the registration is unreliable or that the registration should be considered unreliable. In some embodiments, control system 112 may determine that the current procedural step indicates to proceed. Control system 112 may request a new registration as a subsequent procedural step in a workflow associated with a particular operation. In some embodiments, when a later registration replaces an earlier registration or when control system 112 deems an earlier registration replaceable by a later registration, an alert may be provided to the clinician via a user interface to indicate that a registration change exists, a new registration is required, or a better registration is available. In some embodiments, control system 112 may require operator approval via a user interface before implementing a better registration. For example, when a better registration is identified, an alert may be presented to display system 110 and buttons or other user interface elements (through which the clinician can approve or disapprove the new registration). The new registration is then implemented or not implemented based on the clinician's decision.

[0090] As described above, the model and localized instruments can be displayed to the operator O to assist in performing medical procedures. Optionally, the assisted operator can provide operator input to control the operation or movement of the instruments or arms 124A-124D.

[0091] Although the systems and methods of this disclosure have been described for use in liver surgery, the principles of this disclosure can be applied to the execution of other procedures for registering modeled surfaces to actual surfaces. For example, a model of a kidney can be registered to a kidney, a model of a prostate can be registered to a prostate, and so on. Because the surfaces of these organs may be difficult to register to preoperative or intraoperative models, information about the orientation and pose of the imaging system 126, as well as information about the procedure settings of the operation, may be particularly valuable when performing the initial seed transformation.

[0092] exist Figure 7In some embodiments of method 700, the registration process 712 may include an additional process of changing the weights of the measurement points according to parameters or rules. This weighting can be changed by adding weights where they were not previously weighted or by changing the weights associated with a given measurement point. For example, in some embodiments, relatively high weights may be assigned to recently acquired points. As more time passes, relatively high weights can be provided by gradually decreasing the weights assigned to non-recently acquired points. Recently measured points may be more accurate due to patient movement, so the registration process 712 can be biased to reflect the latest information by giving more weight to recently measured points compared to non-recently measured points. The weights may be standardized or non-standardized weights.

[0093] In some embodiments, when a later registration replaces an earlier registration, or when the control system 112 determines that an earlier registration can be replaced by a later registration, an alert can be provided to the clinician via a user interface to indicate that a registration change has occurred or a better registration is available. In some embodiments, the control system 112 may require the clinician's approval via a user interface before implementing the better registration. For example, when a better registration is identified, an alert may be presented to the display system 110 and to buttons or other user interface elements (through which the clinician can approve or disapprove the new registration). The new registration is then implemented or not implemented based on the clinician's decision.

[0094] One or more elements (including methods 400 and 500) in embodiments of the invention can be implemented in software to execute on a processor of a computer system such as control system 112. When implemented in software, the elements of embodiments of the invention are essentially code segments that perform necessary tasks. Program segments or code segments can be stored in a non-transitory processor-readable storage medium or device, including any medium capable of storing information, including optical, semiconductor, and magnetic media. Examples of processor-readable storage devices include electronic circuits, semiconductor devices, semiconductor storage devices, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. Code segments can be downloaded via computer networks such as the Internet or intranets. As described herein, operations such as accessing, detecting, initiating, registering, displaying, receiving, generating, determining, moving data points, segmenting, and matching can be performed at least in part by control system 112 or by the processor of control system 112.

[0095] Note that the presented processes and displays may not inherently relate to any particular computer or other device. The desired architectures of various such systems will appear as elements in the claims. Furthermore, embodiments of the invention are not described with reference to any particular programming language. It should be understood that the teachings of the invention as described herein can be implemented using various programming languages.

[0096] While certain exemplary embodiments of the invention have been described and illustrated in the accompanying drawings, it should be understood that these embodiments are merely illustrative of the broad invention and not limiting, and that various other modifications will be apparent to those skilled in the art, and that the embodiments of the invention are not limited to the specific structures and arrangements shown and described.

[0097] Embodiments of the present invention facilitate the registration of a modal point set of a preoperatively acquired defined anatomical model to a modal point set acquired intraoperatively using a vision-based method. After generating an initial seed transformation based on system information including extracted dynamic and workflow information, the model point set can be registered. This information enables registration of tissue surfaces to facilitate image-guided and robotic surgery on such tissue surfaces.

Claims

1. A medical system comprising: Manipulator; An imaging probe is coupled to the manipulator so that the imaging probe can move together with the manipulator; and A control system, which communicates with the manipulator and the imaging probe, wherein the control system is configured to perform operations including the following steps: Access the model point set representing the patient anatomy of interest; Receive a first set of capture points acquired during visualization of a first portion of the patient anatomy of interest using the imaging probe; It is determined that the first set of captured points will not be used for registration; Receive a second set of capture points acquired during visualization of a second portion of the patient anatomy of interest using the imaging probe; and Generate a registration between the model point set and the second captured point set.

2. The medical system of claim 1, wherein determining that the first set of capture points will not be used for registration is based on procedural steps in the workflow.

3. The medical system of claim 2, wherein the procedural steps in the workflow include a variation of the first portion of the patient anatomy of interest.

4. The medical system of claim 3, wherein the second portion of the patient anatomy of interest is not altered by the procedural steps in the workflow.

5. The medical system of claim 2, wherein the procedural step in the workflow includes making an incision in the first portion of the patient anatomy of interest.

6. The medical system according to any one of claims 1-5, wherein the control system is further configured to perform operations including the following steps: Instructions to redirect the imaging probe from the visualization of the first portion of the patient anatomy of interest to the visualization of the second portion of the patient anatomy of interest.

7. The medical system according to any one of claims 1-5, wherein the control system is further configured to perform operations including the following steps: Extract system information, wherein the system information includes kinematic information from the manipulator; and Before generating the registration between the model point set and the second capture point set, an initial seed transformation is generated based on the system information.

8. The medical system of claim 7, wherein extracting system information includes obtaining information from a plurality of encoders arranged along the manipulator.

9. The medical system of claim 7, wherein extracting system information includes acquiring information from an optical fiber shape sensor extending along the imaging probe.

10. The medical system according to any one of claims 1-5, wherein the control system is further configured to perform operations including the following steps: Based on the determination that the first captured point set will not be used for registration, the default setting is overridden, thereby updating the registration between the model point set and the first captured point set.

11. A method executed by a control system, comprising: Access the model point set representing the patient anatomy of interest; Receive a first set of capture points obtained during visualization of a first portion of the patient anatomy of interest using an imaging probe coupled to the manipulator; It is determined that the first set of captured points will not be used for registration; Receive a second set of capture points acquired during visualization of a second portion of the patient anatomy of interest using the imaging probe; and Generate a registration between the model point set and the second captured point set.

12. The method of claim 11, wherein determining that the first set of capture points will not be used for registration is based on procedural steps in the workflow.

13. The method of claim 12, wherein the procedural steps in the workflow include a modification of the first portion of the patient anatomy of interest.

14. The method of claim 13, wherein the second portion of the patient anatomy of interest is not altered by the procedural steps in the workflow.

15. The method of claim 12, wherein the procedural step in the workflow includes making an incision in the first portion of the patient anatomy of interest.

16. The method according to any one of claims 11-15, further comprising: Instructions to redirect the imaging probe from the visualization of the first portion of the patient anatomy of interest to the visualization of the second portion of the patient anatomy of interest.

17. The method according to any one of claims 11-15, further comprising: Extract system information, including kinematic information from the manipulator; and Before generating the registration between the model point set and the second capture point set, an initial seed transformation is generated based on the system information.

18. The method of claim 17, wherein extracting system information includes obtaining information from a plurality of encoders arranged along the manipulator.

19. The method of claim 17, wherein extracting system information includes acquiring information from an optical fiber shape sensor extending along the imaging probe.

20. The method according to any one of claims 11-15, further comprising: Based on the determination that the first captured point set will not be used for registration, the default setting is overridden, thereby updating the registration between the model point set and the first captured point set.

Images