Systems and methods related to registration for image-guided surgery

Abstract

A system includes one or more processors configured to read instructions to cause the system to perform operations including accessing a set of model points of a model of a patient's anatomy. The model points are associated with a model space. The operations also include collecting a set of measurement points of the patient's anatomy, associated with a patient space; determining a set of matches between the set of model points and the set of measurement points; determining a first plurality of weights for the set of matches; and registering the set of model points to the set of measurement points based on the first plurality of weights to generate a first registration.

Description

[0001] Cross-reference of related applications

[0002] This application claims the benefit of U.S. Provisional Application 62 / 686,854, filed June 19, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to systems and methods for performing image-guided procedures, and more specifically to systems and methods for using registered real-time images and anatomical images from previous times during an image-guided procedure. Background Technology

[0004] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. These techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. An operator can insert minimally invasive medical instruments (including surgical instruments, diagnostic instruments, therapeutic instruments, or biopsy instruments) through these natural openings or incisions to reach the target tissue location. To aid in reaching the target tissue location, the position and movement of the medical instrument can be correlated with preoperative or intraoperative images of the patient's anatomy. By correlating the image-guided instrument with the images, the instrument can navigate through naturally or surgically created pathways in anatomical systems such as the lungs, colon, intestines, kidneys, heart, circulatory system, or the like. Typically, this correlation is determined based on a rigid match between the position and movement of the image-guided instrument and preoperative or intraoperative images of the patient's anatomy. However, this rigid match can affect the quality of the correlation and thus the quality of the image-guided procedure.

[0005] Therefore, it would be advantageous to provide improved registration for performing image bootstrapping procedures. Summary of the Invention

[0006] Embodiments of the invention are best summarized by the claims appended to the specification.

[0007] In one illustrative embodiment, a method is performed by a computational system. The method includes: accessing a set of model points of a model of a patient's anatomy, the model points being associated with a model space; and collecting a set of measurement points of the patient's anatomy, the measurement points being associated with a patient space. The method further includes determining a set of matches between the set of model points and the set of measurement points, determining a first plurality of weights for the set of matches, and registering the set of model points to the set of measurement points based on the first plurality of weights to generate a first registration.

[0008] In another illustrative embodiment, a method is performed by a computational system. The method includes: accessing a set of model points of a model of a patient's anatomical structure, the model points being associated with a model space; and collecting a set of measurement points of the patient's anatomical structure, the measurement points being associated with a patient space. The method further includes determining a first plurality of weights for the set of model points based on target anatomical locations, and registering the set of model points to the set of measurement points based on the first plurality of weights to generate a registration.

[0009] In yet another illustrative embodiment, a method is performed by a computational system. The method includes: accessing a set of model points of a model of a patient's anatomy, the model points being associated with a model space; and collecting a set of measurement points of the patient's anatomy, the measurement points being associated with a patient space. The method further includes determining a first plurality of weights for each set of measurement points; and registering the set of model points to the set of measurement points based on the first plurality of weights to generate a registration.

[0010] In yet another illustrative embodiment, a method is performed by a computational system. The method includes accessing a set of model points of a model of a patient's anatomical structure, the model points being associated with a model space. The method also includes collecting a set of measurement points of the patient's anatomical structure, the measurement points being associated with a patient space. The method further includes registering the set of model points with the set of measurement points to generate a first registration, dividing the anatomical structure into multiple anatomical regions; generating multiple region registrations for the multiple anatomical regions based on the first registration; and generating a second registration for converting the model space into a patient space using the multiple region registrations.

[0011] In yet another illustrative embodiment, a method is performed by a computational system. The method includes accessing a set of model points of a model of a patient's anatomy, associated with a model space. The method also includes collecting a set of measurement points of the patient's anatomy, associated with a patient space. The method further includes registering the set of model points with the set of measurement points to generate a first registration; providing a patient anatomical image from a distal location of a medical device; and determining a mismatch between the patient anatomical image and a first visual representation of the model from a first navigation path location, the first navigation path location being determined based on the distal location and the first registration. The method also includes providing a second visual representation of the model from a second navigation path location different from the first navigation path location; receiving a matching indication that the patient anatomical image matches the second visual representation of the model; and generating a second registration for converting the model space into a patient space based on the distal location and the second navigation path location.

[0012] It should be understood that both the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting its scope. In this regard, additional aspects, features, and advantages of the present disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0013] Figure 1 This is a simplified diagram of a remotely operated medical system according to some embodiments.

[0014] Figure 2A This is a simplified diagram of a medical device system according to some embodiments.

[0015] Figure 2B This is a simplified diagram of a medical device with extended medical tools according to some embodiments.

[0016] Figure 3A and Figure 3B This is a simplified side view of a medical device mounted on an insertion assembly, according to some embodiments, in patient coordinate space.

[0017] Figure 4A , Figure 4B , Figure 4C and Figure 4D Figure 2 illustrates the insertion into a human lung according to some embodiments. Figure 3A , Figure 3B The remote end of the medical device system.

[0018] Figure 5 This is a flowchart illustrating a method for image-guided surgical procedures or a portion thereof according to some embodiments.

[0019] Figure 6A , Figure 6B and Figure 6C The steps in the segmentation process according to some embodiments are shown, which generate a model of the patient's human lungs for registration.

[0020] Figure 7 This is a flowchart of a method for updating the registration of an anatomical model with a patient's anatomy, according to some embodiments.

[0021] Figure 8A and Figure 8B The distal end of a medical device system during insertion into a human lung, according to some embodiments, is shown.

[0022] Figure 9 This is a flowchart of a method for updating the registration of an anatomical model to a patient's anatomy, according to some embodiments.

[0023] Figure 10A model of a patient's human lung for registration is shown according to some embodiments.

[0024] Figure 11 This is a flowchart of a method for updating the registration of an anatomical model with a patient's anatomy, according to some embodiments.

[0025] Figure 12 and Figure 13 The illustration shows a phase of a re-registration technique according to some embodiments.

[0026] The embodiments of this disclosure and their advantages will be best understood by referring to the following detailed description. It should be understood that similar reference numerals are used to identify similar elements shown in one or more of the figures, wherein the illustrations are for illustrative purposes and not for limiting the embodiments of this disclosure. Detailed Implementation

[0027] In the following description, specific details are set forth, which are consistent with some embodiments of the present disclosure. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not limiting. Although not specifically described herein, those skilled in the art will recognize other elements within the scope and spirit of this disclosure. Furthermore, to avoid unnecessary repetition, one or more features shown and described in association with one embodiment may be incorporated into other embodiments, unless otherwise specifically described or if one or more features would render the embodiment inoperable.

[0028] In some instances, well-known methods, procedures, components, and circuits have not been described in detail to avoid unnecessarily obscuring various aspects of the embodiments.

[0029] This disclosure describes various instruments and parts thereof based on their states in three-dimensional space. As used herein, the term "orientation" refers to the position of an object or part thereof in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or part thereof (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the orientation of an object or part thereof in at least one translational degree of freedom and the orientation of the object or part thereof in at least one rotational degree of freedom (up to six degrees of freedom). As used herein, the term "shape" refers to a set of poses, orientations, or orientations measured along the object.

[0030] Figure 1This is a simplified diagram of a remotely operated medical system 100 according to some embodiments. In some embodiments, the remotely operated medical system 100 may be suitable for, for example, surgical, diagnostic, therapeutic, or biopsy procedures. Although some embodiments of such procedures are provided herein, any references to medical or surgical instruments and medical or surgical methods are non-limiting. The systems, instruments, and methods described herein can be used with animals, human cadavers, animal carcasses, parts of human or animal anatomy, for non-surgical diagnostics, and for industrial systems and general-purpose robots or remote operating systems.

[0031] like Figure 1 As shown, the medical system 100 typically includes a manipulator assembly 102 for operating the medical device 104 while performing various procedures on the patient P. The manipulator assembly 102 may be a remotely operated, non-remotely operated, or hybrid remotely and non-remotely operated assembly, having selectable degrees of freedom of motion that can be motorized and / or remotely operated, as well as selectable degrees of freedom of motion that can be non-motorized and / or non-remotely operated. The manipulator assembly 102 is mounted to or near the operating table T. A master control assembly 106 allows the operator (e.g., as...) Figure 1 The surgeon, clinician, or physician shown observes the intervention site and controls the manipulator assembly 102.

[0032] The master control unit 106 may be located at the operator console, typically in the same room as the operating table T, such as to the side of the operating table where the patient P is located. However, it should be understood that the operator O may be located in a different room or in a completely different building from the patient P. The master control unit 106 typically includes one or more control devices for controlling the manipulator assembly 102. The control devices may include any number of various input devices, such as joysticks, trackballs, data gloves, trigger guns, manual controllers, voice recognition devices, body motion or presence sensors, and / or the like. To provide the operator O with a strong sense of direct control of the instrument 104, the control devices may be provided with the same degrees of freedom as the associated medical instrument 104. In this way, the control devices provide the operator O with a remote presentation or perception of the control devices integrated with the medical instrument 104.

[0033] In some embodiments, the control device may have more or fewer degrees of freedom than the associated medical device 104, and still provide remote presentation to operator O. In some embodiments, the control device may optionally be a manual input device that moves in six degrees of freedom, and may also include an actuable handle for actuating the device (e.g., for closing a gripping clamp, applying a potential to an electrode, delivering drug treatment, and / or the like).

[0034] Manipulator assembly 102 supports medical device 104 and may include one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in place, commonly referred to as mounting / setting structures), and / or one or more servo-controlled links (e.g., another link that can be controlled in response to commands from a control system) and the kinematic structure of the manipulator. Manipulator assembly 102 may optionally include multiple actuators or motors that drive inputs on medical device 104 in response to commands from a control system (e.g., control system 112). Actuators may optionally include drive systems that, when coupled to medical device 104, can advance medical device 104 into anatomical openings created naturally or surgically. Other drive systems may move the distal end of medical device 104 with multiple degrees of freedom, which may include three linear degrees of freedom (e.g., linear motion along the X, Y, Z Cartesian axes) and three rotational degrees of freedom (e.g., rotation about the X, Y, Z Cartesian axes). Additionally, the actuator can be used to actuate the articulated end effector of medical device 104 for gripping tissue in a clamp of a biopsy device and / or the like. Actuator orientation sensors (such as resolvers, encoders, potentiometers, and other mechanisms) can provide sensor data to medical system 100 describing the rotation and orientation of the motor shaft. This orientation sensor data can be used to determine the motion of the object manipulated by the actuator.

[0035] The remotely operated medical system 100 may include a sensor system 108 having one or more subsystems for receiving information about the instrument of the manipulator assembly 102. Such subsystems may include an orientation / position sensor system (e.g., an electromagnetic (EM) sensor system); a shape sensor system for determining orientation, orientation, velocity, rate, posture, and / or shape along a distal end and / or one or more segments of a flexible body that may constitute the medical device 104; and / or a visualization system for capturing images from the distal end of the medical device 104.

[0036] The remote-operated medical system 100 also includes a display system 110 for displaying images or representations of the surgical site and medical device 104 generated by a subsystem of the sensor system 108. The display system 110 and the main control component 106 can be oriented such that the operator O can control the medical device 104 and the main control component 106 through remotely presented perception.

[0037] In some embodiments, medical device 104 may have a visualization system (discussed in more detail below) that may include an endoscope assembly that records concurrent or real-time images of the surgical site and provides the images to an operator or operator O via one or more displays of medical system 100 (such as one or more displays of display system 110). The concurrent images may be, for example, two-dimensional or three-dimensional images captured by an endoscope positioned within the surgical site. In some embodiments, the visualization system includes an endoscope component that may be integrally or removably coupled to medical device 104. However, in some embodiments, a separate endoscope attached to a separate manipulator assembly may be used with medical device 104 to image the surgical site. The visualization system may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which may include the processor of control system 112.

[0038] Display system 110 can also display images of surgical sites and medical instruments captured by a visualization system. In some examples, remote-operated medical system 100 can configure controls for medical instrument 104 and master control component 106 such that the relative orientation of the medical instrument is analogous to the relative orientation of the operator O's eyes and hands. In this way, operator O can manipulate medical instrument 104 and hand controls as if observing in a substantially real workspace. "Substantially real" means that the image presentation is a realistic perspective image, simulating the viewpoint of a physician physically manipulating medical instrument 104.

[0039] In some examples, the display system 110 may use image data from imaging technologies such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescein scanning, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar techniques to present images of the surgical site recorded preoperatively or intraoperatively. The preoperative or intraoperative image data may be presented as two-dimensional, three-dimensional, or four-dimensional images (including, for example, time-based or rate-based information) and / or as images derived from models created from the preoperative or intraoperative image dataset.

[0040] In some embodiments, often for the purpose of image-guided surgical procedures, the display system 110 may display a virtual navigation image in which the actual position of the medical device 104 is registered with (i.e., dynamically referenced) preoperative or concurrent images / models. This can be done to present a virtual image of the internal surgical site to the operator O from the viewpoint of the medical device 104. In some examples, this viewpoint may be from the tip of the medical device 104. Images and / or other graphic or alphanumeric indicators of the tip of the medical device 104 may be overlaid on the virtual image to assist the operator O in controlling the medical device 104. In some examples, the medical device 104 may be invisible in the virtual image.

[0041] In some embodiments, display system 110 may display a virtual navigation image in which the actual position of medical device 104 is registered with preoperative or concurrent images to present a virtual image of medical device 104 within the surgical site to operator O from an external viewpoint. An image of a portion of medical device 104 or other graphic or alphanumeric indicators may be overlaid on the virtual image to assist operator O in controlling medical device 104. As described herein, a visual representation of data points may be rendered onto display system 110. For example, measurement data points, movement data points, registration data points, and other data points described herein may be displayed on display system 110 in a visual representation manner. Data points may be visually represented in the user interface by multiple points or spots on display system 110, or they may be visually represented as a rendered model, such as a grid or line model created based on a set of data points. In some examples, data points may be color-coded according to the data they represent. In some embodiments, the visual representation may be refreshed in display system 110 after each processing operation to change the data points has been implemented.

[0042] The remote-operated medical system 100 may also include a control system 112. The control system 112 includes at least one memory and at least one computer processor (not shown) for implementing control between the medical device 104, the main control component 106, the sensor system 108, and the display system 110. The system 112 also includes programmed instructions (e.g., a non-transitory machine-readable medium storing the instructions) to implement some or all of the methods described according to the aspects disclosed herein, including instructions for providing information to the display system 110. Although the control system 112... Figure 1The system, shown as a single block in a simplified schematic, may include two or more data processing circuits, wherein a portion of the processing may be executed on or near the manipulator assembly 102, another portion of the processing may be executed at the master control assembly 106, and / or so on. The processor of the control system 112 may execute instructions corresponding to the processes disclosed herein and described in more detail below. A wide variety of centralized or distributed data processing architectures may be employed. Similarly, the programmed instructions may be implemented as multiple individual programs or subroutines, or they may be integrated into multiple other aspects of the remote operating system described herein. In one embodiment, the control system 112 supports wireless communication protocols such as Bluetooth, IrDA (Infrared Data Communication), HomeRF (Home Radio Frequency), IEEE 802.11, DECT (Digital Enhanced Wireless Communication), and wireless telemetry.

[0043] In some embodiments, the control system 112 may receive force and / or torque feedback from the medical device 104. In response to this feedback, the control system 112 may transmit a signal to the master control component 106. In some examples, the control system 112 may transmit a signal instructing one or more actuators of the manipulator component 102 to move the medical device 104. The medical device 104 may extend through an opening within the patient P to an internal surgical site within the patient P. Any suitable conventional and / or specialized actuators may be used. In some examples, one or more actuators may be separate from or integrated with the manipulator component 102. In some embodiments, one or more actuators and the manipulator component 102 are provided as part of a remotely operated surgical trolley positioned adjacent to the patient P and the operating table T.

[0044] The control system 112 may optionally also include a virtual visualization system to provide navigational assistance to the operator O when controlling the medical device 104 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system may be based on a reference to an acquired preoperative or intraoperative dataset of anatomical access routes. The virtual visualization system processes images of the surgical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescein examination, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or the like. Software, which may be used in conjunction with manual input, is used to convert the recorded images into segmented two-dimensional or three-dimensional composite representations of parts or entire anatomical organs or regions. The image dataset is associated with the composite representation. The composite representation and the image dataset describe the various locations and shapes of the access routes and their connectivity. During the clinical procedure, images used to generate the composite representation may be recorded preoperatively or intraoperatively. In some embodiments, the virtual visualization system may use standard representations (i.e., not patient-specific) or a mixture of standard representations and patient-specific data. Composite representations and any virtual images generated from them can represent the static posture of deformable anatomical regions during one or more phases of motion (e.g., during the inspiratory / expiratory cycle of the lungs).

[0045] During the virtual navigation procedure, sensor system 108 can be used to calculate the approximate position of medical device 104 relative to the anatomy of patient P. This position can be used to generate both a macroscopic horizontal (external) tracking image of the anatomy of patient P and a virtual internal image of the anatomy of patient P. The system can implement one or more electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors to register and display the medical device together with preoperatively recorded surgical images, such as those from a virtual visualization system. For example, such a system is disclosed by reference to PCT Publication WO 2016 / 191298 (published on December 1, 2016) ("Systems and Methods of Registration for Image Guided Surgery"), the entire contents of which are incorporated herein by reference. The remotely operated medical system 100 may also include optional operating and support systems (not shown), such as lighting systems, steering control systems, flushing systems, and / or suction systems. In some embodiments, the remotely operated medical system 100 may include more than one manipulator component and / or more than one master control component. The exact number of remote-controlled manipulator components will depend on factors such as the surgical procedure and space constraints within the operating room. The master control components 106 can be arranged side-by-side, or they can be positioned separately. Multiple master control components allow more than one operator to control one or more remote-controlled manipulator components in various combinations.

[0046] Figure 2A This is a simplified diagram of a medical device system 200 according to some embodiments. In some embodiments, the medical device system 200 may be used as a medical device 104 in an image-guided medical procedure performed using a remotely operated medical system 100. In some examples, the medical device system 200 may be used in non-remotely operated exploratory procedures or in procedures involving conventionally manually operated medical devices, such as endoscopy. Optionally, the medical device system 200 may be used to collect (i.e., measure) a set of data points corresponding to a location within an anatomical passage of a patient, such as patient P.

[0047] The medical device system 200 includes an elongated device 202, such as a flexible catheter, coupled to a drive unit 204. The elongated device 202 includes a flexible body 216 having a proximal end 217 and a distal or tip portion 218. In some embodiments, the flexible body 216 has an outer diameter of approximately 3 mm. Other flexible bodies may have larger or smaller outer diameters.

[0048] The medical device system 200 also includes a tracking system 230 for determining, using one or more sensors and / or imaging devices, the orientation, velocity, rate, posture, and / or shape along the distal end 218 and / or one or more segments 224 of the flexible body 216, as described in further detail below. The entire length of the flexible body 216 between the distal end 218 and the proximal end 217 can be effectively divided into segments 224. The tracking system 230 may optionally be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which may include… Figure 1 The processor of the control system 112.

[0049] The tracking system 230 may optionally track one or more of the distal end 218 and / or segments 224 using a shape sensor 222. The shape sensor 222 may optionally include an optical fiber aligned with the flexible body 216 (e.g., provided within an internal channel (not shown) or mounted externally). In one embodiment, the optical fiber has a diameter of approximately 200 μm. In other embodiments, the size may be larger or smaller. The optical fiber of the shape sensor 222 forms an optical fiber bending sensor for determining the shape of the flexible body 216. In an alternative, an optical fiber including a fiber Bragg grating (FBG) is used to provide strain measurements in one or more dimensions within the structure. Various systems and methods for monitoring the shape and relative position of optical fibers in three dimensions are described in the following patent applications: U.S. Patent Application No. 11 / 180,389 (filed July 13, 2005) (publishing "Fiber optic position and shape sensing device and method relating thereto"); U.S. Patent Application No. 12 / 047,056 (filed July 16, 2004) (publishing "Fiber-optic shape and relative position sensing"); and U.S. Patent No. 6,389,187 (filed June 17, 1998) (publishing "Optical Fibre Bend Sensor"), the entire contents of which are incorporated herein by reference. In some embodiments, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In some embodiments, other techniques may be used to determine the shape of the elongated device. For example, the history of the distal pose of the flexible body 216 may be used to reconstruct the shape of the flexible body 216 over time intervals. In some embodiments, the tracking system 230 may optionally and / or additionally use an orientation sensor system 220 to track the remote end 218. The orientation sensor system 220 may be a component of an EM sensor system, wherein the orientation sensor system 220 includes one or more conductive coils that can withstand an externally generated electromagnetic field. Each coil of the EM sensor system then generates an induced electrical signal whose characteristics depend on the orientation and orientation of the coil relative to the externally generated electromagnetic field. In some embodiments, the orientation sensor system 220 may be configured and positioned to measure six degrees of freedom or five degrees of freedom, wherein the six degrees of freedom are, for example, three position coordinates X, Y, Z and three orientation angles indicating the pitch, yaw, and roll of a base point, and the five degrees of freedom are, for example, three position coordinates X, Y, Z and two orientation angles indicating the pitch and yaw of a base point.A further description of the orientation sensor system is provided in U.S. Patent No. 6,380,732 (filed August 11, 1999) (published "Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked"), the entire contents of which are incorporated herein by reference.

[0050] In some embodiments, the tracking system 230 may alternatively and / or additionally rely on historical posture, orientation, or orientation data stored for known points of the instrument system along cycles of alternating movements (such as breathing). This stored data can be used to develop shape information about the flexible body 216. In some examples, a series of orientation sensors (not shown) (such as electromagnetic (EM) sensors similar to those in orientation sensor 220) can be positioned along the flexible body 216 and then used for shape sensing. In some examples, the history of data acquired during the procedure from one or more of these sensors can be used to represent the shape of the elongated device 202, particularly if the anatomical passage is typically static.

[0051] The flexible body 216 includes a channel 221, the size and shape of which are designed to receive a medical device 226. Figure 2BThis is a simplified diagram of a flexible body 216 having an extended medical device 226 according to some embodiments. In some embodiments, the medical device 226 can be used for procedures such as surgery, biopsy, ablation, illumination, irrigation, or aspiration. The medical device 226 can be deployed through channels 221 of the flexible body 216 and can be used at a target location within the anatomy. The medical device 226 may include, for example, an image capture probe, a biopsy instrument, laser ablation fibers and / or other surgical tools, diagnostic tools, or therapeutic tools. Medical tools may include end effectors having a single working member, such as scalpels, blunt blades, optical fibers, electrodes, and / or the like. Other end effectors may include, for example, forceps, grippers, scissors, clamps, and / or the like. Other end effectors may also include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, and / or the like. In various embodiments, the medical device 226 is a biopsy instrument that can be used to remove sample tissue or cell samples from a target anatomical location. Medical device 226 can be used in conjunction with an image capture probe also within flexible body 216. In various embodiments, medical device 226 may be an image capture probe comprising a distal portion of the flexible body at or near the distal end 218 having a stereo or single-view camera 216 for capturing images (including video images) processed by visualization system 231 for display and / or provided to tracking system 230 to support tracking of one or more segments of distal end 218 and / or segment 224. The image capture probe may include a cable coupled to the camera for transmitting the captured image data. In some examples, the image capture device may be a bundle of optical fibers, such as a fiber optic endoscope, coupled to visualization system 231. The image capture device may be monospectral or multispectral, for example, capturing image data in one or more of the visible spectrum, infrared spectrum, and / or ultraviolet spectrum. Alternatively, medical device 226 itself may be an image capture probe. Medical device 226 may be advanced from an opening in channel 221 to perform a procedure and then retracted into the channel upon completion of the procedure. The medical device 226 can be removed from the proximal end 217 of the flexible body 216 or from another optional device port (not shown) along the flexible body 216.

[0052] Medical device 226 may additionally accommodate a cable, linkage, or other actuation control (not shown) extending between its proximal and distal ends to controllably bend the distal end of medical device 226. Steering devices are described in detail in U.S. Patent No. 7,316,681 (filed October 4, 2005) (published "Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity") and U.S. Patent Application No. 12 / 286,644 (filed September 30, 2008) (published "Passive Preload and Capstan Drive for Surgical Instruments"), the entire contents of which are incorporated herein by reference.

[0053] The flexible body 216 may also accommodate cables, linkages, or other steering controls (not shown) extending between the drive unit 204 and the distal end 218 to controllably bend the distal end 218 (e.g., shown by dashed line 219 through the distal end 218). In some examples, at least four cables are used to provide independent "up-down" steering to control the pitch of the distal end 218 and "left-right" steering to control the yaw of the distal end 281. Steering elongated devices are described in detail in U.S. Patent Application No. 13 / 274,208 (filed October 14, 2011) (published "Catheter with Removable Vision Probe"), the entire contents of which are incorporated herein by reference. In embodiments in which the medical device system 200 is actuated by a remotely operated component, the drive unit 204 may include a drive input that is removably coupled to and receives power from a drive element (such as an actuator) of the remotely operated component. In some embodiments, the medical device system 200 may include a gripping feature, a manual actuator, or other components for manually controlling the movement of the medical device system 200. The elongated device 202 may be steerable, or alternatively, the system may be non-steerable and lack an integrated mechanism for operator control of bending of the distal end 218. In some examples, one or more cavities are defined in the walls of the flexible body 216, through which the medical device can be deployed and used at a target surgical site.

[0054] In some embodiments, the medical device system 200 may include flexible bronchial instruments, such as bronchoscopes or bronchial tubes, for examining, diagnosing, biopsiing, or treating the lungs. The medical device system 200 is also suitable for navigating and treating other tissues via naturally or surgically created access pathways in any of a variety of anatomical systems, including the colon, intestine, kidneys and renal calyces, brain, heart, circulatory system including the vascular system, and / or the like.

[0055] Information from tracking system 230 can be sent to navigation system 232, where it is combined with information from visualization system 231 and / or preoperatively acquired models to provide real-time location information to the physician or other operator. In some examples, the real-time location information can be displayed... Figure 1 The display system 110 is used to control the medical device system 200. In some examples, Figure 1 The control system 116 can utilize orientation information as feedback for positioning the medical device system 200. Various systems for registering and displaying surgical instruments and surgical images using fiber optic sensors are provided in the following patent applications: U.S. Patent Application No. 13 / 107,562, filed May 13, 2011, entitled "Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery"; and PCT Publication WO 2016 / 1033596 (filed May 20, 2016) (entitled "Systems and Methods of Registration for Image Guided Surgery"), the entire contents of which are incorporated herein by reference.

[0056] In some examples, the medical device system 200 can Figure 1 The medical system can be remotely operated within 100 units. In some embodiments, Figure 1 The manipulator component 102 can be replaced by direct operator control. In some examples, direct operator control may include various handles and operator interfaces for handheld operation of the instrument.

[0057] Figure 3A and Figure 3B This is a simplified side view of a medical device mounted on an insertion assembly, according to some embodiments, in patient coordinate space. Figure 3A and Figure 3B As shown, the surgical environment 300 includes a location in Figure 1Patient P is positioned on a table T. Patient P can be stationary within the surgical environment in the sense of being sedated, restrained, and / or otherwise restricting overall patient movement. Unless the patient is instructed to hold their breath to temporarily cease respiratory movement, circulatory anatomical movements (including respiratory and cardiac movements) of patient P can continue. Therefore, in some embodiments, data can be collected at specific stages of respiration and labeled and identified using that stage. In some embodiments, the stage at which data was collected can be inferred from physiological information collected from patient P. Within the surgical environment 300, a point collection device 304 is coupled to an instrument holder 306. In some embodiments, the point collection device 304 may use an EM sensor, shape sensor, and / or other sensor form. The instrument holder 306 is mounted to an insertion stage 308 fixed within the surgical environment 300. Alternatively, the insertion stage 308 may be movable but has a known position within the surgical environment 300 (e.g., via a tracking sensor or other tracking device). The instrument holder 306 may be a component of a manipulator assembly (e.g., manipulator assembly 102) coupled to the point-collecting instrument 304 to control insertion motion (i.e., motion along the A-axis) and optionally control the motion of the distal end 318 of the elongated device 310 in multiple directions including yaw, pitch, and roll. The instrument holder 306 or insertion stage 308 may include actuators, such as servo motors (not shown), that control the motion of the instrument holder 306 along the insertion stage 308.

[0058] An elongated device 310 is coupled to an instrument body 312. The instrument body 312 is coupled and fixed relative to an instrument support 306. In some embodiments, a fiber optic shape sensor 314 is fixed at a proximal point 316 on the instrument body 312. In some embodiments, the proximal point 316 of the fiber optic shape sensor 314 may move with the instrument body 312, but the location of the proximal point 316 may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 314 measures the shape from the proximal point 316 to another point (such as the distal end 318 of the elongated device 310). The point collection device 304 may be substantially similar to the medical device system 200.

[0059] The orientation measuring device 320 provides information about the orientation of the instrument body 312 as it moves along the insertion axis 308 on the insertion stage 308. The orientation measuring device 320 may include a resolver, encoder, potentiometer, and / or other sensors that determine the rotation and / or orientation of actuators that control the movement of the instrument carriage 306 and thus the movement of the instrument body 312. In some embodiments, the insertion stage 308 is linear. In some embodiments, the insertion stage 308 may be curved or have a combination of curved and linear segments.

[0060] Figure 3AThe instrument body 312 and instrument holder 306 are shown in the retracted position along the insertion stage 308. In this retracted position, the proximal point 316 is at orientation L0 on axis A. In this orientation along the insertion stage 308, the A component of the position of the proximal point 316 can be set to zero and / or another reference value to provide a basic reference for describing the orientation of the instrument holder 306, and thus the orientation of the proximal point 316 on the insertion stage 308. With the retracted orientation of the instrument body 312 and instrument holder 306, the distal end 318 of the elongated device 310 can be precisely positioned within the inlet orifice of the patient P. Also in this orientation, the orientation measuring device 320 can be set to zero and / or another reference value (e.g., I = 0). Figure 3B In this configuration, the instrument body 312 and instrument holder 306 are advanced along a linear track of the insertion stage 308, and the distal end 318 of the elongated device 310 has been advanced into the patient P. At this advanced position, the proximal point 316 is at orientation L1 on axis A. In some examples, encoders and / or other positional data from one or more actuators controlling the movement of the instrument holder 306 along the insertion stage 308 and / or one or more orientation sensors associated with the instrument holder 306 and / or the insertion stage 308 are used to determine the orientation L1 of the proximal point 316 relative to orientation L0. In some examples, orientation L1 may also serve as an indicator of the distance or depth of insertion of the distal end 318 of the elongated device 310 into the channel of the patient P's anatomy.

[0061] Figure 4A , Figure 4B , Figure 4C and Figure 4D It shows Figure 3A and Figure 3B The slender device 310 through Figure 1 and Figure 3A and Figure 3B The patient P's lung 400 is advanced through anatomical channels 402. These channels 402 include the trachea and bronchi. As the elongated device 310 is advanced along the insertion stage 308 with the carriage 306, the operator O can steer the distal end 318 of the elongated device 310 to navigate through the anatomical channels 402. While navigating through the anatomical channels 402, the elongated device 310 adopts a shape that can be "read" by shape sensors 314 extending within the elongated device 310.

[0062] refer to Figure 5 , Figure 6A , Figure 6B , Figure 6C , Figure 7 , Figure 8A , Figure 8B , Figure 9 , Figure 10 , Figure 11 and Figure 12Various embodiments of using image-guided surgical procedures with weighted and / or non-rigid registration are described. Figure 5 This is a flowchart illustrating a general method 500 for image-guided surgical procedures. Figure 6A , Figure 6B and Figure 6C The segmented process of generating a general method 500 for registering human lung models is shown. Figure 7 , Figure 8A and Figure 8B A method for performing weighted registration based on the real-time position of the distal end of an elongated device during insertion into a patient's anatomy is shown. Figure 9 and Figure 10 A method for performing non-rigid registration is shown, which takes into account the deformation, deflection, and orientation of different anatomical regions of the anatomy. Figure 11 , Figure 12 and Figure 13 A method for performing registration by matching patient anatomical images with visual representations of anatomical models is shown.

[0063] Figure 5 This is a flowchart illustrating a general method 500 for use in image-guided surgical procedures. Method 500 in Figure 5 The methods are shown as a set of operations or processes 502 to 512. Not all of the shown processes 502 to 512 can be performed in all embodiments of method 500. Additionally, in Figure 5 One or more processes not explicitly shown may be included before, after, between, or as part of processes 502 to 512. In some embodiments, one or more of the processes may be implemented at least in part in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors (e.g., the processor of control system 112), may cause one or more processors to execute one or more of the processes.

[0064] At procedure 502, preoperative or intraoperative image data is acquired from imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescein scanning, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, or nanotube X-ray imaging. The preoperative or intraoperative image data may correspond to two-dimensional, three-dimensional, or four-dimensional images (including, for example, time-based or rate-based information). For example, the image data may represent... Figures 4A to 4D 400 human lungs.

[0065] At process 504, a computer system, operating alone or in combination with manual input, is used to convert the recorded images into segmented two-dimensional or three-dimensional composite representations or models of parts or entire anatomical organs or regions. For example, Figure 6A It shows Figures 4A to 4D A segmented model 600 of the lung 400. Due to naturally occurring limitations or operator-set limitations, the segmented model 600 may not include all channels present within the human lung, but rather include certain channels 601. For example, relatively narrow and / or distal channels of the lung may not be fully included in the segmented model 600. The segmented model 600 may be a three-dimensional model, such as a mesh model or another suitable model, which includes walls defining the internal cavities or channels of the lung. Typically, the model provides mechanisms or means for distinguishing points within and outside the anatomical body region. Composite representation diagrams and image datasets describe the various locations and shapes of the channels and their connectivity, and may omit unwanted portions of the anatomy included in preoperative or intraoperative image data. In some embodiments, the model 600 may include particularly desired features, such as suspected tumors or other tissue portions of interest.

[0066] During the segmentation process, the image is divided into segments or elements (e.g., pixels or voxels) that share certain characteristics or calculated attributes (such as color, density, intensity, and texture). This segmentation process leads to two-dimensional or three-dimensional reconstruction, which forms a model of the target anatomy based on the acquired image, such as model 600. To represent this model, the segmentation process may depict multiple sets of voxels representing the target anatomy and then apply functions such as the traveling cube function to generate 3D surfaces surrounding the voxels. The model can be created by generating meshes, volumes, or voxel maps. This model can be displayed on display 110 to assist operator O in visualizing anatomy, such as the internal passages of a lung.

[0067] Alternatively, the model may include a centerline model comprising a set of interconnected line segments or points extending through the center of the modeling channel. Figure 6B An exemplary centerline model 602, derived from model 600 or directly from imaging data, is shown. The centerline segmented model 602 may include a set of three-dimensional straight lines or curves corresponding to the approximate center of the channel contained in the segmented model 602. The higher the resolution of the model, the more accurately the set of straight lines or curves will correspond to the center of the channel. Representing the lung with the centerline segmented model 602 can provide a smaller dataset that can be processed more efficiently by one or more processors or processing cores compared to the dataset of the segmented model 602 representing the channel walls of model 600. This can improve the operation of the control system 112.

[0068] like Figure 6B As shown, the centerline segmentation model 602 includes several branch points, in order to Figure 6B The visibility of the branching points is highlighted. Branching points A, B, C, D, and E are shown at each of several branching points. Branching point A may represent the point in the model where the trachea divides into the left main bronchus and the right main bronchus. The right main bronchus can be identified in the centerline segmentation model 602 as being located between branching points A and B. Similarly, secondary bronchi are identified by branching points B and C, and between branching points B and E. Another generation can be defined between branching points C and D. Each of these generations can be associated with a representation of the lumen diameter of the corresponding passage. In some embodiments, model 602 may include an average diameter value for each segmented generation. The average diameter value may be a patient-specific value or a more general value derived from multiple patients.

[0069] In the case where the model includes a centerline model with a set of interconnecting line segments, those line segments can be converted into a cloud or a set of points 604, called model points, which are... Figure 6C The dashed lines represent the model. By converting line segments into points, the desired number of model points corresponding to the interconnected line segments can be manually or automatically selected during the registration process to represent the centerline model 602 (and thus model 600). In the data, each point in this set of model points 604 may include coordinates such as a set of X... M Y M and Z M Coordinates, or other coordinates identifying the location of each point in the three-dimensional model space. In some embodiments, each point may include a generation identifier identifying which channel generation the point is associated with and / or a diameter or radius value associated with that portion of the centerline segmented model 602. In some embodiments, information describing the radius or diameter associated with a given point may be provided as part of a separate dataset.

[0070] The centerline segmentation model 602 is generated and stored in the data as... Figure 6C Following the set of points 604 shown, model points 604 can be retrieved from the data storage for use in image-guided surgical procedures. To use the centerline segmentation models 602 and 600 in an image-guided surgical procedure, model points 604 can be registered to associate the modeling channels in model 600 with the actual anatomy of the patient present in the surgical environment.

[0071] return Figure 5 At step 506, measurement points can be obtained from the patient's anatomy corresponding to the anatomical model, such as... Figures 3A to 3B and Figures 4A to 4DAs shown. The measurement point is associated with the patient space and may also be referred to as a patient space point. The measurement point can be generated by driving through the anatomy and / or touching landmarks in the anatomy and based on the tracking orientation of electromagnetic coils and / or sensor systems (e.g., sensor system 108).

[0072] At process 508, a point weighting scheme is determined for registering the anatomical model to the patient's anatomy. Weights can be assigned to measurement points, model points, and / or pairs of measurement and model points. In embodiments where weights are assigned to measurement points, the weights can be determined independently of the model. For example, weights can be based solely on reference... Figures 3A to 3B The description describes the insertion depth of an elongated device measured by an insertion or orientation sensor. In this example, if the elongated device is inserted a short distance, the weight of the measurement point can be low, while the weight of the measurement point increases as the insertion depth increases. In some embodiments, if the elongated device is inserted a relatively large distance, the weight of the measurement point can be low. In embodiments where weights are assigned to model points, the weights can be determined solely based on the model. For example, points not connected to other points can be considered noise and weighted with very low or zero values. In embodiments where weights are assigned to matches between measurement points and corresponding model points, the model points and corresponding measurement points are considered match points or matches and are assigned weights. In some embodiments, the point weighting scheme for matching is determined based on the proximity of the match to the target anatomical location. For example, the weight of the match is determined based on the distance between the match and the target anatomical location. In this example, a match associated with a model point closer to the target anatomical location can have a greater weight. In another example, the weight of the match is determined based on the distance between the associated model point of the match and a predetermined navigation path to the target anatomical location. In this example, a match associated with a model point closer to the predetermined navigation path to the target anatomical location can have a greater weight. In some embodiments, a sliding weight scale is used to determine the weights of the matches. In some embodiments, a weight with a zero value may be assigned to a match when the distance between the matched model point and the target anatomical location / a predetermined navigation path to the target anatomical location is greater than a predetermined target anatomical location distance threshold. In those examples, matches with a weight of zero may be discarded during subsequent registration procedures.

[0073] At process 510, before and / or during the progress of an image-guided surgical procedure on the patient, anatomical model data in model space is registered to the patient anatomy in patient space (and vice versa). In some embodiments, a point weighting scheme is used to apply weights to measurement points, model points, and / or the matching between measurement points and corresponding model points during registration. Typically, registration involves matching measurement points to model points of the model using rigid and / or non-rigid transformations. Within the scope of this disclosure, point set registration methods (e.g., Iterative Closest Point (ICP) techniques) can be used during the registration process. Such point set registration methods can generate transformations that align measurement points (also called measurement point sets) and model points (also called model point sets). In some embodiments, registration can also generate a deformed model associated with deformations of the patient anatomy (which are associated with measurement points and / or model points).

[0074] After process 510, in which the anatomical model is registered to the patient's anatomy so that a medical device positioned relative to the patient's anatomy can be represented relative to the anatomical model, the medical device can be advanced within the patient's anatomy. As the medical device moves to a new position, the registration can be updated at process 512. Registration updates can be performed continuously throughout the surgical procedure. In this way, changes caused by patient movement (both general movement and periodic physiological movement), patient respiration, movement of the medical device, and / or any other factors that may cause changes in the patient's anatomy can be compensated for.

[0075] Other registration methods used in image-guided surgery typically involve the use of techniques based on electromagnetic or impedance sensing. Metallic objects or certain electronic devices used in the surgical environment can cause interference, affecting the quality of the sensed data. Other registration methods can hinder clinical workflows. Some embodiments of the systems and methods described herein perform registration based on ICP or another point-set registration algorithm and the calibrated movement of a point-collecting instrument with a fiber optic shape sensor, thus eliminating or minimizing interference in the surgical environment. Other registration techniques can be used to register a set of measurement points to a preoperative model or a model obtained using another modality. In the embodiments described below, EM sensors on the patient and instrument, as well as optical tracking systems for the instrument, can be eliminated.

[0076] refer to Figure 7 , Figure 8A and Figure 8B The process used to update the registration (e.g., Figure 5 The process 512) may include method 700 to provide improved registration by using a weighting scheme based on the distal position of the elongated device and / or the target anatomical position.

[0077] refer to Figure 7 , Figure 7Method 700 is shown as a set of operations or procedures 702 to 712. Not all of the illustrated procedures 702 to 712 can be performed in all embodiments of method 700. Additionally, Figure 7 One or more processes not explicitly shown may be included before, after, between, or as part of processes 702 to 712. In some embodiments, one or more of the processes may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when run by one or more processors (e.g., the processor of control system 112), may cause one or more processors to execute one or more of the processes.

[0078] Method 700 begins at process 702, where the current registration of the anatomical model to the patient's anatomy is received. In this example, the current registration is performed before driving the elongated device toward the target anatomical location. Figure 5 The registration process generates a registration at point 510. In another example, referencing... Figure 8A The elongated device 310 is driven toward the target anatomical location 804. In this example, during process 702, the distal end 318 of the elongated device 310 is at the distal position 802. A point-weighted scheme can be used based on the distal position 802 and / or the target anatomical location 804. Figure 5 The current registration is generated in the update registration process 512.

[0079] At process 704, the position of the distal end 318 of the elongated device 310 is determined to have changed. (Reference) Figure 8B For example, at process 704, it is determined that the distal end 318 of the elongated device 310 is advanced from the distal position 802 to the distal position 806.

[0080] Despite Figure 8B In the example, the target anatomical location 804 remains the same, but in some embodiments, the target anatomical location may be shifted (e.g., based on operator input). In those embodiments, at process 706, it is determined that the target anatomical location has been moved to the updated target anatomical location.

[0081] At process 710, the point weighting scheme can be updated based on the changed distal location and / or the changed (current) target anatomical location. Specifically, the weights updated based on the changed distal location and / or the changed target anatomical location can be assigned to measurement points of the patient's anatomy (e.g., at...). Figure 5The process involves collecting 506 measurement points, and / or new measurement points collected as the elongated device is driven toward the target anatomical location. In some embodiments, the weight of a measurement point is determined based on its distance from the current distal location. In this example, a measurement point closer to the current distal location may have a greater weight. In some embodiments, a sliding weight scale is used to determine the weight of a measurement point based on its distance from the current distal location. In some embodiments, a weight with a zero value may be assigned to a measurement point when the distance from the current distal location is greater than a predetermined distal distance threshold. In those examples, measurement points with a weight of zero may be discarded during subsequent registration processes.

[0082] In some embodiments, matching weights may be alternatively or otherwise determined based on the distance between the associated model point and the target anatomical location, and / or the distance between the model point and the predetermined navigation path to the target anatomical location, for example, as referenced above. Figure 5 The process discussed in 508.

[0083] At step 712, the point-weighted scheme generated in step 710 is used to re-perform the registration of the anatomical model to the patient's anatomy. In this way, as the elongated device 310 is driven toward the target anatomical location, the registration can be continuously updated based on the current distal position and the target anatomical location.

[0084] refer to Figure 9 and Figure 10 In some embodiments, the process for updating registration (e.g., Figure 5 The process 512) may include method 900 to provide improved registration by taking into account deformation, deflection, and rotation of different regions of the anatomical structure. In various embodiments, the anatomical structure may be divided into multiple anatomical regions (e.g., based on the stiffness of the anatomical regions). In some examples, local registration may be performed for each of those anatomical regions to generate a corresponding regional registration. Those regional registrations can then be used to update the registration of the anatomical model to the measurement points. In some examples, the registration method may use deflection and rotation of different anatomical regions (e.g., utilizing global registration of the anatomical structure or local registration of the anatomical regions) and generate deflection and / or rotation parameter estimates for each anatomical region.

[0085] refer to Figure 9 , Figure 9 Method 900 is shown as a set of operations or procedures 902 to 908. Not all of the illustrated procedures 902 to 908 can be performed in all embodiments of method 900. Additionally, Figure 9One or more processes not explicitly shown may be included before, after, between, or as part of processes 902 to 908. In some embodiments, one or more of the processes may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when run by one or more processors (e.g., the processor of control system 112), may cause one or more processors to execute one or more of the processes.

[0086] Method 900 begins at process 902, where the current registration of the anatomical model to the patient's anatomy is received. In this example, the current registration is performed before driving the elongated device toward the target anatomical location. Figure 5 The registration process generates a registration at point 510. In another example, the current registration occurs during the drive of the elongated device toward the target anatomical location. Figure 5 The registration process generates registrations at 512 points.

[0087] In process 904, the anatomical structure is divided into multiple anatomical regions. In various embodiments, the lung can be divided into any suitable number of anatomical regions. (See reference...) Figure 10 The example shown is an anatomical model of a patient's human lung 600. Figure 10 In one example, it was determined that the left and right lungs of a human lung tend to deform at the main carina (e.g., near the branching point A where the trachea divides into the left and right main bronchi), while preserving the individual structures of the left and right lungs. Therefore, the anatomical model 600 is divided into anatomical regions 1002, 1004, and 1006 based on the main carina. Anatomical region 1002 includes the central region of the lung, anatomical region 1004 includes the right lung, and anatomical region 1006 includes the left lung. In another example, the patient's lungs can be divided into six anatomical regions, including the central region 1002, the upper lobe region of the right lung, the middle lobe region of the right lung, the lower lobe region of the right lung, the upper lobe region of the left lung, and the lower lobe region of the left lung.

[0088] At process 906, each anatomical region of the anatomical structure is registered with a corresponding model region of the anatomical model to generate a local registration. For example, measurement points (collected during process 506 and / or as the elongated device is driven toward the target anatomical location) can be divided into multiple sets of measurement points corresponding to the anatomical regions based on the current registration and the anatomical model. Figure 10 In the example, for each of anatomical regions 1002, 1004, and 1006, a subset of model points in the corresponding anatomical region is registered to a subset of measurement points in that anatomical region to generate a region registration. This registration method may include point set registration algorithms such as the Iterative Closest Point (ICP) technique, or another registration algorithm.

[0089] At process 908, those region registrations are used to update the registration from the anatomical model to the patient anatomy. In some embodiments, the updated registration includes three separate region registrations. In those embodiments, a point in patient space (e.g., the distal location of the elongated device) can be transformed to model space strictly based on the anatomical region of that point. For example, a region registration for anatomical region 1002 in patient space is used to transform a point in anatomical region 1002 to model space, and a region registration for anatomical region 1004 in patient space is used to transform a point in anatomical region 1004 to model space. In such embodiments, when the distal end of the elongated device is driven through the transition region between two adjacent anatomical regions (e.g., the transition region between adjacent anatomical regions 1002 and 1004, the transition region between adjacent anatomical regions 1002 and 1006), there may be jumps in one or more images displayed to the operator (e.g., using display system 110). Those images can be used to facilitate operator-guided navigation and / or surgery. In the example, the images include virtual navigation images, which include virtual images of the elongated device within the patient anatomy from an external viewpoint. In another example, the image includes an internal view of a portion of the anatomical model from the distal end of an elongated device registered to the anatomical model. Such jumps in the images could introduce interference and uncertainty into image-guided surgery. Therefore, in some embodiments, at process 908, the registration blends individual region registrations in the transition regions between adjacent anatomical regions, thus smoothing the transition between adjacent anatomical regions.

[0090] In some embodiments, at process 908, registration takes into account the deflection and / or rotation of each anatomical region. In the example, for each anatomical region, deflection and rotation parameters are estimated. Various optimization methods (e.g., random parameter variation and minimization or any other suitable minimization method) can be used during registration. In some examples, the optimization method may include a cost function to minimize point match residues and penalize excessive or unnatural large deflections and / or rotations. In some examples, the optimization method may use the number and quality of measurement points (e.g., total measurement points, a subset of measurement points for each anatomical region) to avoid overfitting the anatomical model. By considering the various rigidities, deflections, and rotations of individual anatomical regions of the anatomical structure, the registration of the anatomical model to the measurement points is improved.

[0091] refer to Figure 11 , Figure 12 and Figure 13 In some embodiments, the process for updating registration (e.g., Figure 5 The process 512) may include method 1100 to provide improved registration by matching an image of the patient's anatomy with a rendered internal view of the anatomical model.

[0092] refer to Figure 11 , Figure 11 Method 1100 is shown as a set of operations or procedures 1102 to 1114. Not all of the illustrated procedures 1102 to 1114 can be performed in all embodiments of method 1100. Additionally, Figure 11 One or more processes not explicitly shown may be included before, after, between, or as part of processes 1102 to 1114. In some embodiments, one or more of the processes may be implemented at least in part in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when run by one or more processors (e.g., the processor of control system 112), may cause one or more processors to execute one or more of the processes.

[0093] Method 1100 begins at process 1102, where the current registration of the anatomical model to the patient's anatomy is received. In this example, the current registration is performed before driving the elongated device toward the target anatomical location. Figure 5 The registration process generates a registration at point 510. In another example, the current registration occurs during the drive of the elongated device toward the target anatomical location. Figure 5 The registration process generates registrations at 512 points.

[0094] At process 1104, concurrent or real-time images of the patient's anatomy (e.g., captured by a visualization system) from a distal view of the elongated device, and a first visual representation of the internal view of the anatomical model from the distal view, are provided. Reference Figure 12 For example, the display system 110 displays concurrent or real-time images 1202 of the patient's anatomy from the distal end of the elongated device, and a first visual representation 1204 showing an internal view of the anatomical model from the perspective of the distal end of the elongated device.

[0095] like Figure 12 As shown in the example, the parallel or real-time image 1202 includes channels 1208-1, 1208-2, and 1208-3 of the patient's lungs. The first visual representation diagram 1204 illustrates the navigation path 1206 to the target anatomical location and a rendered model image of channels 1208-1, 1208-2, and 1208-3 based on the anatomical model. In some embodiments, the current distal position of the elongated device is registered to a first navigation path position 1210 based on (e.g., received at process 1102) the current registration. The first visual representation diagram 1204 is generated by generating an internal view of the anatomical model from the perspective of the first navigation path position 1210 along the navigation path 1206 toward the target anatomical location.

[0096] At process 1106, it is determined that the concurrent or real-time image 1202 and the first visual representation image 1204 do not match. Figure 12 In the example, channels 1208-1, 1208-2, and 1208-3 in the first visual representation of Figure 1204 are farther from the first navigation path location 1210 than channels 1208-1, 1208-2, and 1208-3 in the real-time image 1202 from the current distal location (corresponding to the current distal location registered in the anatomical model using the current registration). In some embodiments, this mismatch may be caused by lung deformation in the patient (e.g., by patient movement, including, for example, general movement and periodic physiological movement, movement of elongated devices, etc.).

[0097] In some embodiments, this mismatch between the concurrent or real-time image 1202 and the first visual representation image 1204 is automatically determined by the control system (e.g., by performing an image processing method to compare the concurrent or real-time image 1202 with the first visual representation image 1204). Alternatively, as Figure 12 As shown in the examples, in some embodiments, the mismatch is determined and provided by the operator. Figure 12 In the example, the operator (e.g., using selection 1212) determines that the concurrent or real-time image 1202 and the first visual representation image 1204 do not match, and (e.g., using button 1214) submits the mismatch determination to the control system.

[0098] At process 1108, a second visual representation of the internal view of the anatomical model from the perspective of the second navigation path location is provided. (Reference) Figure 13 For example, the display system 110 displays concurrent or real-time images 1202 of the patient's anatomy from the distal end of the elongated device, and a second visual representation 1302 of the internal view of the anatomical model from a second navigation path location 1304. In some embodiments, Figure 13 Concurrent or real-time images 1202 and Figure 12 The concurrent or real-time image 1202 is the same because the distal position of the elongated device remains the same during processes 1106 and 1108.

[0099] like Figure 13 As shown in the example, the second navigation path position 1304 is closer to the target anatomical location than the first navigation path position 1210. Thus, channels 1208-1, 1208-2, and 1208-3 in the second visual representation diagram 1302 are closer to the viewpoint than the first visual representation diagram 1204. In other examples, the second navigation path position 1304 can be farther from the target anatomical location, making channels 1208-1, 1208-2, and 1208-3 in the second visual representation diagram 1302 farther from the viewpoint.

[0100] In some embodiments, the position of the second navigation path is automatically determined by the control system. Alternatively, in some embodiments, the operator can use an input device to adjust the position of the second navigation path along the navigation path.

[0101] At process 1110, an instruction is received to match concurrent or real-time images with a second visual representation of the anatomical model. In some embodiments, this instruction is provided by the control system after comparing the concurrent or real-time images with the second visual representation of the anatomical model. Alternatively, as Figure 13 As shown, in some embodiments, this matching instruction is determined and provided by the operator. Figure 13 In one example, an operator (e.g., using selection 1306) determines that the concurrent or real-time image 1202 matches the second visual representation image 1302, and (e.g., using button 1308) submits a matching instruction to the control system.

[0102] At process 1112, the deformation of the anatomical structure is determined based on the current distal position, the current registration, and the second navigation path position. In the example, this deformation is determined by using a model of possible lung deformations (e.g., based on respiratory motion), where the current distal position and the second navigation path position have the closest fit when using this deformation.

[0103] At step 1114, the registration is then updated using the deformation determined at step 1112. In other words, the updated registration is improved by taking into account the determined deformation of the anatomical structure.

[0104] The systems and methods disclosed herein can be used for bronchial access routes connecting to the lungs. The systems and methods are also applicable to navigating and treating other tissues in any of a variety of anatomical systems, including the colon, intestine, kidneys, brain, heart, circulatory system, or the like, via natural or surgically created access routes. The systems and methods are also applicable to navigation around trackable surfaces of organs. The methods and embodiments of this disclosure are also applicable to non-surgical applications.

[0105] In some exemplary embodiments, a method performed by a computing system includes accessing a set of model points of a model of a patient's anatomy, the model points being associated with a model space; collecting a set of measurement points of the patient's anatomy, the measurement points being associated with a patient space; determining a first plurality of weights for the set of measurement points; and registering the set of model points to the set of measurement points based on the first plurality of weights to generate a registration. The set of measurement points is collected during the insertion of a medical device into the patient's anatomy. The first plurality of weights are determined based on the insertion depth of the medical device at the time the set of measurement points are collected.

[0106] In some exemplary embodiments, a method performed by a computing system includes accessing a set of model points of a model of a patient's anatomy, the model points being associated with a model space; collecting a set of measurement points of the patient's anatomy, the measurement points being associated with a patient space; registering the set of model points with the set of measurement points to generate a first registration; providing a patient anatomical image from a distal location of a medical device; determining a mismatch between the patient anatomical image and a first visual representation of the model from a first navigation path location, the first navigation path location being determined based on the distal location and the first registration; providing a second visual representation of the model from a second navigation path location different from the first navigation path location; receiving a matching indication that the patient anatomical image matches the second visual representation of the model; and generating a second registration for converting the model space to a patient space based on the distal location and the second navigation path location. In some embodiments, the matching indication is provided by an operator. In some embodiments, generating the second registration includes: determining a deformation of the anatomical structure based on the distal location and the second navigation path location; and updating the second registration using the deformation. In some embodiments, a non-transitory machine-readable medium includes a plurality of machine-readable instructions that, when executed by one or more processors, are adapted to cause one or more processors to perform one or more methods described herein.

[0107] In some exemplary embodiments, a method performed by a computing system includes accessing a set of model points of a model of a patient's anatomy, the model points being associated with a model space; collecting a set of measurement points of the patient's anatomy, the measurement points being associated with a patient space; determining a set of matches between the set of model points and the set of measurement points; determining a first plurality of weights for the set of matches; and registering the set of model points to the set of measurement points based on the first plurality of weights to generate a first registration. In some embodiments, the first plurality of weights for the set of matches are based on the proximity of each match to an anatomical target, wherein the anatomical target is associated with a model space. In some embodiments, the method further includes determining a second plurality of weights for the set of model points or the set of measurement points. In some embodiments, registering the set of model points to the set of model points is also based on the second plurality of weights. In some embodiments, the method further includes obtaining a first distal location of a medical device inserted into the anatomy. In some embodiments, determining the first plurality of weights includes: determining a distal distance between each measurement point and the first distal location; and assigning weights to the measurement points based on the distal distance. In some embodiments, assigning weights to measurement points based on distal distance includes: determining that the distal distance is greater than a predetermined distal distance threshold; and assigning a weight with a value of zero to the measurement point. In some embodiments, a first measurement point has a first distance from a first distal location, wherein a second measurement point has a second distance from the first distal location, the second distance being less than the first distance, and wherein a first weight assigned to the first measurement point is less than a second weight assigned to the second measurement point. In some embodiments, the method further includes detecting movement from the distal point to the second distal location; determining a second plurality of weights for the set of measurement points based on the second distal location; and registering the set of model points to the set of measurement points based on the second plurality of weights to generate a second registration. In some embodiments, the method further includes determining a target distance between the measurement point and a target anatomical location for each measurement point; and assigning weights to the measurement point based on at least one of the distal distance and the target distance.

[0108] In some exemplary embodiments, a method performed by a computing system includes accessing a set of model points of a model of a patient's anatomical structure, the model points being associated with a model space; collecting a set of measurement points of the patient's anatomical structure, the measurement points being associated with a patient space; determining a first plurality of weights for the set of model points based on a target anatomical location; and registering the set of model points to the set of measurement points based on the first plurality of weights to generate a registration. In some embodiments, determining the first plurality of weights includes: determining a target distance between the model point and the target anatomical location for each model point; and assigning weights to the measurement points based on the target distance. In some embodiments, determining the first plurality of weights includes: determining a navigation path distance between the model point and a predetermined navigation path to the target anatomical location for each model point; and assigning weights to the measurement points based at least on the target distance and the navigation path distance. In some embodiments, assigning weights to the model points includes: determining that the target distance is greater than a predetermined target distance threshold; and assigning weights with a value of zero to the model points. In some embodiments, a first model point has a first distance from the target anatomical location, wherein a second measurement point has a second distance from the target anatomical location, the second distance being less than the first distance, and wherein a first weight assigned to the first model point is less than a second weight assigned to the second measurement point.

[0109] One or more elements in embodiments of the present invention can be implemented in software to execute on a processor of a computer system, such as control system 112. When implemented in software, the elements of embodiments of the present invention are essentially code segments that execute desired instructions. The program or code segment can be stored in a processor-readable storage medium or device that can be downloaded via computer data signals embodied as a carrier wave over a transmission medium or communication link. A processor-readable storage device can include any medium capable of storing information, including optical, semiconductor, and magnetic media. Examples of processor-readable storage devices include electronic circuits; semiconductor devices, semiconductor memory devices, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. The code segment can be downloaded via a computer network such as the Internet, intranet, etc.

[0110] Note that the presented processes and displays may not inherently relate to any particular computer or other device. Various general-purpose systems may be used with the programs taught herein, or it may prove convenient to construct more specialized devices to perform the described operations. The desired structures of various such systems will appear as elements in the claims. Furthermore, embodiments of the invention are described without reference to any particular programming language. It should be understood that the teachings of the invention as described herein can be implemented using various programming languages.

[0111] Although certain exemplary embodiments of the invention have been described and illustrated in the accompanying drawings, it should be understood that such embodiments are merely illustrative of the invention in a broad sense and not as limiting of it, and that embodiments of the invention are not limited to the specific constructions and arrangements shown and described, as various other modifications can be made by those skilled in the art.

Claims

1. A system related to registration for image-guided surgery, comprising: a non-transitory memory; one or more processors coupled to the non-transitory memory and configured to read instructions to cause the system to perform operations comprising: accessing a set of model points of a model of an anatomical structure of a patient, the model points associated with a model space; collecting a set of measured points of the anatomical structure of the patient, the measured points associated with a patient space; determining a set of matches between the set of model points and the set of measured points; determining a first plurality of weights for the set of matches based on a distance between each match in the set of matches and a target anatomical location, wherein a second match in the set of matches is closer to the target anatomical location than a first match in the set of matches, and wherein a second weight determined for the second match is greater than a first weight determined for the first match; and registering the set of model points to the set of measured points based on the first plurality of weights to generate a first registration.

2. The system of claim 1, wherein the target anatomical location is associated with the model space.

3. The system of claim 2, further comprising determining a second plurality of weights for the set of model points or the set of measured points.

4. The system of claim 3, wherein registering the set of model points to the set of measured points is further based on the second plurality of weights.

5. The system of claim 1, further comprising obtaining a first distal location of a distal end of a medical instrument inserted into the anatomical structure.

6. The system of claim 5, wherein determining the first plurality of weights comprises: determining, for each measured point, a distal distance between the measured point and the first distal location; and assigning a weight to the measured point based on the distal distance.

7. The system of claim 6, wherein assigning the weight to the measured point based on the distal distance comprises: determining that the distal distance is greater than a predetermined distal distance threshold; and assigning the weight to the measured point with a value of zero.

8. The system of claim 6, wherein the operations further comprise: detecting movement of the distal end to a second distal location; determining a second plurality of weights for the set of measured points individually based on the second distal location; and registering the set of model points to the set of measured points based on the second plurality of weights to generate a second registration.

9. The system of claim 6, further comprising: determining, for each measured point, a target distance between the measured point and a target anatomical location; and assigning the weight to the measured point based on at least one of the distal distance and the target distance.

10. A system related to registration for image-guided surgery, comprising: a non-transitory memory; one or more processors coupled to the non-transitory memory and configured to read instructions to cause the system to perform operations comprising: accessing a set of model points of a model of an anatomical structure of a patient, the model points associated with a model space; ​ ​ ​ ​ collecting a set of measurement points of the anatomical structure of the patient, the measurement points being associated with a patient space; determining a first plurality of weights for the set of model points based on a distance between each model point of the set of model points and a target anatomical location, wherein a second model point of the set of model points is closer to the target anatomical location than a first model point of the set of model points, and wherein a second weight determined for the second model point is greater than a first weight determined for the first model point; and registering the set of model points to the set of measurement points based on the first plurality of weights to generate a registration.

11. The system of claim 10, wherein determining the first plurality of weights comprises: determining, for each model point, a navigation path distance between the model point and a predetermined navigation path to the target anatomical location; and determining, for each model point, the weight based on at least the distance between the model point and the target anatomical location and the navigation path distance.

12. The system of claim 10, wherein determining the weight for the model point comprises: determining that the distance between each model point and the target anatomical location is greater than a predetermined target distance threshold; and determining that the weight for the model point has a value of zero.

13. A non-transitory machine-readable medium comprising a plurality of machine- readable instructions that, when executed by one or more processors, are adapted to cause the one or more processors to perform a method comprising: accessing a set of model points of a model of an anatomical structure of a patient, the model points being associated with a model space; collecting a set of measurement points of the anatomical structure of the patient, the measurement points being associated with a patient space; registering the set of model points to the set of measurement points to generate a first registration; dividing the anatomical structure into a plurality of anatomical regions; generating, for each anatomical region of the plurality of anatomical regions, a respective local region registration based on the first registration; and updating the first registration using the respective local region registrations to generate a second registration for converting the model space to the patient space.

14. The non-transitory machine-readable medium of claim 13, wherein generating the respective local region registration for each anatomical region of the plurality of anatomical regions comprises, for each anatomical region: determining a subset of measurement points in the anatomical region based on the first registration; and registering the model points in the anatomical region to the corresponding subset of measurement points to generate a corresponding local region registration.

15. The non-transitory machine-readable medium of claim 13, wherein dividing the anatomical structure into the plurality of anatomical regions comprises: dividing the anatomical structure based on a structural rigidity of the plurality of anatomical regions, respectively.

16. The non-transitory machine-readable medium of claim 13, wherein the anatomical structure comprises lungs of the patient, and wherein the anatomical regions comprise a right lung region, and a central region comprising a trachea.

17. The non-transitory machine-readable medium of claim 13, wherein the anatomical structure comprises lungs of the patient, and wherein the anatomical regions comprise an upper lobe region of a right lung, a middle lobe region of the right lung, a lower lobe region of the right lung, an upper lobe region of a left lung, a lower lobe region of the left lung, and a central region comprising a trachea.

18. The non-transitory machine-readable medium of claim 13, wherein updating the first registration to generate the second registration comprises: blending the local region registrations for adjacent anatomical regions in a transition region of the adjacent anatomical regions.

19. The non-transitory machine-readable medium of claim 13, wherein the second registration comprises at least one of a deflection estimate and a rotation estimate for a first anatomical region.

Images