Systems and methods for navigation in image-guided medical procedures

Abstract

This paper provides a medical imaging system and method that eliminates the need for segmentation to provide navigation and procedural planning. One method includes receiving three-dimensional image data of a patient's anatomy by a medical imaging system having at least one processing device. The method also includes filtering the three-dimensional data to display a portion of the three-dimensional image data associated with the patient's anatomy, and receiving input from an operator input device at the processing device. The input includes navigation directions for virtual movement within a space defined by the three-dimensional image data. The method further includes tracking the virtual movement and generating a first model of the patient's anatomy based on the tracked virtual movement.

Description

[0001] This application is a divisional application of Chinese Patent Application 201780067272.5 (PCT / US2017 / 065162), filed on December 7, 2017, entitled "System and method for navigation in image-guided medical procedures".

[0002] Related applications

[0003] This patent application claims priority and benefit to U.S. Provisional Patent Application US62 / 431,696, filed December 8, 2016, entitled “Systems and Methods for Navigation in Image-Guided Medical Procedures,” which is incorporated herein by reference in its entirety. Technical Field

[0004] This disclosure relates to systems and methods for controlling steerable elongated devices. Background Technology

[0005] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. These techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, the operator can insert minimally invasive medical instruments (including surgical, diagnostic, therapeutic, or biopsy instruments) to reach the target tissue location. One such minimally invasive technique uses a flexible and / or steerable elongated device, such as a flexible catheter, which can be inserted into an anatomical channel and navigated toward an area of ​​interest within the patient's anatomy. Controlling such an elongated device by the medical staff involves the management of several degrees of freedom, including at least the management of insertion and retraction of the elongated device and its steerability. Furthermore, different operating modes can be supported.

[0006] To facilitate this minimally invasive medical technique, imaging registration technology can be used to correlate at least preoperative or intraoperative imaging modalities with the orientation and / or orientation of the inserted minimally invasive medical device for navigation and positioning of the device relative to the target tissue location within the patient's body. In this way, the operator or other personnel can more accurately guide and control the operation of the minimally invasive medical device.

[0007] Therefore, it would be beneficial to provide improvements in the use of medical imaging during minimally invasive medical techniques. Summary of the Invention

[0008] Embodiments of the invention are fully summarized by the claims appended to the specification.

[0009] Consistent with some embodiments, a method includes receiving three-dimensional image data of at least a portion of a patient's anatomy by a medical imaging system having at least one processing device; identifying by the processing device a portion of the three-dimensional image data associated with that portion of the patient's anatomy; receiving at the processing device input from an operator input device, the input including a navigation direction for virtual movement within a space defined by the three-dimensional image data; tracking the virtual movement; and generating a model of that portion of the patient's anatomy based on the tracked virtual movement. Other embodiments include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each computer program configured to perform actions of the method.

[0010] Consistent with other embodiments, a system for processing medical images is provided. The system may include a memory storing a set of three-dimensional image data of at least a portion of a patient's anatomy; and a processing means in communication with the memory, configured to execute instructions to perform operations. Operations may include receiving three-dimensional image data of at least a portion of the patient's anatomy, identifying a portion of the three-dimensional image data, and receiving input from an operator input device. The input may define a path within an image space defined by the portion of the three-dimensional image data. Operations may also include generating a model of that portion of the patient's anatomy based on the path within the image space defined by the three-dimensional image data. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each computer program configured to perform actions of the method.

[0011] Consistent with other embodiments, a system for displaying and interacting with medical images is provided. The system may include a memory storing a set of three-dimensional image data of at least a portion of a patient's anatomy; and a processing means communicating with the memory. The processing means is configured to execute instructions stored in the memory to perform operations. Operations may include rendering a graphical user interface in a display communicating with the processing means, receiving three-dimensional image data of at least a portion of the patient's anatomy, and identifying a portion of the three-dimensional image data associated with a portion of the patient's anatomy. Operations may also include receiving input from an operator input device defining a path within an image space defined by the three-dimensional image data. The system also includes generating a model of that portion of the patient's anatomy based on the path within the image space defined by the three-dimensional image data. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each computer program configured to perform actions of the method.

[0012] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the disclosure without limiting its scope. In this regard, other aspects, features, and advantages of the disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0013] Figure 1 This is a simplified diagram of a remotely operated medical system according to some embodiments.

[0014] Figure 2A This is a simplified diagram of a medical device system according to some embodiments.

[0015] Figure 2B This is a simplified diagram of a medical device with extended medical tools according to some embodiments.

[0016] Figure 3A and Figure 3B This is a simplified side view of a medical device mounted on an insertion assembly, according to some embodiments, in patient coordinate space.

[0017] Figure 4 This is a flowchart illustrating a general method for providing image-guided guidance for image-guided minimally invasive medical procedures, according to some embodiments of the present disclosure.

[0018] Figure 5A This is a flowchart of a method for generating a model or a portion thereof from three-dimensional image data without performing a segmentation process, according to some embodiments of the present disclosure.

[0019] Figure 5B This is a flowchart of a method for generating a model or a portion of a model from three-dimensional image data using a segmentation process for a portion of model generation, according to some embodiments of the present disclosure.

[0020] Figure 6A , Figure 6B and Figure 6C This is a two-dimensional rendering of exemplary medical image data according to some embodiments of this disclosure.

[0021] Figure 7A , Figure 7B and Figure 7C Some embodiments according to this disclosure are described by means of... Figures 6A-6C Multiple two-dimensional views of filtered image data are obtained by filtering the image data separately.

[0022] Figure 7D , Figure 7E and Figure 7F According to some embodiments of this disclosure Figures 7A-7C The image shown is a front view of the three-dimensional filtered image data.

[0023] Figure 8A This is a three-dimensional rendering of exemplary medical image data according to some embodiments of this disclosure.

[0024] Figure 8B It has a line-based navigation path model. Figure 8A Three-dimensional rendering of exemplary medical image data.

[0025] Figure 9A , Figure 9B , Figure 9C and Figure 9D Exemplary image data, including data corresponding to the objectives of a medical procedure, is shown according to some embodiments of this disclosure.

[0026] Figure 10 According to some embodiments of this disclosure, a perspective view within three-dimensional image data is shown pointing to... Figures 9A-9D The indicator arrow for the identified target.

[0027] Figure 11A An internal view of a surface model generated by segmenting image data according to some embodiments of the present disclosure is shown.

[0028] Figure 11B The illustration shows a combination of some embodiments according to this disclosure. Figure 11A The internal view of the surface model and the view from which the filtered image data of the surface model is obtained.

[0029] Figure 12A , Figure 12B , Figure 12C and Figure 12D The process of generating a model of an exemplary patient anatomy is illustrated according to some embodiments of the present disclosure.

[0030] Figure 13 These are exemplary user interfaces according to some embodiments of this disclosure.

[0031] Figure 14 A flowchart is shown of a method 1300 for providing image-based guidance during minimally invasive medical procedures according to some embodiments of the present disclosure.

[0032] The embodiments and advantages of this disclosure can be best understood by referring to the following detailed description. It should be understood that the same reference numerals are used to identify the same elements shown in one or more of the figures, wherein the elements shown in the figures are for illustrative purposes of embodiments of this disclosure and not for limiting the scope of embodiments of this disclosure. Detailed Implementation

[0033] In the following description, specific details are set forth to describe some embodiments consistent with this disclosure. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are illustrative and not restrictive. Those skilled in the art will recognize that while other elements are not specifically described in this disclosure, they are also within the scope and spirit of this disclosure. Furthermore, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments unless otherwise specifically described or if one or more features render the embodiment inoperable.

[0034] In some cases, well-known methods, procedures, components, and circuits are not described in detail to avoid unnecessarily obscuring aspects of the embodiments.

[0035] This disclosure describes various instruments and parts thereof in the state of three-dimensional space. As used herein, the term "position" refers to the location of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or part of an object (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the orientation of an object or part of an object in at least one translational degree of freedom and the orientation of an object or part of an object in at least one rotational degree of freedom (up to six total degrees of freedom). As used herein, the term "shape" refers to a set of poses, orientations, or orientations measured along an object.

[0036] Figure 1 This is a simplified diagram of a remotely operated medical system 100 according to some embodiments. In some embodiments, the remotely operated medical system 100 may be suitable for, for example, surgical, diagnostic, therapeutic, or biopsy procedures. Figure 1 As shown, the medical system 100 typically includes a remote operating manipulator component 102 for operating the medical device 104 to perform various procedures on the patient P. The remote operating manipulator component 102 is mounted on or near the operating table T. The master control component 106 allows the operator O (e.g., as...) Figure 1 The surgeon, clinician, or physician shown observes the intervention site and controls the remote manipulator component 102.

[0037] The master control unit 106 can be located at the operator's console, which is typically located in the same room as the operating table T, such as on the side of the operating table where the patient P is located. However, it should be understood that the operator O can be located in a different room from the patient P or in a completely different building. The master control unit 106 typically includes one or more control devices for controlling the remote operation manipulator unit 102. The control devices can include any number of various input devices, such as joysticks, trackballs, data gloves, trigger guns, manual controllers, voice recognition devices, body motion or presence sensors and / or similar devices. To provide the operator O with a strong sense of direct control of the instrument 104, the control devices can be provided with the same degrees of freedom as the associated medical instrument 104. In this way, the control devices provide the operator O with a remote presentation / presence or perception that the control devices and the medical instrument 104 are integrated.

[0038] In some embodiments, the control device may have more or fewer degrees of freedom than the associated medical device 104, and still provide remote presentation to the operator O. In some embodiments, the control device may optionally be a manual input device that moves in six degrees of freedom and may also include an actuable handle for actuating the device (e.g., for closing gripping jaws, applying a potential to electrodes, delivering drug therapy, and / or similar operations).

[0039] The remote-controlled manipulator assembly 102 supports the medical device 104 and may include a kinematic structure of one or more non-servo-controlled links (e.g., one or more links that can be manually positioned and locked in place, typically referred to as an assembly structure) and a remote-controlled manipulator. The remote-controlled manipulator assembly 102 may optionally include multiple actuators or motors that drive inputs on the medical device 104 in response to commands from a control system (e.g., control system 112). The actuators may optionally include a drive system that, when coupled to the medical device 104, can advance the medical device 104 into an anatomical opening created naturally or surgically. Other drive systems may move the distal end of the medical device 104 with multiple degrees of freedom, which may include three linear degrees of freedom (e.g., linear motion along the X, Y, Z Cartesian axes) and three rotational degrees of freedom (e.g., rotation about the X, Y, Z Cartesian coordinate axes). Additionally, the actuator can be used to actuate the articulated end effector of medical device 104 for grasping tissue in the jaws of a biopsy device and / or the like. Actuator orientation sensors, such as decoders, encoders, potentiometers, and other mechanisms, can provide sensor data to medical system 100 describing the rotation and orientation of the motor shaft. This orientation sensor data can be used to determine the motion of the object manipulated by the actuator.

[0040] The remote-operated medical system 100 may include a sensor system 108 having one or more subsystems for receiving information about an instrument of the remote-operated manipulator assembly 102. Such subsystems may include: an orientation / position sensor system (e.g., an electromagnetic (EM) sensor system); a shape sensor system for determining orientation, orientation, rate, velocity, posture, and / or shape along a distal end and / or one or more segments that may constitute a flexible body of the medical device 104; and / or a visualization system for capturing images from the distal end of the medical device 104.

[0041] The remote-operated medical system 100 also includes a display system 110 for displaying images or representations of the surgical site and medical device 104 generated by a subsystem of the sensor system 108. The display system 110 can also replace image data during planning and navigation operations. The display system 110 and the master control component 106 can be oriented such that the operator O can control the medical device 104 and the master control component 106 using telepresence / remote presence perception.

[0042] In some embodiments, medical device 104 may have a visualization system (discussed in more detail below) that may include an observation range component that records momentary or real-time images of the surgical site and provides the images to an operator or operator O via one or more displays of medical system 100 (such as one or more displays of display system 110). The momentary image may be, for example, a two-dimensional or three-dimensional image captured by an endoscope located within the surgical site. In some embodiments, the visualization system includes an endoscope component that may be integrally or detachably coupled to medical device 104. However, in some embodiments, a separate endoscope attached to a separate manipulator assembly may be used with medical device 104 to image the surgical site. The visualization system may be implemented as hardware, firmware, software, or a combination thereof, interacting with or otherwise executed by one or more computer processors, which may include a processing unit 114 of control system 112, which may include a central processing unit (CPU) and a graphics processing unit (GPU).

[0043] Display system 110 can also display images of the surgical site and medical instruments captured by the visualization system. In some examples, remote-operated medical system 100 can configure the controls of medical instrument 104 and master control component 106 such that the relative orientation of the medical instrument is similar to the relative orientation of the operator O's eyes and hands. In this way, operator O can manipulate medical instrument 104 and hand controls as if observing the workspace in a substantially realistic presence. In terms of realistic presence, it means that the image presentation is a realistic perspective image simulating the viewpoint of the operator physically manipulating medical instrument 104.

[0044] In some examples, the display system 110 can use image data from imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescence examination, temperature recording, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar techniques to present images of the surgical site recorded preoperatively or intraoperatively. The preoperative or intraoperative image data can be presented as two-dimensional, three-dimensional, or four-dimensional images (including, for example, time-based or velocity-based information) and / or as images of models created from the preoperative or intraoperative image dataset.

[0045] In some embodiments, typically for the purpose of image-guided surgical procedures, the display system 110 may display a virtual navigation image in which the actual position of the medical device 104 is registered (i.e., dynamically referenced) with preoperative or real-time images / models. This presents a virtual image of the internal surgical site to the operator O from the viewpoint of the medical device 104. In some examples, the viewpoint may be from the tip of the medical device 104. Images and / or other graphic or alphanumeric indicators of the tip of the medical device 104 may be overlaid on the virtual image to aid the operator O in controlling the medical device 104. In some examples, the medical device 104 may be invisible in the virtual image.

[0046] In some embodiments, display system 110 may display a virtual navigation image in which the actual position of medical device 104 is registered with preoperative or live images to present a virtual image of medical device 104 within the surgical site to operator O from an external viewpoint. An image of a portion of medical device 104 or other graphic or alphanumeric indicators may be overlaid on the virtual image to assist operator O in controlling medical device 104. As described herein, a visual representation of data points may be presented to display system 110. For example, measurement data points, movement data points, registration data points, and other data points described in this disclosure may be displayed visually on display system 110. Data points may be visually represented in a user interface as multiple points or dots on display system 110, or as a rendered model, such as a grid or line model created based on a set of data points. In some examples, data points may be color-coded according to the data they represent. In some embodiments, the visual representation may be refreshed in display system 110 after each processing operation to change the data points has been implemented.

[0047] The remote-operated medical system 100 may also include a control system 112. The control system 112 includes at least one memory 116 and at least one processing device 114 for implementing control between the medical device 104, the main control component 106, the sensor system 108, and the display system 110. The control system 112 also includes programming instructions 120 (e.g., a non-transitory machine-readable medium storing instructions) to implement some or all of the methods described according to various aspects of the disclosure herein, including instructions for providing information to the display system 110. While the control system 112... Figure 1 The simplified schematic is shown as a single block, but the system may include two or more data processing circuits, wherein a portion of the processing is optionally executed on or near the remote operating manipulator component 102, another portion of the processing is executed at the master control component 106, and / or the like. The processing means 114 of the control system 112 can execute instructions 120, which include instructions corresponding to the processes disclosed herein and described in more detail below. Any of a variety of centralized or distributed data processing architectures can be employed. Similarly, the programming instructions 120 can be implemented as multiple separate programs or subroutines, or they can be integrated into many other aspects of the remote operating system described herein. In one embodiment, the control system 112 supports wireless communication protocols such as Bluetooth, IrDA (Infrared Data Communication), HomeRF (Home Radio Frequency), IEEE 802.11, DECT (Digital Enhanced Wireless Communication), and wireless telemetry.

[0048] As will be described in more detail herein, memory 116 may store medical image data 122, including data obtained from various medical imaging modalities, including high-resolution and low-resolution CT imaging systems. Additionally, as will be described in more detail herein, memory 116 of control system 112 may include one or more models 124. These models 124 may originate from image data 122 and / or from user input received via master component 106 or other input mechanisms (such as input device 130, which may include a keyboard, mouse, drawing tablet, etc.), whereby operator O can virtually navigate within and / or draw on the image data, as will be described in more detail below. For example, in some embodiments, control system 112 may be a medical workstation providing an interface through which operator O can plan medical procedures. In some other or alternative embodiments, control system 112 may be part of a minimally invasive surgical system for performing medical procedures.

[0049] In some embodiments, the control system 112 may receive force and / or torque feedback from the medical device 104. In response to the feedback, the control system 112 may send a signal to the master control component 106. In some examples, the control system 112 may send a signal instructing one or more actuators of the remote operation manipulator component 102 to move the medical device 104. The medical device 104 may extend to an internal surgical site within the patient P's body via an opening in the patient P's body. Any suitable conventional and / or specialized actuators may be used. In some examples, one or more actuators may be separate from or integrated with the remote operation manipulator component 102. In some embodiments, one or more actuators and the remote operation manipulator component 102 are provided as part of a remote operation trolley positioned adjacent to the patient P and the operating table T.

[0050] The control system 112 may optionally further include a virtual visualization system to provide navigational assistance to the operator O when controlling the medical device 104 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system may be based on a reference to a preoperative or intraoperative dataset of acquired anatomical passages. The virtual visualization system processes images of the surgical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescein examination, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar techniques. Software used in conjunction with manual input is used to convert the recorded images into segmented two-dimensional or three-dimensional composite representations of parts or entire anatomical organs or regions. The image dataset is associated with the composite representation. The composite representation and the image dataset describe the various locations and shapes of the passages and their connectivity. Images used to generate the composite representation may be recorded preoperatively or intraoperatively during the clinical procedure. In some embodiments, the virtual visualization system may use a standard representation (i.e., not patient-specific) or a mixture of standard representations and patient-specific data. Composite representations and any virtual images generated from composite representations can represent the static posture of deformable anatomical regions during one or more phases of motion (e.g., during the inspiratory / expiratory cycle of the lungs).

[0051] During the virtual navigation procedure, sensor system 108 can be used to calculate the approximate position of medical device 104 relative to the anatomical structure of patient P. This position can be used to generate both a macroscopic (external) tracking image of the anatomical structure of patient P and a virtual internal image of the anatomical structure of patient P. The system can implement one or more electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors to register the medical device with pre-recorded surgical images and will display the medical device with the pre-recorded surgical images together. Such systems, such as those from virtual visualization systems, are known. For example, U.S. Patent Application US13 / 107,562 (filed May 13, 2011) (disclosing “Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery”) discloses such a system, which is incorporated herein by reference in its entirety. The remotely operated medical system 100 may also include optional operating and support systems (not shown), such as lighting systems, steering control systems, flushing systems, and / or suction systems. In some embodiments, the remote-operated medical system 100 may include one or more remote-operated manipulator components and / or one or more master control components. The exact number of remote-operated manipulator components will depend on the surgical procedure and space constraints within the operating room, as well as other factors. The master control components 106 may be juxtaposed or may be located in different locations. Multiple master control components allow more than one operator to control one or more remote-operated manipulator components in various combinations.

[0052] Figure 2A This is a simplified diagram of a medical device system 200 according to some embodiments. In some embodiments, the medical device system 200 may be used as a medical device 104 in an image-guided medical procedure performed using a remotely operated medical system 100. In some examples, the medical device system 200 may be used for non-remotely operated exploratory procedures or procedures involving conventional manual medical instruments, such as endoscopy. Optionally, the medical device system 200 may be used to collect (i.e., measure) a set of data points corresponding to locations within an anatomical passageway of a patient (such as patient P).

[0053] The medical device system 200 includes an elongated device 202, such as a flexible catheter, coupled to a drive unit 204. The elongated device 202 includes a flexible body 216 having a proximal end 217 and a tip or distal end. In some embodiments, the flexible body 216 has an outer diameter of approximately 3 mm. Other flexible bodies may have larger or smaller outer diameters.

[0054] The medical device system 200 also includes a tracking system 230 that uses one or more sensors and / or imaging devices to determine the orientation, rate, velocity, posture, and / or shape along the distal end 218 and / or one or more segments 224 of the flexible body 216, as described in further detail below. The entire length of the flexible body 216 between the distal end 218 and the proximal end 217 can be effectively divided into segments 224. The tracking system 230 is used if the medical device system 200 is aligned with the medical device 104 of the remotely operated medical system 100. The tracking system 230 may optionally be implemented as hardware, firmware, software, or a combination thereof, which interacts with or is otherwise executed by one or more computer processors, which may include Figure 1 The processor of the control system 112.

[0055] The tracking system 230 may optionally use a shape sensor 222 to track the distal end 218 and / or one or more segments 224. The shape sensor 222 may optionally include an optical fiber aligned with the flexible body 216 (e.g., provided within an internal channel (not shown) or externally mounted). In one embodiment, the optical fiber has a diameter of approximately 200 μm. In other embodiments, the size may be larger or smaller. The optical fiber of the shape sensor 222 forms an optical fiber bending sensor for determining the shape of the flexible body 216. In an alternative, an optical fiber including a fiber Bragg grating (FBG) is used to provide strain measurements of the structure in one or more dimensions. Various systems and methods for monitoring the shape and relative orientation of optical fibers in three dimensions are described in the following documents: U.S. Patent Application US11 / 180,389 (filed July 13, 2005) (disclosing "Fiber optic position and shape sensing device and method relating thereto"); U.S. Patent Application US12 / 047,056 (filed July 16, 2004) (disclosing "Fiber-optic shape and relative positions sensing"); and U.S. Patent US6,389,187 (filed June 17, 1998) (disclosing "Optical Fiber BendSensor"), which are incorporated herein by reference in their entirety. In some embodiments, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In some embodiments, the shape of the elongated device may be determined using other techniques. For example, the history of the distal orientation of the flexible body 216 may be used to reconstruct the shape of the flexible body 216 over time intervals. In some embodiments, the tracking system 230 may optionally and / or additionally use an orientation sensor system 220 to track the remote end 218. The orientation sensor system 220 may be a component of an EM sensor system, wherein the orientation sensor system 220 includes one or more conductive coils capable of withstanding an externally generated electromagnetic field. Each coil of the EM sensor system 220 then generates an induced electrical signal having characteristics dependent on the orientation and orientation of the coil relative to the externally generated electromagnetic field. In some embodiments, the orientation sensor system 220 may be configured and positioned to measure six degrees of freedom (e.g., three azimuth coordinates X, Y, Z and three orientation angles indicating pitch, yaw, and roll of a reference point) or five degrees of freedom (e.g., three azimuth coordinates X, Y, Z and two orientation angles indicating pitch and yaw of a reference point).A further description of the orientation sensor system is provided in U.S. Patent 6,380,732 (filed August 11, 1999) (disclosed as "Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked"), which is incorporated herein by reference in its entirety.

[0056] In some embodiments, the tracking system 230 may alternatively and / or additionally rely on historical posture, orientation, or orientation data stored for known points of the instrument system along alternating cycles of motion (such as breathing). This stored data can be used to develop shape information about the flexible body 216. In some examples, a series of orientation sensors (not shown) (such as electromagnetic (EM) sensors similar to those in orientation sensor 220) may be positioned along the flexible body 216 and then used for shape sensing. In some examples, the history of data acquired from one or more of these sensors during the procedure can be used to represent the shape of the elongated device 202, particularly where the anatomical passage is typically static.

[0057] The flexible body 216 includes a passage 221, the size and shape of which are designed to receive a medical device 226. Figure 2B This is a simplified diagram of a flexible body 216 having an extended medical device 226 according to some embodiments. In some embodiments, the medical device 226 can be used for procedures such as surgery, biopsy, ablation, illumination, irrigation, or aspiration. The medical device 226 can be deployed through a passage 221 of the flexible body 216 and used at a target location within an anatomical structure. The medical device 226 may include, for example, an image capture probe, a biopsy instrument, laser ablation fibers, and / or other surgical, diagnostic, or therapeutic tools. Medical tools may include end effectors having a single working component, such as a scalpel, a blunt blade, an optical fiber, an electrode, and / or the like. Other end effectors may include, for example, forceps, grippers, scissors, clamps, and / or the like. Other end effectors may also include electrically activated end effectors, such as electrosurgical electrodes, transducers, sensors, and / or the like. In various embodiments, the medical device 226 is a biopsy instrument that can be used to remove sample tissue or cell samples from a target anatomical location. The medical device 226 may also be used in conjunction with an image capture probe within the flexible body 216.

[0058] In various embodiments, medical device 226 may be an image capture probe including a distal portion having a stereo or single-field-of-view camera at or near the distal end 218 of flexible body 216 for capturing images (including video images) processed by visualization system 231 for display and / or providing to tracking system 230 to support tracking of distal end 218 and / or one or more segments 224. The image capture probe may include a cable coupled to the camera for transmitting captured image data. In some examples, the image capture device may be a bundle of optical fibers, such as a fiber optic endoscope, coupled to visualization system 231. The image capture device may be monospectral or multispectral, capturing image data of one or more in visible, infrared, and / or ultraviolet light. Alternatively, medical device 226 itself may be an image capture probe. Medical device 226 may be advanced from an opening in passage 221 to perform the procedure and then retracted into the passage when the procedure is complete. The medical device 226 can be removed from the proximal end 217 of the flexible body 216 or from another optional device port (not shown) along the flexible body 216.

[0059] Medical device 226 may additionally accommodate a cable, linkage, or other actuation control (not shown) extending between its proximal and distal ends to controllably bend the distal end of medical device 226. Steering instruments are described in U.S. Patent 7,316,681 (filed October 4, 2005) (disclosing “Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity”) and U.S. Patent Application 12 / 286,644 (filed September 30, 2008) (disclosing “Passive Preload and Capstan Drive for Surgical Instruments”), which are incorporated herein by reference in their entirety.

[0060] The flexible body 216 may also accommodate cables, linkages, or other steering controls (not shown) extending between the drive unit 204 and the distal end 218 to controllably bend the distal end 218, as illustrated, for example, through the dashed line depicting portion 219 of the distal end 218. In some examples, at least four cables are used to provide independent “up-down” steering to control the pitch of the distal end 218 and “left-right” steering to control the yaw of the distal end 281. Steering elongated devices are described in detail in U.S. Patent Application US13 / 274,208 (filed October 14, 2011) (published “Catheter with Removable Vision Probe”), which is incorporated herein by reference in its entirety. In embodiments where the medical device system 200 is actuated by a remotely operated component, the drive unit 204 may include a drive input that is detachably coupled to and receives power from a drive element (such as an actuator) of the remotely operated component. In some embodiments, the medical device system 200 may include clamping features, manual actuators, or other components for manually controlling the movement of the medical device system 200. The elongated device 202 may be steerable, or alternatively, the system may be non-steerable and lack integrated mechanisms for allowing an operator to control the bending of the distal end 218. In some examples, one or more lumens are defined in the walls of the flexible body 216 through which the medical device can be deployed and used at a target surgical site.

[0061] In some embodiments, the medical device system 200 may include flexible bronchial instruments, such as bronchoscopes or bronchial tubes for the examination, diagnosis, biopsy, or treatment of the lungs. The medical device system 200 is also suitable for navigating and treating other tissues via naturally or surgically created access channels in any of a variety of anatomical systems, including the colon, intestine, kidneys and renal tubules, brain, heart, circulatory system including the vascular system, and / or the like.

[0062] Information from tracking system 230 can be sent to navigation system 232, where it is combined with information from visualization system 231 and / or preoperatively acquired models to provide the operator with real-time orientation information. In some examples, the real-time orientation information can be displayed... Figure 1 The display system 110 controls the medical device system 200. In some examples, Figure 1The control system 112 can use orientation information as feedback for positioning the medical device system 200. Various systems using fiber optic sensors to register and display surgical instruments and surgical images are disclosed in U.S. Patent Application US13 / 107,562, filed May 13, 2011, entitled "Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery," which is incorporated herein by reference in its entirety.

[0063] In some examples, the medical device system 200 can Figure 1 Remote operation of the medical system 100. In some embodiments, Figure 1 The remote operating manipulator component 102 can be replaced by a direct operator control. In some examples, the direct operator control may include various handles and operator interfaces for handheld operation of the instrument.

[0064] Figure 3A and Figure 3B This is a simplified side view of a medical device mounted on an insertion assembly, according to some embodiments, in patient coordinate space. Figure 3A and 3BAs shown, the surgical environment 300 includes a patient P positioned on a platform 302. The patient's overall movement is restricted by sedation, restraint, and / or other means, in which case patient P can remain stationary within the surgical environment. Circulatory anatomical movement, including the patient P's breathing and cardiac movements, can continue unless the patient is instructed to hold their breath to temporarily cease respiratory movement. Therefore, in some embodiments, data can be collected at specific stages of breathing and labeled and identified with that stage. In some embodiments, the stage at which data was collected can be inferred from the physiological information collected from patient P. Within the surgical environment 300, a medical device 304 is coupled to an instrument holder 306. In some embodiments, the medical device 304 can be acquired using EM sensors, shape sensors, and / or other sensor modalities that characterize the orientation, orientation, and / or shape information of the medical device 304. The instrument holder 306 is mounted to an insertion stage 308 fixed within the surgical environment 300. Alternatively, the insertion stage 308 may be movable but has a known position within the surgical environment 300 (e.g., via a tracking sensor or other tracking device). The instrument holder 306 may be a component of a remotely operated manipulator assembly (e.g., remotely operated manipulator assembly 102) coupled to the medical device 304 to control the insertion movement (i.e., movement along the A-axis) of the distal end 318 of the elongated device 310, and optionally control the movement of the distal end 318 of the elongated device 310 in multiple directions including yaw, pitch, and roll. The instrument holder 306 or the insertion stage 308 may include actuators, such as servo motors (not shown), that control the movement of the instrument holder 306 along the insertion stage 308.

[0065] An elongated device 310 is coupled to an instrument body 312. The instrument body 312 is coupled to and fixed relative to an instrument holder 306. In some embodiments, a fiber optic shape sensor 314 is fixed at a proximal point 316 on the instrument body 312. In some embodiments, the proximal point 316 of the fiber optic shape sensor 314 may be movable along the instrument body 312, but the location of the proximal point 316 may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 314 measures the shape from the proximal point 316 to another point (such as the distal end 318 of the elongated device 310). The medical device 304 may be substantially similar to the medical device system 200.

[0066] As the orientation measuring device 320 moves along the insertion axis A on the insertion stage 308, it provides information about the orientation of the instrument body 312. The orientation measuring device 320 may include a decoder, encoder, potentiometer, and / or other sensors that determine the rotation and / or orientation of actuators that control the movement of the instrument carriage 306 and thus the movement of the instrument body 312. In some embodiments, the insertion stage 308 is linear. In some embodiments, the insertion stage 308 may be curved or have a combination of curved and linear portions.

[0067] Figure 3A The device body 312 and device holder 306 are shown in a retracted position along the insertion stage 308. In this retracted position, the proximal point 316 is located at orientation L0 on axis A. In this orientation along the insertion stage 308, the component A of the position of the proximal point 316 can be set to zero and / or other reference values ​​to provide a reference for describing the orientation of the device holder 306 on the insertion stage 308, and thus the orientation of the proximal point 316 on the insertion stage 308. In this retracted orientation of the device body 312 and device holder 306, the distal end 318 of the elongated device 310 can be positioned precisely within the access port of the patient P. Also in this orientation, the orientation measuring device 320 can be set to zero and / or other reference values ​​(e.g., I = 0). Figure 3B In the process, the instrument body 312 and the instrument holder 306 are advanced along the linear track of the insertion stage 308, and the distal end 318 of the elongated device 310 has been advanced into the patient P.

[0068] Figure 4 This is a flowchart illustrating a general method 400 for enabling image-guided minimally invasive medical procedures. Method 400 can provide medical images and instruments (such as the elongated device 202 in Figure 2 and / or...) Figure 3A and Figure 3B Registration between medical devices 304. Method 400 is shown as a series of steps or operations. Embodiments of method 400 may include additional or alternative operations before, after, between, or as part of the enumerated operations. In some embodiments of this disclosure, a set of instructions may be stored on a computer-readable medium that, when executed by a processor, will cause a machine having a processor to perform embodiments of method 400.

[0069] Therefore, method 400 can begin at operation 402, wherein preoperative or intraoperative image data is obtained from an imaging system that uses computed tomography (CT), magnetic resonance imaging (MRI), fluorescein scanning, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, or other suitable imaging modalities to provide image data. In some embodiments, the image data is stored image data obtained by retrieval from memory. In various embodiments, the preoperative or intraoperative image data may correspond to two-dimensional, three-dimensional, or four-dimensional (including, for example, time-based or velocity-based information) images. For example, the image data may be low-resolution or low-dose three-dimensional CT data representing a portion of patient P. The image data may represent the upper or lower torso of patient P and include data representing patient P's heart, lungs, stomach, liver, intestines, ribs, muscles, etc. Other embodiments may include image data from any other region of patient P. The image data may be stored in... Figure 1 Image data 122 in memory 116.

[0070] At operation 404, standalone computer software or computer software combined with manual input is used to convert the recorded images into a two-dimensional or three-dimensional composite representation or model of a part or the whole of an anatomical organ or region. Some methods for generating a model from image data may include a segmentation process that identifies certain features of the model (such as the curvature of bronchial passages in the lungs), extracts the curvature, and uses the curvature to generate a centerline model representing the bronchial passages. This segmentation relies on artificial intelligence to generate the model, such as the centerline model. However, segmentation may fail when image quality is insufficient or for other reasons. Because image quality is critical for segmentation, higher doses of imaging agents and / or radiation may be used and / or required to provide image data of sufficient quality for automatic segmentation. Therefore, as described herein, the conversion from imaging data to a model can be done without segmenting the image data, without applying a segmentation process, and at operation 404, input from operator O can be received and used to generate the model. For example, the input may include navigation directions for virtual movement within a space defined by image data received from one or more input devices 130. The navigation direction can be used in place of the centerline model or as a centerline model, similar to the centerline model that can be obtained from the segmentation process.

[0071] At operation 406, when patient P is on the operating table T, the model obtained from the operator's input can be used to register image data to patient P, such as... Figure 1 and Figures 3A-3BAs shown. By registering the obtained model and associated image data to a patient P, the image data can be used for navigation and positioning of a minimally invasive instrument within the patient P to perform image-guided procedures. Typically, registration involves matching measurement points to points obtained from the model using rigid and / or non-rigid transformations. Measurement points can be generated by driving the minimally invasive instrument to landmarks in the anatomical structure and tracking the instrument's orientation using electromagnetic coil scanning and tracking during the procedure or using a shape sensor system. Measurement points can be generated for use in iterative nearest point (ICP), or other point set registration methods can also be used in the registration process within the scope of this disclosure. Suitable examples of point cloud registration techniques can be found in U.S. Provisional Patent Application US62 / 205440, filed August 14, 2015, entitled “SYSTEMS AND METHODS OF REGISTRATION FORIMAGE-GUIDED SURGERY,” and PCT / US16 / 046633, filed August 11, 2016, entitled “SYSTEMS AND METHODS OF REGISTRATION FORIMAGE-GUIDED SURGERY,” the disclosures of which are incorporated herein by reference in their entirety. Other registration techniques may be used in other embodiments.

[0072] Figure 5A This is a flowchart of method 500 for generating a model or a portion of a model from 3D image data without performing a segmentation process. Some embodiments of method 500 can be understood as follows: Figure 4 An embodiment of operation 404. Method 500 is described as a series of enumerated operations. Embodiments of method 500 may include, for example... Figure 5A Additional or alternative operations are performed before, after, between, or as part of method 500. Some embodiments of method 500 may omit one or more enumerated operations. Additionally, some embodiments of method 500 include a set of instructions stored on a computer-readable medium, similar to those stored in... Figure 1 The instructions 120 are stored in memory 116. The processing device can execute this set of instructions to enable medical systems (such as...) Figure 1 The system 100 or its components perform the operation of method 500.

[0073] As shown in the figure, method 500 can begin at operation 502, wherein a medical system having at least one processing device receives three-dimensional image data of at least a portion of a patient's anatomical structure. For example, processing device 114 of control system 112 can receive image data. The three-dimensional image data can be CT data, including low-dose CT data, or other image data obtained from different imaging modalities. The image data can represent the imaged tissues and anatomical structures as a set of voxels located in a three-dimensional image space or image reference frame. Each voxel can include a density value (such as a radiation density value) or other values ​​that can be used to distinguish different types of tissues, fluids, structures, etc., within the image space. For example, when the image data is CT image data, a Hounsfield value measured in Hounsfield units / Henness units can be associated with each voxel. Other radiation density values ​​may be used in other embodiments. Figures 6A-6C Exemplary medical image data that can be used at operation 502 is shown.

[0074] At operation 504, the processing device can identify a portion of three-dimensional image data associated with one or more anatomical channels in the imaging portion of the patient's anatomy. Figures 7A-7F Two-dimensional and three-dimensional views of filtered image data obtained by filtering image data are shown. As part of identifying a portion of the three-dimensional image data, the processing device can filter the image data based on the Hounsfield value of each voxel. For example, the processing device can use a lower Hounsfield value threshold and / or a higher Hounsfield value threshold to filter the image data to identify specific features within the image space. In some examples, the density of anatomical structures or tissues can vary depending on the structure or tissue type, and is cross-correlated with high or low Hounsfield values. For example, because the air density within the anatomical passages of the lung is very low, a correspondingly low Hounsfield value filter can be applied to the image data to effectively isolate the air within the anatomical passages of the lung. In this way, the processing device can identify the boundaries of the anatomical passages within the lung within the image data.

[0075] Additionally, one or more thresholds can be applied adaptively, causing different regions of the 3D image data to undergo different thresholds. For example, to identify structures within the image data, a first threshold for identifying major airways within the image data can be applied. This first threshold can be applied during the first pass of the data. Subsequently, a second pass of the data filter can be applied. The second pass may include a second threshold, which better identifies smaller branches within the airways included in the 3D image data. In this and other ways, adaptive airway thresholding can be used to identify anatomical pathways in the image data. In some embodiments, the second threshold can be applied based on the terminal voxel identified in the first pass. In other embodiments, a replicated dataset including the 3D image data can undergo different thresholds and then be combined. Such a process can resolve a certain amount of noise present in the image data.

[0076] In other embodiments, the processing device may filter the image data to identify tissue that forms the actual wall of the bronchial passages of the lung or blood vessels located just outside the bronchial wall of the lung. In some embodiments, user input may be received in a request to display a specific type of tissue, or as a request to adapt the Hounsfield value filter to a specific setting or adjustment. Some other types of tissue or material that can be identified and selectively displayed include: bone, muscle, blood vessels, bronchial walls, pleura, tumors, lesions, and fluids such as blood. As described herein, organs other than the lung can be analyzed using the features and processing described herein, allowing other tissues and materials to be displayed. The filtered image data may be presented to operator O on a display (such as display system 110), and operator O may interact with control system 112 to adjust one or more filters applied to the data.

[0077] At operation 506, after the filtered image data is displayed on the monitor, the processing device can access the data from the operator input device (such as...). Figure 1 One of the input devices 130 receives input. The input can define the navigation direction of virtual movement within the image space of the filtered image data. Figure 8AThe image space of filtered image data that an operator can view while navigating and providing input to an operator input device is illustrated. In this example, operator O can manipulate the filtered image data such that the perspective of the image data is focused on the upper opening of the trachea, which operator O can visually identify in the image data. Once the perspective is set, operator O can move within the filtered image data using input device 130, while updating the display to show the new perspective of the filtered image data after each input is received. For example, operator O can use a keyboard with arrow keys, a mouse, a scroll wheel, a trackball, a 3D input device, and / or any other suitable input device to navigate within the filtered image data.

[0078] Optionally, at operation 507, the processing device may receive a portion of specified image data from the operator input device as a target (e.g., Figures 9B-9D The input of the target (800).

[0079] At operation 508, the processing device can track the input as a virtual movement within the image space. For example, the processing device can generate a list or history of received commands relative to the image space, such that a path defined by the input received from one or more input devices 130 can be generated by the processing device 114. As the operator O moves within the image space and the processing device tracks the virtual movement, the virtual movement can provide information about the model of the anatomical passage for virtual navigation. At operation 510, the tracked path can be used to generate a model of the navigation portion of the patient's anatomy. For example, the tracked path can form a linear model with one or more lines in three-dimensional space. By maintaining a perspective / viewpoint within the anatomical passage while virtually navigating through it, the operator O can generate these lines in the three-dimensional image space. The lines or path can then define a model similar to a centerline model, which will be generated by a segmentation process. However, the navigation path is obtained without using segmentation of the image data. The operator O can interact with the input device 130 to indicate whether the navigation path is approximately centered within the anatomical passage or whether it is set near the bottom or top edge of the anatomical passage. Figure 8B A line-based navigation path model 604 is shown. One or more models can be generated from the navigation path using the diameter associated with the navigation channel.

[0080] Optionally, during operation 511, the processing device may provide guidance information to help guide the operator to a designated target (e.g., Figures 9B-9D The target is 800.

[0081] In some embodiments, input provided by operator O can generate the entire model, which can then be used in the registration process. In another embodiment, as... Figure 5B As described, the hybrid technique uses available segmentation information to supplement method 500. Figure 5B This is a flowchart of method 520 for generating a hybrid model or a portion of a model from 3D image data, wherein some portions of the model are generated using a segmentation process, and some portions of the model are generated without performing a segmentation process. Some embodiments of method 520 can be understood as follows: Figure 4 An embodiment of operation 404. Method 520 is described as a series of enumerated operations. Embodiments of method 520 may include, for example... Figure 5B Additional or alternative operations performed before, after, between, or as part of method 520. Some embodiments of method 520 may omit one or more enumerated operations. Additionally, some embodiments of method 520 include a set of instructions stored on a computer-readable medium, similar to those stored in... Figure 1 The instructions 120 are stored in memory 116. The processing device can execute this set of instructions to enable medical systems (such as...) Figure 1 The system 100 or its components perform the operation of method 500.

[0082] As shown in the figure, method 520 can begin at operation 502, as previously described for method 500. At operation 522, a segmentation algorithm can be used to segment the three-dimensional image data. Certain features of the segmentation-identified model, such as the curvature of the bronchial passages in the lungs, are extracted to extract the curvature. At operation 524, the features extracted from the segmentation process are used to generate a centerline model and / or surface model (e.g., a mesh model) representing the bronchial passages. For example, the segmentation algorithm can be used to generate a centerline model defining the trachea and main bronchi of the lungs. The centerline model can be displayed in the display system 110 along with the CT image data. At operation 526, optionally, the processing device can receive input from an operator input device to navigate through or along the model generated by the segmentation. For example, the operator can navigate along the centerline model until reaching the distal end of a portion of the centerline model. At operation 528, the termination point of the model generated by the segmentation is identified and can be used as the starting point for generating the model based on user input. Therefore, the segmentation algorithm can be used to generate a centerline model of a portion of the patient's anatomy, and then the input provided by the operator O can be used to continue the model, enhance the model, or add missing parts of the patient's anatomy to the model.

[0083] After reaching the end of the segmentation model, the method can continue to operation 506, as previously described, where the processing device can receive input from the operator's input device (such as...). Figure 1The input device 130 receives input. The input can define the navigation direction of virtual movement within the image space of the filtered image data. Optionally, the processing device can detect the end of the centerline model, and can automatically begin tracking the operator O's navigation movement when the operator O navigates beyond the centerline model within the filtered image data. As mentioned above, segmentation may fail when image quality degrades below a threshold. For example, low-dose CT image data can provide sufficient information for segmentation algorithms of larger channels within the lungs, but may fail when the channels narrow and when the number or resolution of voxels defining more distal channels decreases. However, the operator O can visually determine the approximate boundaries of these distal channels in the filtered image data shown in the display system 110. In the event of segmentation failure, the operator O can virtually navigate (i.e., navigate within the image space) through the image data to define a path that can be used to expand the segmentation model. At operation 508, the processing device can track the input as virtual movement within the image space. Therefore, in this example, operator-driven tracking navigation movements of the trachea and main bronchi beyond the lungs can provide input data that can be used to extend the centerline model to the secondary bronchi and further generation bronchi within the lungs, thus providing a more complete patient anatomy model. At operation 510, a hybrid model is generated from the segmentation data and the tracking path. For example, the hybrid model may include at least one centerline obtained from the segmentation and at least one navigation path obtained from the virtual navigation input received by the operator. Thus, an exemplary model may include a proximal centerline model portion obtained from the segmentation, having a distal end connected to a navigation path model portion, which in turn has a distal end connected to a distal centerline model portion obtained from the segmentation.

[0084] In some cases, unique aspects of a portion of an anatomical passage may cause segmentation failure at that particular section. For example, lesions, tumors, obstructions, or wounds may be present at that section of the anatomical passage and deform it in a way that the segmentation algorithm cannot resolve or is difficult to resolve. In some embodiments, operator O can virtually navigate through the unsegmented portion. Operator O can then request, via input device 130, that the segmentation algorithm can resume operation based on the distal end of the path defined by the virtual navigation. Because the problem causing the segmentation failure may not exist far from the specific portion, the segmentation algorithm can continue even after operator O has navigated beyond the problematic portion.

[0085] Now for reference Figure 6A , Figure 6B and Figure 6C The rendering of exemplary medical image data is shown in methods (e.g., methods 500, 520) that can generate models without performing a segmentation process. Figure 6A A front view of the chest region of patient P, taken in the axial plane, is shown. Figure 6B It is a side view of the chest region taken in the coronal plane, and Figure 6C This is a frontal view of the chest region taken in the sagittal plane. Figures 6A-6C The views shown are cross-sectional views obtained from 3D data. These views are "slices" that display two-dimensional planes within the 3D image data. Figures 6A-6C The air in the lungs is depicted as black.

[0086] Now for reference Figure 7A , Figure 7B and Figure 7C In these figures, the diagrams depict the process of... Figures 6A-6C Multiple on-axis views of filtered image data obtained by filtering the image data in the image. Figures 7A-7C Voxel 700 with a Hounsfield value is highlighted, which can be correlated with the air in the anatomical passages of the lungs. The air in patient P's lungs can be the least dense part of patient P and can be filtered by density value to isolate the anatomical passages. Figures 7A-7C These are cross-sectional slices depicting voxels with air density within the trachea and other channels in the lungs of patient P. These voxels can be identified by filtering according to Hounsfield values ​​and rendered differently from other voxels to highlight the anatomical channels to operator O. In some embodiments, the image data can be filtered to display the air within the anatomical channels of the lungs as a unique color to provide a model of the channels.

[0087] Figure 7D , Figure 7E and Figure 7F yes Figures 7A-7C The image data shown is an orthogonal view of the image data, which has been further filtered to highlight or depict specific aspects of the patient's anatomy included in the image data. Figure 7D Based on Hounsfield values, the image is filtered to depict the bone 702 of patient P and the vascular system 704 portion surrounding the bronchial walls 706 of patient P's lungs. Through the user interface, operator O can select a specific tissue for rendering in the user interface from a menu of selectable tissues and select the vascular system 704. Based on the selection, Figure 7E It can be presented accurately. Figure 7E The vascular system 704 and bronchial wall 706 are shown. The operator O can also interact with the user interface to select only the bronchial wall 706 for display, such as... Figure 7FAs seen below, selective filtering of imaging data based on the features of each voxel rather than its position can provide the desired rendering to operator O. Furthermore, when operator O virtually navigates through the data to generate a model as part of the planning process, by filtering the image data, computational resources (CPU and / or GPU resources) need to render the data and update the data view. This is combined with the following... Figure 13 As described in the user interface 1200, the operator O can interact with user interface elements to selectively apply filters to image data and modify the filters.

[0088] Figure 8A Depicted from a specific perspective relative to the anatomical channels within the image data Figures 6A-6C and Figures 7A-7D The three-dimensional image data. More specifically, Figure 8A A 3D rendering of the anatomical passage is shown from a driving perspective (e.g., from the perspective of a device (such as an endoscope) being navigated through the anatomical passage). Figure 8A The air in the middle lungs is rendered transparent, allowing the operator (O) to see the interior of the bronchial passages. By making the air in the lungs transparent, Figure 8A The 3D view provides the operator with a navigable perspective view of the CT image data. For example... Figure 8A As shown, the image data is filtered to render the air as transparent, while voxels with Hounsfield values ​​of the lung wall 600 are rendered. In this way, the inner surface of the lung wall 600 is presented to the operator O so that the operator O can virtually navigate within the wall 600. Figure 8A The perspective view shown is by Figures 6A-6C and Figures 7A-7C The vertical and horizontal alignments shown indicate (the vertical and horizontal alignments are in...) Figure 8A (The two lines intersect at the fluoroscopic point) and point towards the main carina of patient P's lung. For example... Figure 8A As shown, looking distally from the trachea, there are left bronchus 602A and right bronchus 602B.

[0089] Figure 8B It has a line-based navigation path model. Figure 8A Three-dimensional rendering of exemplary medical image data. Figure 8B Navigation path 604 is shown, which has been generated by performing previously described methods (such as method 500). Navigation path 604 can be generated by tracking the virtual movement of operator O within the image space. Figure 8BAs shown, operator O has navigated along the right bronchus 602B, forming navigation path 604, and has returned to the trachea. Operator O can then navigate along the left bronchus 602A, further executing procedures 506, 508, and 510 of method 500, which will generate another navigation path, such as navigation path 604. This other navigation path can be connected to navigation path 604 and combined to form a single line-based model of the patient P's lungs. The line-based model is generated based on the navigation input received from operator O as operator O virtually navigates within the image space.

[0090] Now for reference Figure 9A , 9B 9C and 9D, where the image data is shown, are similar to Figures 6A-6C and Figures 7A-7F The image data shown may include CT image data that includes data corresponding to a target 800 in a medical procedure. For example, target 800 may be a tumor, such as a lung tumor. In some embodiments, density filtering of CT data can facilitate the identification of the location and shape of the tumor. In some embodiments of method 500, processing device 114 may identify or preliminarily identify target 800. This can be done using Hounsfield values ​​associated with voxels of image data and / or comparisons based on stored CT data that do not include the target or include targets of different sizes from patient anatomy. For example, identification of target 800 can be facilitated by comparing image data collected at different times. In some embodiments, such as those described in optional procedures 507, 511 of method 500, a portion of specified image data may be received from operator O as input for target 800 via a user interface. For example, operator O may provide a display on display screen 110. Figures 9B-9D The input is taken from each of the two-dimensional slices. The input received on each of the three views can be used to generate a target recognition region, which is... Figure 9A The three-dimensional perspective view in the image shows the orientation in the three-dimensional image space as the center. The target recognition region may also include a three-dimensional shape formed around the central orientation. For example, the three-dimensional shape 802 may include, for example, Figure 9A The ellipse, cube, etc. shown are examples. The operator O can interact with UI (user interface) elements to control the size and shape of the three-dimensional shape 802. After receiving the orientation and shape of the three-dimensional shape 802, the three-dimensional shape 802 can be used as a target 800, or the processing device can identify the voxels within the shape 802 that constitute the target 800.

[0091] Figure 10 It shows in Figure 5AA portion of the image data obtained at operation 502 of method 500. After the target 800 has been identified within the image data, the processing device can determine the vector between the current viewpoint / perspective within the image data and the three-dimensional orientation of the target 800. For example... Figure 10 As shown, the current viewpoint / perspective can originate from the trachea at the carina of the lung. In this example, target 800 is identified as being located within a specific lobe (e.g., the left lobe), such that a vector provides directional indication of the target's location. The vector can be rendered as arrow 900 in the user interface, or another user interface element representing the vector can be used to show the operator O the position of target 800 relative to its navigational orientation within the 3D data. Alternatively, portions of the image data can be displayed as different colors, shadows, textures, or transparency indicating the target's location.

[0092] In some embodiments, after the target 800 has been identified within the image data, the processing device may determine that some modeling channels do not provide an entry point to the target. For example, the processing device may determine that certain channels have no points within a predetermined threshold distance from the target 800. As another example, the processing device may determine a subset of modeling channels that have at least one point within a threshold distance from the target 800. The threshold distance may range from about 0.5 cm to about 5 cm. In some embodiments, the threshold distance may range from about 1.5 cm to about 3 cm. After the processing device determines which modeling channels can provide an entry point to the target 800, the processing device may prevent navigation along any of the channels that do not provide an entry point to the target 800. Figure 10 As shown, marker 902 can be rendered along with image data to convey information to the operator. Figure 10 The channel shown on the right side of the image does not provide an entrance to target 800. While in some embodiments, the processing device may display marker 902 without preventing the operator from navigating along the corresponding model channel, other embodiments may display marker 902 and prevent navigation beyond the three-dimensional position associated with marker 902.

[0093] Now for reference Figure 11A and Figure 11B As shown, some embodiments of method 500 may additionally include the steps of segmenting image data to generate a surface model and / or generating a centerline model associated with the surface model or image data based on the image data received in process 502. Figure 11A As shown, the internal perspective view of surface model 1000 can be viewed from the inside. Figure 1 The surface model 1000 is presented in the display system 110. It can be obtained through any suitable segmentation process. However, as mentioned above, segmentation may not provide a partially segmented surface model. Therefore, as... Figure 11BAs shown, doctor O can request that surface model 1000 and filtered image data 1002 be displayed simultaneously in display system 110. Furthermore, operator O can interact with user interface elements to select between displaying surface model 1000 and displaying filtered image data 1002. Figure 11B The information view provided to the operator allows the operator to visualize any differences between the surface model 1000 and the associated filtered image data 1002. The operator can then continue processes 504 to 510 to supplement the partially segmented surface model and produce a more complete segmented surface model.

[0094] Additionally, in some embodiments of method 500, a segmentation process may be used first to generate a first surface model with an associated centerline model. However, because the segmentation process may occur far from the target (such as...), Figures 9A-9D If the target (800) fails at a certain distance, the operator O can then provide input to generate a model that extends further into the patient's anatomy, making the complete model available for image-guided medical procedures. In other words, the input received from the operator O can be used to expand the dissecting model.

[0095] therefore, Figure 12A , Figure 12B , Figure 12C and Figure 12D It shows Figure 5A Some steps of some embodiments of the method 500 shown provide an extension of the initial segmentation model when the initial segmentation model fails at a certain distance from the target. Figure 12A A set of filtered image data 1102 is shown, comprising a set of anatomical channels created from preoperative or intraoperative imaging data. In this embodiment, the channels are the airways of the human lungs, but method 500 is not limited to any particular anatomical structure. For ease of illustration, the filtered image data 1102 is depicted as two-dimensional. Thus, the depicted image data 1102 can be understood as... Figure 7F A two-dimensional representation of the three-dimensional bronchial wall 706. This set of filtered image data 1102 may also include information related to the target 1104 (such as, as described herein). Figures 9A-9D The image data corresponding to the target 800). Due to the quality of the image data 1102 or due to abnormalities in the structure of the anatomical channel, from Figure 1 The segmentation process performed by the processing device 114 cannot generate a surface model and / or cannot generate a centerline model extending from the trachea to the target 1104 through the anatomical channel, such as Figure 11A As shown. For example, Figure 12B Depicting what can be done Figure 5A The results of the segmentation process used in some embodiments of method 500. Figure 12BThe outline of an exemplary surface model 1106 is shown in solid lines. Figure 12B The undivided portion 1108 of the anatomical passage, which was not divided by the segmentation process, is also shown by dashed lines. Because the target 1104 can only be accessed via the undivided portion 1108 of the anatomical passage, it may be impossible or difficult to perform a minimally invasive medical procedure at the site of the target 1104 using only the surface model 1106.

[0096] After the initial segmentation process of generating surface model 1106, centerline segmentation model 1110 can be generated, such as... Figure 12C As shown. The centerline segmentation model 1110 may include a set of curves and / or straight lines that extend in three-dimensional space and correspond to the approximate center of the channel contained in the segmentation model 1106. The higher the resolution of the model, the more accurately the set of straight lines or curves in model 1110 will correspond to the center of the channel. Representing the lung with the centerline segmentation model 1110 can provide a smaller set of data, which can be processed more efficiently by one or more processors or processing cores compared to the data set of the segmentation model 1106 representing the channel wall. In this way, the functionality of the control system 112 can be improved. Figure 12C As shown, the central line segmentation model 1110 includes several branch points, some of which are highlighted for visibility. Branch points A, B, C, D, and E are shown at each of the branch points. Branch point A may represent the point in model 1110 where the trachea divides into the left main bronchus and the right main bronchus. The right main bronchus may be identified in the central line segmentation model 1110 as being located between branch points A and B. Similarly, secondary bronchioles are identified by branch points B and C and are located between branch points B and E. Other tributary bronchioles may be defined between branch points C and D, each of which may be associated with a representation of the luminal diameter of the corresponding passage. In some embodiments, the central line model 1110 may include an average diameter value for each segmented tributary bronchus. The average diameter value may be a patient-specific value or a more general value from multiple patients. Additionally, the central line model 1110 includes terminal points, with exemplary terminal points F, G, and H at... Figure 12C The markers are used to indicate the limitations of the segmentation process. Terminal points F, G, and H represent constraints on the segmentation results. For example... Figure 12C As shown, the centerline model 1110 does not extend beyond the undivided portion 1108 to reach the target 1104.

[0097] As described above Figure 5A As discussed in operation 506 of method 500, operator O can provide input via input device 130 or another operator input device available to control system 112 to expand centerline model 1110, such as... Figure 12D As shown. Figure 12DAn extended model 1112 is shown. The extended model 1112 includes a centerline model 1110 and an input-based model 1114. In conjunction with the description of operation 506, operator O can provide navigation directions via input device 130 to perform virtual navigation within and beyond the centerline segmentation model 1110 in the image space, while visualizing at least some image data 1102, such as voxels with Hounsfield values ​​associated with the bronchial wall 706. Based on the received virtual movement, control system 112 can track and record the movement to identify unsegmented portions of the anatomical passage to extend the centerline segmentation model 1110 and thus create the input-based model 1114.

[0098] in addition, Figure 1 Some embodiments of the input device 130 may allow operator O to "draw" on a drawing board or on a display system 110, which may be a touchscreen, to define lines or paths. The drawn lines can be added to the centerline model 1110 to generate an expanded model 1112. For example, operator O can draw in each of multiple views of image data (such as...) Figures 6A-6C Lines are drawn on the cross-sectional plane shown. The control system 112 can process multiple lines to generate three-dimensional lines to be added to the centerline model 1110. In some processes, the operator O may be able to select one of the endpoints F, G, or H. The operator O can then select an internal perspective view that allows virtual navigation within the anatomical passage to generate an input-based model or model section starting from the selected endpoint. Alternatively or additionally, multiple cross-sectional planes of image data may be presented to the operator O, on which multiple two-dimensional lines are drawn, and these two-dimensional lines are then decoded into a single three-dimensional line to generate an input-based model 1114.

[0099] In other embodiments, operator O can select any point on centerline model 1110 to provide virtual navigation or for drawing to extend the centerline model. For example, operator O can select midpoint I on centerline model 1110. Operator O can then perform virtual navigation from point I and draw lines extending from point I to generate input-based lines 1116. Operator O can provide additional input indicating the end of the input-based lines 1116, the end of which is indicated as terminal point J. After the operator finishes the input-based lines 1116, the operator can request control system 112 to attempt segmentation of the image data. The segmentation process can begin at terminal point J and automatically segment the image data 1102 to generate centerline model portion 1118.

[0100] Combinations of these methods can be used to generate complete or partial models of the anatomical passage based on user interaction with image data 1102. In some embodiments, a combination of segmentation-based modeling techniques and non-segmentation-based modeling techniques (such as virtual navigation or centerline mapping) can be used to generate models, such as the augmented model 1112, which can then be registered to a medical device inserted into the patient's anatomy during a medical procedure. By attaching the augmented model 1112 to the medical device (such as...) Figure 2A and Figure 2B The elongated device 202 is registered, and the image data associated with the model 1112 and the elongated device 202 can be brought into a common reference frame for image-guided medical procedures. Subsequently, a graphical representation of the registered elongated device 202 and the registered image data can be presented in the display system 110.

[0101] In other embodiments, segmentation model 1106 can be used to generate centerline segmentation model 1110 or another suitable model including point clouds, point sets, or point collections. When segmentation model 1106 includes a mesh representing an inner surface of one or more channels, a subset of vertices of the mesh represented in a stored data file including model 1106 can be used. Alternatively, the geometric center of the mesh or surface can be used to represent a volume or channel in segmentation model 1106.

[0102] In some embodiments, the centerline segmentation model 1110 is represented in the data as a point cloud, point set, 3D model of points, or point collection in three-dimensional space, rather than a continuous line. After the centerline segmentation model 1110 is generated and stored in the data as a point set, it can be retrieved from the data store for use in image-guided surgical procedures. To use the centerline segmentation model 1110 in an image-guided surgical procedure, the model 1110 can be registered to associate the modeled channels in the model 1110 with the actual anatomical structures of the patient present in the surgical environment. Using the model 1110 in point set registration includes using a point set obtained from the model 1110 or a point set that constitutes the model 1110. Furthermore, an extended model 1112, including portions obtained from virtual navigation or mapping, can be represented as a point set used in the point set registration algorithm.

[0103] Now for reference Figure 13 The illustration shows an exemplary user interface 1200 rendered on a display 1202. The display 1202 may be... Figure 1 It is part of the display system 110, or may be Figure 1The display system 110. The user interface 1200 includes a main view 1204 in which image data can be rendered. As shown in FIG12, the main view 1204 includes an internal perspective view of an anatomical passage from, for example, the viewpoint of the device, which is included in preoperative or intraoperative three-dimensional image data. The user interface 1200 also includes an auxiliary view 1206, which may include one or more displays of image data. Figure 13 As shown, the auxiliary view includes three two-dimensional slices of CT image data. In some embodiments, the main view 1204 and the auxiliary view 1206 can be displayed on separate display screens.

[0104] like Figure 13 As shown, the user interface 1200 is in virtual navigation mode, and therefore includes an interior perspective view in the main view 1204 and three two-dimensional slices in the auxiliary view 1206. On the right, the user interface 1200 includes user interface options 1208, which includes virtual navigation options 1210, drawing options 1220, and filter selections 1230.

[0105] Virtual navigation option 1210 includes user interface elements 1212A, 1212B, and 1212C. By selecting the start navigation element 1212A, operator O can begin virtual navigation, causing navigation input for assembly to be recorded as a path for generating a model (such as input-based model 1114). By selecting the pause navigation element 1212B, operator O can temporarily stop recording virtual navigation, and operator O can move within the 3D image space without recording navigation input. By selecting the resume navigation element 1212C, operator O can resume recording navigation input for assembly (such as model 1114). In some embodiments, the perspective (i.e., orientation and orientation within the 3D image data) can be selectively returned to the last recorded perspective, or the perspective can be retained anywhere after unrecorded navigation.

[0106] When operator O intends to draw lines or paths to generate or add to a line-based model of the anatomical passage, drawing mode option 1220 can be used. In some embodiments, as the user selects elements from drawing mode option 1220, after operator O draws on each view, the main view 1204 can be sequentially filled with a first two-dimensional view of image data, followed by a second two-dimensional view, and then a third two-dimensional view. When display 1202 is a touchscreen display, operator O can draw on display 1202. In other embodiments, operator O can use a mouse, keyboard, or other input mechanism to draw lines on multiple perspective views of image data, and then combine these lines to generate three-dimensional lines or curves that can be included in the line-based anatomical passage model.

[0107] Additionally, user interface option 1208 includes filter option 1230. The user interface elements included in filter option 1230 can be workflow-based. For example, if operator O communicates with control system 112 using input device 130 and the procedure to be performed is a lung biopsy, control system 112 can pre-populate filter option 1230 with user interface elements suitable for a lung biopsy. For example, as shown, filter option 1230 includes bronchial wall element 1232A, adjacent blood vessel element 1232B, and bone element 1232C. Operator O can interact with elements 1232A-1232C to turn the display of CT data with Hounsfield values ​​on and off within an associated range. In some embodiments, elements 1232A-1232C may each have consecutive input elements that provide different levels of transparency / opacity. Additionally, filter option 1230 may include other elements when the procedure being performed does not involve the lungs.

[0108] Many different variations of the user interface 1200 are included within the scope of this disclosure. For example, other user interface options and controls may be included, such as options for segmenting image data or options for continuing to segment image data, as described above. Figure 12A-12D The subject of discussion.

[0109] Now for reference Figure 14 The diagram illustrates a flowchart of method 1300 for providing image-based guidance during minimally invasive medical procedures. Like other methods described herein, method 1300 is depicted as a series of operations. Embodiments of method 1300 may include additional or alternative operations and / or one or more enumerated operations may be omitted. Some embodiments of method 1300 may include instructions readable by control system 112 or processing device 114 to cause control system 112 to perform the operations of method 1300.

[0110] Some embodiments of method 1300 may begin at operation 1302, wherein a processing device (such as processing device 114 of control system 112) receives three-dimensional image data of at least a portion of the patient's anatomy. For example, control system 112 may receive CT image data showing the torso of patient P (including the lungs of patient P). In other embodiments, the image data may be obtained from another portion of the patient's anatomy.

[0111] At operation 1304, the processing device can register the three-dimensional image data with the surgical environment. In some embodiments, the image data can be registered with the surgical environment by registering reference markers included in the image data to corresponding reference markers in the surgical environment. At operation 1306, the processing device registers a medical device coupled to a remotely operated medical system with the surgical environment, for example by the following steps: driving the medical device within a portion of the patient's anatomy and capturing measurement points associated with the tracking orientation of the medical device. In some embodiments, operations 1304 and 1306 can be combined into a single registration operation that directly registers the three-dimensional image data with the medical device. The registration process included in method 1300 may include operations such as generating a model from the received image data, or generating a portion of such a model based on a tracking virtual movement command of an operator O within a space defined by the image data, as combined above. Figure 5A Method 500 is described. After operations 1304 and 1306, the control system 112 associates the information obtained using the medical device with the three-dimensional image data. For example, the control system 112 may cause a real-time view obtained by a camera at the distal end of the medical device to be displayed alongside a portion of the image data viewed from the distal end of the medical device. For example, the main view 1204 of the user interface 1200 may include a split view and a perspective view with real-time feeds from the camera.

[0112] In operation 1308, the processing device can apply filters to three-dimensional image data to alter the rendering of one or more voxels in the three-dimensional image data. For example, when displaying image data from a distal viewpoint of a medical device, portions of the image data can be filtered to provide the desired presentation of the image data. For example, the image data can be filtered based on Hounsfield values ​​associated with voxels. Operator O can selectively make certain tissues fully or partially transparent, or control system 112 can automatically make certain tissues fully or partially transparent to make certain aspects easier to visualize. For example, voxels with Hounsfield values ​​associated with air can be rendered transparently, while tissues defining bronchial passages can be rendered semi-transparently, and blood vessels surrounding bronchial passages can be rendered opaquely. As another example, targets contained within the image data can be rendered as opaque blocks, while all tissues can be rendered semi-transparently. Other combinations can be used to allow operator O to visualize information deemed most useful in any given situation. In addition to the transparency of certain tissues, filters can also provide different shading.

[0113] In operation 1310, the processing device can render three-dimensional image data on a display from a perspective associated with the medical device. As described above, this can be done in a first portion of the display or a first display screen, while real-time video from the same or different perspectives is provided in a second portion of the display or a second display screen. In this way, if the real-time video from the medical device is blocked, the operator O can use CT data to continue navigation. Additionally, the operator O can compare tissues, fluids, etc., shown in the real-time view with the same objects included in the image data. In other embodiments, other imaging modalities (intraoperative or preoperative) can be compared with the CT image data. The operator O can make selections in the user interface (user interface 1200 as shown in Figure 12) to provide selection of tissue types and / or receive rendering settings associated with each tissue type to the control system 112. For example, when the operator O selects bronchial wall element 1232A, the user interface 1200 can present additional user interface options, such as sliders or input areas, whereby the operator O can input transparency and / or color settings associated with bronchial wall element 1232A. The settings can then be applied to voxels associated with the Hounsfield values ​​of the bronchial walls.

[0114] In some embodiments of method 1300, when the control system 112 detects a change in the orientation of the medical device, the viewing angle of the observed image data can be updated to reflect the change in the orientation of the medical device, such that navigation commands sent to turn / position the medical device are simultaneously used for virtual navigation of the image data.

[0115] Embodiments of this disclosure can provide significant improvements to image-guided medical procedures. For example, some embodiments may allow the use of lower-resolution CT scans to generate models of patient anatomy. By improving the utility of such lower-resolution CT scans, patient exposure to radioactive imaging agents used to acquire image data can be reduced. Some embodiments may allow the use of both segmentation and user input to generate hybrid models of patient anatomy, which can improve the efficiency and accuracy of generating such anatomical models that allow for registration. Furthermore, embodiments of this disclosure can facilitate the simultaneous viewing of image data and real-time video data during medical procedures.

[0116] Embodiments of the methods and systems described herein include computer-readable storage media, such as CD-ROMs, DVDs, flash memory, and other storage media, on which machine-readable instructions are stored. Processing apparatuses, such as those comprising... Figure 1 The processing device in the control system 112. Further embodiments of this disclosure include a system (such as a workstation) having one or more processing devices (such as a CPU and a GPU) and a memory storing instructions that, when read by the processing device, cause the system to perform one of the embodiments of methods 400, 500, and 1300. Additionally, features described with respect to various embodiments of this disclosure may be combined. For example, the operations discussed with respect to method 1300 may be included in an embodiment of method 500. Furthermore, variations and modifications that are obvious to those skilled in the art may be made to embodiments of this disclosure without departing from the scope of this disclosure. While some more specific embodiments have been described with respect to the lungs of a patient, other anatomical pathways, structures, and organs can be imaged, registered, observed, and treated using embodiments of this disclosure. These organs may include the heart, digestive tract, kidneys, and other organs. Therefore, the scope and spirit of this disclosure can be fully understood by referring to the appended claims.

Claims

1. A non-transitory computer-readable medium comprising a plurality of machine-readable instructions, which, when executed by one or more processors, are adapted to cause the one or more processors to perform a method for facilitating an image-guided medical procedure, the method comprising: Receive three-dimensional image data of at least a portion of the patient's anatomy through a remotely operated medical system having at least one processing device; The medical device coupled to the remote-operated medical system is registered to the surgical environment, and the three-dimensional image data is registered with the surgical environment; A radiation density filter is applied to the 3D image data to alter the rendering of one or more voxels in the 3D image data; and The three-dimensional image data is rendered on a display from a perspective associated with the medical device, wherein the rendered three-dimensional image data is not segmented.

2. The non-transitory computer-readable medium of claim 1, wherein the radiation density filter is applied to the three-dimensional image data such that voxels having a first radiation density value are rendered transparently.

3. The non-transitory computer-readable medium of claim 1, wherein the instructions further cause the one or more processors to perform: assigning voxels in the three-dimensional image data to one of a plurality of tissue types.

4. The non-transitory computer-readable medium of claim 3, wherein the instructions further cause the one or more processors to perform: The selection of an organization type from the various organization types is received via an organization type selection element included in the user interface; Receive the transparency setting associated with the organization type selection element; and Apply the transparency setting to the voxels assigned to the selected tissue type.

5. A system for processing medical images, the system comprising: Memory; and A processing device, which communicates with the memory, is configured to execute instructions to perform operations, the operations including: Receive three-dimensional image data of at least a portion of the patient's anatomy through a remotely operated medical system having at least one processing device; The medical device coupled to the remote-operated medical system is registered to the surgical environment, and the three-dimensional image data is registered with the surgical environment; A radiation density filter is applied to the 3D image data to alter the rendering of one or more voxels in the 3D image data; and The three-dimensional image data is rendered on a display from a perspective associated with the medical device, wherein the rendered three-dimensional image data is not segmented.

6. The system of claim 5, wherein the radiation density filter is applied to the three-dimensional image data such that voxels having a first radiation density value are rendered transparently.

7. The system of claim 5, wherein the processing means is further configured to execute instructions to perform: assigning voxels in the three-dimensional image data to one of a plurality of tissue types.

8. The system of claim 7, wherein the processing means is further configured to execute instructions to perform: The selection of an organization type from the various organization types is received via an organization type selection element included in the user interface; Receive the transparency setting associated with the organization type selection element; and Apply the transparency setting to the voxels assigned to the selected tissue type.

9. The system of claim 5, wherein the processing means is further configured to execute instructions to perform: Tracking movement within the space defined by the three-dimensional image data; and A first model of the patient's anatomy is generated based on tracked movement, wherein registration of the three-dimensional image data with the surgical environment is performed using the first model.

Images