Systems and methods for intra-operative updating of graphical user interfaces based on intra-operative imaging data

Abstract

A system includes a display system, an elongate device, a medical instrument, and a control system. The control system is configured to: display intra-operative imaging data of a patient anatomy received from an external imaging system in a graphical user interface (GUI) on the display system; displaying a graphical indicator superimposed on the intraoperative imaging data; registering the intraoperative imaging data with the elongation device; transforming the graphical indicator from the intra-operative imaging data frame of reference to the elongated device frame of reference based on the registration; and providing navigational guidance to guide the elongate device to a target location in the patient anatomy based on the target location of the distal tip of the medical instrument displayed via the graphical indicator. The target location of the distal tip of the medical device is within the deployment range of the device when the elongate device is located at the target location.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority and benefit to U.S. Provisional Application No. 63 / 500,702, filed May 8, 2023, entitled “Systems and Methods for Intraoperatively Updating a Graphical User Interface Based on Intraoperative Imaging Data,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The examples described herein relate to systems and methods for planning and performing image-guided procedures, such as systems and methods for planning treatment areas and confirming treatment areas intraoperatively. Background Technology

[0004] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and adverse side effects. Such techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, the operator can insert minimally invasive medical instruments to reach the target tissue. Minimally invasive medical instruments include instruments such as therapeutic, diagnostic, biopsy, and surgical instruments. These instruments can be inserted into anatomical channels and navigate toward regions of interest within the patient's anatomy. Preoperative and / or intraoperative images of the anatomical channels can be used to aid navigation. Improved systems and methods are needed to enhance the information provided to the user via a graphical user interface (GUI) based on information received from intraoperative imaging. Summary of the Invention

[0005] Various features can improve navigation guidance, which is provided for guiding the elongation device to target location within the patient's anatomy. The following presents a simplified overview of the various examples described herein and is not intended to identify key or important elements or to define the scope of the claims.

[0006] Consistent with some examples, a system is provided. The system includes: a display system; an elongation device; a medical device configured to extend within the elongation device; and a control system communicatively coupled to the display system. The control system is configured to display intraoperative imaging data of a patient's anatomy received from an external imaging system. The intraoperative imaging data is displayed in a graphical user interface (GUI) on the display system. The control system is also configured to display a graphical indicator superimposed on the intraoperative imaging data. The graphical indicator indicates a target position of the distal tip of the medical device within the patient's anatomy. The control system is further configured to register the intraoperative imaging data with the elongation device. The control system is also configured to transform the graphical indicator from a reference frame of the intraoperative imaging data to a reference frame of the elongation device based on the registration. The control system is further configured to provide navigation guidance based on the target position of the distal tip of the medical device displayed via the graphical indicator, to guide the elongation device to a target location within the patient's anatomy. When the elongation device is at the target location, the target position of the distal tip of the medical device is within the deployment range of the medical device.

[0007] Consistent with other examples, a system is provided. The system includes: a display system; an elongation device; a medical device configured to extend within the elongation device; and a control system. The control system is configured to display intraoperative imaging data of a patient's anatomy received from an external imaging system. The intraoperative imaging data is displayed in a graphical user interface (GUI) on the display system. The control system is also configured to display treatment information superimposed on the intraoperative imaging data. The treatment information indicates one or more treatment parameters for treating anatomical targets in the patient's anatomy. The control system is further configured to register the intraoperative imaging data with the elongation device. The control system is also configured to transform the treatment information from a reference frame of the intraoperative imaging data to a reference frame of the elongation device based on the registration. The control system is further configured to provide navigation guidance based on the treatment information to guide the elongation device to target locations in the patient's anatomy.

[0008] Consistent with other examples, a method is provided. The method includes displaying intraoperative imaging data of a patient's anatomy received from an external imaging system. The intraoperative imaging data is displayed in a graphical user interface (GUI) on the display system. The method also includes displaying a graphical indicator superimposed on the intraoperative imaging data. The graphical indicator indicates a target location of the distal tip of a medical device within the patient's anatomy. The medical device is configured to extend within an elongation device. The method also includes registering the intraoperative imaging data with the elongation device. The method further includes transforming the graphical indicator from a reference frame of the intraoperative imaging data to a reference frame of the elongation device based on the registration. The method also includes providing navigation guidance based on the target location of the distal tip of the medical device displayed via the graphical indicator to guide the elongation device to the target location within the patient's anatomy. When the elongation device is at the target location, the target location of the distal tip of the medical device is within the deployment range of the medical device.

[0009] Other examples include corresponding computer systems, devices, and computer programs, each configured to perform actions of any one or more of the methods described below, stored on one or more computer storage devices.

[0010] It should be understood that both the foregoing general description and the following detailed description are illustrative and exemplary in nature, and are intended to provide an understanding of the various examples described herein without limiting the scope of the examples. In this regard, additional aspects, features, and advantages of the various examples described herein will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0011] Figure 1 A display system showing images of an elongation device registered with an anatomical model, based on some examples.

[0012] Figure 2 The following examples illustrate methods for updating a graphical user interface based on intraoperative image data.

[0013] Figure 3 A simplified diagram of a graphical user interface for displaying image data from an intraoperative imaging procedure is shown, based on some examples.

[0014] Figure 4 The methods for registering image data with shape data from an instrument to update the localization of a target in a model are illustrated, based on some examples.

[0015] Figures 5A to 5E The following examples are shown. Figure 2 Information displayed on the graphical user interface during various processing stages.

[0016] Figures 6A to 6BThe following are examples of the information displayed on the graphical user interface during the navigation of the elongation device to the anatomical target.

[0017] Figure 7A The information displayed on the graphical user interface is shown based on some examples of intraoperative imaging data.

[0018] Figure 7B The diagram shows graphical information overlaid on intraoperative image data displayed in a graphical user interface, based on some examples.

[0019] Figures 7C to 7D The illustration shows graphical information overlaid on intraoperative image data displayed in the graphical user interface during the positioning of updated graphical indicators, based on some examples.

[0020] Figure 7E The following examples are shown based on intraoperative image data and Figure 7B The graphic information superimposed on the intraoperative image data is displayed in the graphical user interface as navigation guidance information.

[0021] Figures 7F to 7I The diagram shows graphical information overlaid on intraoperative image data displayed in a graphical user interface, based on some examples.

[0022] Figure 8A A graphical user interface for displaying intraoperative image data of elongation devices and medical instruments, based on some examples, is shown.

[0023] Figures 8B to 8C The diagram shows a graphical user interface for adjusting the position of a medical device, based on some examples.

[0024] Figure 9 Simplified diagrams of robots or remotely operated medical systems are shown, based on some examples.

[0025] Figure 10 Simplified diagrams of medical device systems and intraoperative imaging systems are shown, based on some examples.

[0026] The various examples described herein and their advantages are described in the following detailed description. It should be understood that, for illustrative purposes, but not limiting, the various examples described herein, similar reference numerals are used to identify similar elements shown in one or more of the accompanying drawings. Detailed Implementation

[0027] The techniques disclosed in this document can be used to provide and update information to a user via a graphical user interface during minimally invasive procedures using intraoperative imaging, such as cone-beam computed tomography (CT) imaging. In some examples, a model of anatomy can be built based on a preoperative imaging process and displayed on the graphical user interface (GUI). This model can be modified based on the intraoperative imaging process performed during the minimally invasive procedure. In some examples, image data generated by intraoperative imaging can be used to modify the positioning of instruments, anatomical structures, or targets in the model built from the preoperative imaging process. In some examples, treatment information related to the minimally invasive procedure can be overlaid on intraoperative imaging data captured by the intraoperative imaging system and displayed on the GUI.

[0028] Reference Figure 1 This system enables image-guided medical procedures, which can be performed manually, with robot assistance, or otherwise remotely. During these procedures, the display system 100 can display a virtual navigation image 102, which may be referred to as a virtual navigation view. The virtual navigation image 102 includes an image reference frame (X). I Y I Z I 150. The elongation device 104, such as a medical device (e.g., catheter, tool, probe, imaging device, etc.), can be registered (e.g., a dynamic reference) with a patient's anatomical model 106 derived from, for example, preoperative image data obtained from a CT scan. The elongation device 104 can be a flexible elongation device. In some examples, the elongation device 104 can be a catheter, tool, probe, imaging device, etc. In some examples, the elongation device 104 can include one or more catheters, tools, probes, imaging devices, etc. The anatomical model 106 can include a target 108, such as a lesion or nodule of interest, and a margin 110 surrounds the target 108. In some examples, the virtual navigation image 102 can also, or alternatively, present the physician with a virtual image of the internal surgical site from the perspective of the elongation device 104, for example, from the distal tip of the elongation device 104. In some examples, such as when the elongation device 104 includes an endoscope, the display system 100 can also, or alternatively, present a real-time view from the distal tip of the elongation device 104. In some examples, the elongation device 104 can be operated by a robot-assisted manipulator, which is controlled by a control system (e.g., Figure 9 The control system 912 in the system or a processing system including one or more processors will be controlled. Figure 9 Examples of robot-assisted medical systems are further described below. In some examples, the ablation probe can extend through the lumen of the extension device 104. In some examples, the ablation probe is the extension device 104.

[0029] Generating a virtual navigation image 102 involves using an image reference frame (X). I Y I Z I )150 and the surgical reference system of anatomical structure (X S Y S Z S ) and / or the elongation device reference system (X) of the elongation device 104 M Y M Z M Registration. This registration can be achieved by rotating, translating, or otherwise manipulating points associated with the segmented instrument shape from image data and / or points associated with shape data from shape sensors positioned along the length of the elongation device 104, using rigid or non-rigid transformations. Such registration between the image reference frame and the elongation device reference frame can be achieved, for example, by using point-based iterative nearest point (ICP) techniques as described in U.S. Provisional Patent Application No. 62 / 205,440, filed August 14, 2015, entitled “Systems and Methods of Registration for Image-Guided Surgery” and U.S. Provisional Patent Application No. 62 / 205,433, filed August 14, 2015, entitled “Systems and Methods of Registration for Image-Guided Surgery”, which are incorporated herein by reference in their entirety. Registration can also be achieved, additionally or alternatively, by another point cloud registration technique.

[0030] Figure 2 Examples of a method 200 for updating a graphical user interface (GUI) to provide navigation guidance during a minimally invasive procedure, according to some aspects of this disclosure, are shown. Method 200 is shown as a set of operations or processes 202 to 234. Processes 202 to 234 can be used in conjunction with... Figure 2 The processes shown may be executed in the same or different order. In some examples of method 200, one or more of the processes shown may be omitted. Additionally, one or more processes not explicitly shown in the flowchart may be included before, after, between, or as part of the processes shown. In some examples, one or more processes in the flowchart may be implemented at least in part by the control system executing code stored on a non-transitory tangible machine-readable medium, which, when run by one or more processors (e.g., the processor of the control system), may cause one or more processors to execute one or more processes.

[0031] At point 202, in the control system (e.g., Figure 9 Preoperative image data is received at the control system 912. For example, a CT scan of the patient's anatomy can be performed using a conventional fan-beam CT scanner, and the CT image data can be received by the control system of the robot-assisted medical system. Alternatively, the preoperative image data can be received from other types of imaging systems, including magnetic resonance imaging systems, fluorescence fluoroscopy systems, or any other suitable method for obtaining the dimension of the anatomical structure. At processing 204, a three-dimensional (3D) model of the anatomical structure can be constructed based on the preoperative image data received by the control system (e.g., Figure 1 (Anatomy model 106). At processing point 206, the target can be identified in the 3D model or in the preoperative image data used to construct the 3D model. For example, Figure 1 Target 108 can be identified as a region of interest for study or treatment in the anatomical model 106. In some examples, the target can be automatically identified by the control system. Target identification can be confirmed by a user. For example, the control system can receive user input confirming target identification. Additionally or alternatively, the target can be visually identified by the user and manually selected or indicated in the 3D model, for example, via display system 100 and / or via a GUI. For example, the control system can receive user input selecting and / or indicating a target via a GUI. At processing 208, a route is generated through an anatomical passage formed in the anatomical structure. The route can be automatically generated by the control system, or the control system can generate the route based on received user input. The route can indicate a medical device (e.g., Figure 1 The extension device 104) can navigate along its path to a target (e.g., Figure 1 The path is very close to the target 108. When the extension device 104 is within a threshold distance from the target 108, the extension device 104 can be very close to the target 108. In some examples, the route can be stored in the control system and incorporated into the image displayed on the display system 100. For example, the route can be displayed in the virtual navigation image 102. The route can be overlaid on the anatomical model 106 in the virtual navigation image 102.

[0032] To provide accurate navigation through the anatomical passage, reference frame 150 of the preoperative image data (and subsequently constructed 3D model) can be registered with reference frame 150 of the elongation device 104 at processing 210. For example, shape sensors (e.g., fiber optic shape sensors or one or more position sensors) positioned along the length of the elongation device 104 can be used to provide real-time shape data (e.g., information about the shape of the elongation device 104 and / or the position of one or more points along the length of the elongation device 104). This shape data can be used to register the elongation device 104 with the 3D model constructed from the preoperative image data and to track the positioning of the elongation device 104 during use. After successful registration, processing 212 can include providing navigation guidance as the elongation device 104 is navigated through the anatomical passage to its deployment location near the target. Navigation can be performed manually by the user using the provided navigation guidance, automatically by the control system, or via a combination of both.

[0033] Intraoperative imaging can be performed when the elongation device 104 is positioned at or near the deployment location within the patient's anatomy (e.g., very close to the target 108). The elongation device 104 can be very close to the target 108 when it is within a threshold distance from the target 108. At processing 214, intraoperative image data can be received from the intraoperative imaging system at the control system. In some examples, the intraoperative imaging system may be a cone-beam CT (“CBCT”) scanner that generates intraoperative CT scan image data; however, any suitable imaging technique may be used without departing from the examples of this disclosure. Compared to other imaging techniques such as conventional CT or fluoroscopy, CBCT imaging can provide faster scanning of areas of the patient's anatomy to reduce procedural delays and can also have more portable and compact hardware.

[0034] Intraoperative image data can be received at the control system or other processing platform associated with the elongation device. Communication of intraoperative image data can originate from the application programming interface of the intraoperative imaging system. As an example, the Cios Spin® imaging system sold by Siemens® MedicalSolutions USA, Inc. utilizes a protocol called NaviLink 3D™, which provides a digital interface for connecting the imaging system to the navigation system and transmitting datasets to it. In some examples, shape data associated with the elongation device can be transmitted to the intraoperative imaging system, or both shape data and intraoperative image data can be transmitted to a common platform for processing. In this regard, registration of the shape data of the elongation device with the intraoperative image data can be performed by the control system, the intraoperative imaging system, or another platform operatively communicating with the intraoperative imaging system and / or the control system. Typically, the transmission of intraoperative image data to or from the control system will use the Medical Digital Imaging and Communications (“DICOM”) standard. Additionally or alternatively, intraoperative image data can also be received in maximum intensity projection (“MIP”) or pseudo-CT stream format. In some examples, receiving intraoperative imaging data may include receiving one or more timestamps associated with the intraoperative imaging data. A first timestamp may indicate the start time of the intraoperative imaging scan, and a second timestamp may indicate the stop time of the intraoperative imaging scan. In some examples, timestamps may be associated with each instance of the intraoperative image data. To ensure accurate correlation, the clock of the control system associated with the elongation device may be synchronized with the clock associated with the intraoperative imaging system. Each instance of shape data may be associated with a timestamp. In some examples, each timestamped instance of intraoperative image data may be paired with a corresponding timestamped instance of shape data.

[0035] To register intraoperative imaging data with the elongation device, during the acquisition of intraoperative imaging data, at processing point 216, the control system can, for example, receive shape data from the elongation device during the intraoperative imaging process. Shape data may be captured only for a short period of time, or it may be captured throughout the entire image capture period of the intraoperative imaging scan. Shape data may also be captured at any other time period during the image capture period of the intraoperative imaging scan. Various synchronization techniques can be used to ensure that only shape data corresponding to the image capture period is used for registration, even though shape data outside the image capture period can also be received.

[0036] At point 218, intraoperative image data or portions thereof can be segmented. For example, discrete units (e.g., pixels and / or voxels) of the intraoperative image data can be analyzed to assign an intensity value to each unit. Discrete units with the same or similar intensity values ​​can be aggregated to form parts. Morphological operations can be used to interconnect discontinuous parts with similar intensity values. In some examples, computer software, alone or in combination with manual input, is used to convert intraoperative image data into a two-dimensional or three-dimensional composite representation or model of the segmentation of parts or entire anatomical organs or regions. The model can describe the various locations and shapes of anatomical passages and their connectivity. More specifically, during segmentation processing, pixels and / or voxels can be divided into segments or elements, or labeled to indicate that they share certain characteristics or computed properties, such as color, density, intensity, and texture. In some examples, segmenting intraoperative image data may include selecting parts to associate with certain objects. For example, segmenting intraoperative image data associated with an elongation device may include selecting an imaging unit or component based on one or more factors, including proximity to the target, shape data, approximate registration of the elongation device with the patient, and the expected elongation device strength value. The expected elongation device strength value may include a series of values ​​associated with the materials constituting the elongation device. In some examples, algorithms (e.g., Gaussian mixture models) may be used to establish the expected elongation device strength. In some examples, segmenting intraoperative image data may also include utilizing processing established by a control system using deep learning techniques. Intraoperative image data corresponding to an elongation device can be segmented or filtered from the intraoperative image data. A model of the shape of the elongation device can be generated based on the segmented intraoperative image data corresponding to the elongation device. For example, through segmentation, the elongation device can be identified as an elongation device in the intraoperative image data. Additionally or alternatively, the elongation device can be identified as an elongation device in the intraoperative image data by filtering the intraoperative image data by means of the CT number or Hounsfield value associated with the elongation device. This data associated with the elongation device can be isolated from other portions of the intraoperative image data associated with the patient or a specific tissue type. A three-dimensional mesh model can be formed around the isolated data and / or a centerline representing the centerline of the elongation device can be determined. The segmented intraoperative image data of the elongation device can be represented in an intraoperative image reference frame.

[0037] In some examples, information about the elongation device can be used to provide initial seeding for segmentation. For instance, the elongation device (e.g., a steerable maneuvering catheter) may include a metallic spine embedded in a non-metallic sheath. High-contrast information associated with the spine can be identified first in the intraoperative imaging data, and the non-metallic sheath can be searched for in pixels or voxels with lower contrast in the region surrounding the spine. In some examples, a high-contrast fiducial marker can be inserted into the working channel through the elongation device during intraoperative imaging to improve segmentation.

[0038] At point 220, the target can be identified in the intraoperative image data (e.g., Figure 1 Target 108). In some examples, when the elongation device has been segmented or identified, target identification may include establishing a region of interest (e.g., a sub-region) within a predetermined range of the elongation device in the intraoperative image data. The region of interest, rather than all intraoperative image data, may be analyzed to segment the target from the region of interest. In some examples, the search domain for locating the target may be reduced based on the assumption that the elongation device was previously navigated to a position very close to the target. In some examples, target identification may include receiving instructions or selections from the user at a user interface, such as a GUI. For example, the control system may receive user input that manually selects portions of intraoperative image data associated with one or more components on the display system to associate with the target. Manual target identification may be required when automatic target identification produces unsatisfactory results. In some examples, the control system may instruct the user to manually identify the target in the intraoperative image data. In some examples, target identification in the intraoperative image data may enable the registration of preoperative image data (or 3D model 106) with the intraoperative image data based on preoperative and intraoperative target localization.

[0039] At processing point 222, intraoperative image data can be registered with the shape data of the elongation device by comparing the shape data received from the elongation device with the corresponding portion of the intraoperative image data. The shape data from the elongation device can be represented in elongation device reference frame 350 and / or surgical reference frame 250. This registration can be performed by rigid or non-rigid transformations to rotate, translate, or otherwise manipulate points associated with the segmented shape of the elongation device and points associated with the shape data of the elongation device. In some examples, this registration can be performed using an iterative nearest-point algorithm or another point cloud registration technique. Optionally, data points can be weighted based on segmentation confidence or quality to have a greater influence on the control system's determination of more likely accurate data point assignments. Additionally or alternatively, registration of intraoperative image data with the shape data of the elongation device can be performed using coherent point drift or an uncertainty metric (e.g., RMS error). In some examples, the segmented shape of the elongation device is registered with shape data, and associated transformations (e.g., vectors applied to each point in the segmented shape to align with the shape data in the shape sensor reference frame) can be applied to the entire 3D image and / or a 3D image subsequently acquired during a medical procedure. The transformation can be a six-degree-of-freedom (6DOF) transformation, allowing the shape data to be translated or rotated in any or all of the X, Y, and Z axes, as well as pitch, roll, and yaw. Discussions of the processing for registering the elongation device with image data can be found, for example, in International Patent Application Publication No. WO 2021 / 092116, filed November 5, 2020, entitled “Systems and Methods for Registering an Instrument to an Image Using Change in InstrumentPosition Data,” and in International Patent Application Publication No. WO 2021 / 092124, filed November 5, 2020, entitled “Systems and Methods for Registering an Instrument to an Image Using PointCloud Data,” both of which are incorporated herein by reference in their entirety.

[0040] When the intraoperative image reference system is registered with the elongation device reference system, the images displayed on the display system 100 (e.g., virtual navigation image 102) enable the operator to: more accurately navigate the elongation device through the patient's anatomy, visualize the target lesion relative to the elongation device, view the perspective from the distal end of the elongation device, and / or improve the efficiency and effectiveness of the target medical procedure.

[0041] In some examples, intraoperative image data can be registered with preoperative image data obtained from the same or different imaging systems. Therefore, by registering shape data with intraoperative image data, the registration of shape data with preoperative image data can also be determined. In some examples, anatomical images generated from intraoperative and / or preoperative image data can be displayed alongside images of the elongation device derived from shape data received from the self-elongating device. For example, a model of the elongation device generated from instrument shape data can be overlaid on an image of the patient's anatomy generated from preoperative or intraoperative image data.

[0042] At process 224, the intraoperative location of the target can be mapped to the elongation device reference frame based on the registration performed in process 222. In some examples, the intraoperative location of the target can be compared with the preoperative location of the target. Additionally or alternatively, the preoperative geometry of the target (e.g., surface boundaries) can be compared with the intraoperative geometry of the target. If a difference exists, at process 226, the target location or geometry can be updated within model 106 such that the preoperative location and / or geometry of the target is modified to match the intraoperative location and / or geometry of the target. The updated location of the target can be displayed in the preoperative image reference frame such that the updated-located target is displayed on display system 100 in 3D model 106.

[0043] At process 228, the GUI can display graphical indicators superimposed on the intraoperative image data. In some examples, the graphical indicators show the target deployment location of the distal tip of a medical device deployed via an extension device. At process 230, a treatment area can be superimposed on the intraoperative image data in the GUI. The treatment area can indicate areas of tissue and other anatomical structures that may be affected by treatment. For example, the treatment area can indicate an ablation zone, which indicates areas of tissue and other anatomical structures that may be affected by an ablation procedure. Additionally or alternatively, the treatment area can indicate a radiation zone, which indicates areas of tissue and other anatomical structures that may be affected by a radiation procedure. Additionally or alternatively, the treatment area can indicate a reference zone, which indicates areas where one or more reference points can be placed in, on, or around tissue and other anatomical structures. The treatment area can indicate areas of tissue and other anatomical structures that may be affected by any other treatment or procedure. The treatment area can indicate a focal area within or around an anatomical structure, such as the target anatomical structure in a biopsy procedure. In some examples, more than one treatment area can be superimposed on the intraoperative image data in the GUI.

[0044] At step 232, when the medical device is deployed (e.g., extended from the distal tip of the elongation device), intraoperative image data can be received from the intraoperative imaging system at the control system. In some examples, intraoperative imaging data can be received from an external intraoperative imaging system, such as a CBCT system, but any suitable imaging technique can be used without departing from the examples of this disclosure. At step 234, the control system can receive confirmation of the elongation device retraction. Reference will be made below. Figures 7A to 7E Describe further details regarding the processing of 228 to 234.

[0045] In some examples, segmentation techniques similar to those discussed above can be used to segment intraoperative image data corresponding to one or more anatomical channels. Segmentation of anatomical channels may result in modifications to the surface boundaries, diameters, locations, etc., of one or more anatomical channels. After registering the intraoperative image data with shape data, the modified geometry and / or location of at least one of the anatomical channels can be updated from the preoperative configuration to the intraoperative configuration in a 3D model displayed on a graphical user interface based on the intraoperative image data. The updated configuration of one or more anatomical channels may result in updated navigation paths for the elongation device. For example, intraoperative imaging may indicate that the diameter of an anatomical channel previously considered too narrow to safely navigate the elongation device may actually be larger than indicated by preoperative imaging. By updating the diameter of the anatomical channels in model 106, the control system can determine that the modified anatomical channel provides a more direct route to the target. The processing of updating anatomical structures is not limited to anatomical channels (e.g., airways). For example, intraoperative image data can be used to modify the anatomical boundaries of tissues and / or organs (e.g., pleura, lung fissures, vascular systems, etc.) in model 106.

[0046] The segmentation and updating of targets, anatomical pathways, and / or other anatomical structures can be performed automatically by the control system independently of user input. Alternatively or additionally, these processes can be initiated by received user input to identify one or more locations in intraoperative image data corresponding to relevant features.

[0047] In some examples, Figure 2 The processing of intraoperative image data received at 214 locations can be displayed as follows: Figure 3 The graphical user interface (GUI) 300 of the display system is shown. The segmented elongation device can be displayed on the display system along with intraoperative image data. Alternatively or additionally, the segmentation target can be displayed on the display system along with intraoperative image data. Figure 3Viewing mode 302, providing two-dimensional and / or three-dimensional views of intraoperative image data, is shown, wherein the elongation device 304 and the intraoperative target 306 (which may be identical to the preoperative target 108) have been segmented. Intraoperative image data is displayed in an intraoperative image reference frame (X). I2 Y I2 Z I2 In 300, pixels or voxels can be displayed according to assigned intensity values ​​that provide initial visual boundaries between different structures. Viewing mode 302 can display the centerline and / or boundaries of the segmented elongation device registered with 3D model 106. When segmentation is unsuccessful, GUI 300 can allow the user to manually identify the elongation device and / or target. In some examples, the elongation device 304 and / or target 306 can be displayed in different colors or can otherwise be visually distinguished from the surrounding anatomical structures.

[0048] As shown above (refer to the reference) Figure 2 As discussed in section 222, the intraoperative image reference system 450 can be registered with the elongation device reference system 350. (Refer to the above...) Figure 2 The processing discussed in section 226 can update the target localization from localization based on preoperative image data to localization based on intraoperative image data. Figure 4 A method 400 is illustrated for registering intraoperative image data with shape data from an elongation device (e.g., elongation device 104) to update the localization of a target (e.g., target 108) in a model (e.g., model 106). At process 402, shape data of the elongation device may be recorded during an image capture period of an imaging system. The imaging system may be a cone-beam CT system or any other imaging system configured to capture intraoperative images of instruments and patient anatomy. At process 404, image data corresponding to the image capture period may be received. The image data may include one or more of the patient anatomy, the target, and the elongation device. At process 406, the target is identified in the image data. For example, the target may be segmented by a control system or other processing platform. Additionally or alternatively, the control system may receive user input identifying the target. At process 408, a portion of the image data corresponding to the elongation device may be segmented. Using the segmented image data and shape data recorded in process 402, the image data may be registered with the shape data based on the shape of the elongation device during the image capture period at process 410. At point 412, using registered image data, the target localization can be updated from preoperative localization based on preoperative imaging to intraoperative localization based on intraoperative imaging. This updated target localization improves navigation of the extension device to the target.

[0049] In some examples, additional or alternative processing can be used to update the localization of anatomical channels in the preoperative image reference system 150. For example, one or more portions of the image data corresponding to the anatomical channels can be segmented. Using the segmented image data and shape data recorded in processing 402, the image data can be registered with the shape data based on the shape of the elongation device during the image capture period. Using the registered image data, the localization of one or more anatomical channels can be updated from preoperative localization based on preoperative imaging to intraoperative localization based on intraoperative imaging. Updating one or more anatomical channels can provide a more accurate path from the current localization of the elongation device to the target. It should be understood that method 400 is optional.

[0050] Figures 5A to 5E The diagram shows the method used to perform the above-mentioned... Figure 2 Examples of GUIs for various processing methods discussed. GUI500 may include multiple view windows for displaying visual information to the user. For example, Figure 5A The GUI 500 is shown, which includes view windows 502, 504, and 506, which respectively display cross-sections of intraoperative image data taken along the coronal, transverse, and sagittal planes. Although three view windows are shown, more or fewer view windows can be displayed at any given time.

[0051] exist Figure 5A In this configuration, the main view window 506 is larger than the two secondary view windows 502 and 504. The control system can receive user input selecting which visual information to display in the main view window 506. For example, it can receive user input selecting the maximize icon 508 on one of the secondary view windows 502 and 504. In some examples, the control system can receive user input to zoom in on the information shown in the main view window 506 to a full-screen view based on received user input selecting the full-screen button 510. A view indicator 514 can be overlaid on each of the view windows 502, 504, and 506 to indicate the plane or orientation corresponding to the currently displayed image.

[0052] Each view window allows the user to scroll through a series of intraoperative images taken along a parallel plane, for example, by hovering the cursor over a given view window and rotating the scroll wheel or other input device. Image indicator 512 can indicate the currently displayed image and the total number of images available for each view. Toolbar 517 can be displayed in the GUI 500 to allow the user to adjust various properties of the displayed visual information, such as zoom, contrast, etc.

[0053] Figure 5A The prompt 516 shown in GUI 500, "Identify the target center and catheter tip," is illustrated. Figure 5B As shown, and regarding Figure 2In processing 220, the control system can receive user input for selecting a target. User input can be, for example, clicking a mouse or touching a touchscreen, but can also be one or more additional or alternative user inputs (e.g., audible input, eye-tracking input, text input, etc.). The user can scroll through each image series before selecting a target until the target becomes visible. For example, in... Figure 5A In the middle, the target is not visible in GUI 500, but target 503 is. Figure 5B As can be seen from the comparison. Figure 5A and Figure 5B The image indicator 512 in each view window indicates that the user has scrolled through each image series until the target 503 becomes visible. For example, Figure 5A A view window 506 is shown displaying the image "291 / 512", and... Figure 5A View window 506 displaying "Image 211 / 512" is shown. A target icon 501 can be generated and displayed at a selected location in each of the view windows. This process can be repeated for each of view windows 502, 504, and 506. Alternatively, in some examples, target 503 can be selected only in one view window, and the control system can automatically identify target 503 in other view windows. As another example, the control system can identify target 503 independently of any user input based on known or expected properties of target 503 (e.g., expected pixel or voxel intensity values).

[0054] Margin 515 (e.g., margin 110) may be overlaid on the intraoperative image data and displayed in one or more view windows 502, 504, and 506. Margin 515 may surround target 503 and may optionally serve as a user-guided safety margin for treating (e.g., ablating) diseased cells. Margin 515 may be sized and shaped to account for portions of target 503 that are not visible in the intraoperative image data. In some examples, only one slice of intraoperative image data is displayed in GUI 500 at a time, and portions of target 503 may only become visible in other slices of image data as the user scrolls through the slices. Alternatively or additionally, margin 515 may be sized to account for any computational and / or execution errors that may exist in determining the size and shape of target 503.

[0055] In some examples, margin 515 may be uniform around the entire target 503. For example, margin 515 may provide a 5 mm buffer around target 503. For example, margin 515 may provide any other size buffer, such as 3 mm, 8 mm, 10 mm, or any other size buffer that may not be visible in a specific slice of intraoperative image data shown in view window 506. In some examples, one or more portions of margin 515 may be non-uniform around the entire target 503 and may have different lengths at different portions of target 503. For example, one portion of margin 515 may provide a 5 mm buffer, and another portion of margin 515 may provide a 3 mm buffer. Margin 515 may be non-uniform due to factors such as the type of target 503, proximity of one or more key structures (e.g., pleura, pulmonary fissure, blood vessels, heart, etc.), proximity of anatomical structures (e.g., anatomical passages, bones, etc.), or any other similar factors.

[0056] In some examples, the dimension of the margin 515 can be automatically set and / or adjusted by the control system based on the type of the target 503 to be treated. For example, the control system can retrieve information from a target database that includes the scale and other physical properties of different types of anatomical targets. Additionally or alternatively, the image processor of the control system can perform image analysis on intraoperative image data to determine the size and shape of the target 503.

[0057] In some examples, the size of margin 515 can be set and / or adjusted by the user. The size and / or shape of margin 515 can be adjusted or changed to account for patient movement (e.g., respiratory or circulatory movement) and / or deviations of CT from the body. For example, the control system can receive user input adjusting the size of margin 515 via a menu (not shown) of GUI 500. In some examples, a table or menu (not shown) may be displayed in GUI 500, including a list of spheres and / or ellipses with pre-specified sizes. The control system can receive user input selecting one or more of the spheres and / or ellipses. The selected spheres and / or ellipses can be overlaid on intraoperative image data and displayed in one or more view windows 502, 504, 506. In some examples, the control system can receive user input via GUI 500. Additionally or alternatively, user input may be a numerical value defining the distance by which margin 515 should extend beyond the outer edge of target 503.

[0058] Reference Figure 5CThe control system can receive user input selecting a location associated with the elongation device 507 (e.g., elongation device 104) in one or more view windows. Similar to target 503, the user can scroll through the corresponding image in each view window until the elongation device 507 or a portion thereof becomes visible. In the example shown, the distal tip of the elongation device 507 has been selected in each view window 502, 504, 506. An elongation device icon 505 has been generated at each selected location corresponding to the distal tip of the elongation device 507. The selected location of the distal tip of the elongation device 507 can be used for segmentation processing (e.g., Figure 2 The processing (220) provides initial information. For example, during segmentation, the control system may search for voxels with intensity values ​​similar to those of the voxel corresponding to the distal tip of the elongation device 507. Alternatively or additionally, the control system may automatically identify and segment the elongation device 507 from intraoperative image data. In such an example, the control system may populate the view window of the GUI 500 with the corresponding elongation device icon 505. The control system may request confirmation from the user to confirm that the identified location of the distal tip of the elongation device 507 is accurate.

[0059] After segmentation, the GUI 500 can display the segmentation extension device 507 superimposed on the intraoperative image data, such as... Figure 5D As shown. One or more view windows 502, 504, and 506 can display the elongation device boundary 509 and / or the elongation device centerline 511. Users can scroll through the image series and verify that the segmentation process appears to produce satisfactory results. For example, with... Figure 5D compared to, Figure 5E Different sets of images are shown in each view window 502, 504, 506, each image showing a segmented elongation device 507, a target icon 501, and an elongation device icon 505.

[0060] Figure 6A and Figure 6B It shows Figure 2 The update process handles the GUI before and after version 226. (e.g., GUI 600) Figure 6A As shown, a GUI 600, which may be the same as or similar to GUI 500, includes a virtual navigation image 602. The virtual navigation image 602 may display a 3D model 601. In some examples, an extension device 607 (e.g., extension device 507) and / or a target 108 may be superimposed on the 3D model 601. The navigation path 603 determined in process 208 may also be superimposed on the 3D model 601.

[0061] GUI 600 may also include a virtual camera device view 604 and / or a physical camera device view 606. The physical camera device view 606 displays video from a camera device positioned within the patient's anatomical structures. For example, the extension device 607 may include an endoscope, and the physical camera device view 606 may display video feeds from the endoscopic camera device. The virtual camera device view 604 displays a computer-generated image of the target 108 from the angle of the distal tip of the extension device 607. As can be seen in the virtual navigation image 602, the distal tip of the extension device 607 does not point to the location of the target 108. Therefore, in Figure 6A In the example shown, the computer-generated image of target 108 is not displayed in the virtual camera device view 604.

[0062] Additionally or alternatively, GUI 600 may include a navigation overview 608 that shows the width and branching relationships of various anatomical passages along the length of navigation path 603, as well as the progress of the extension device 607 along navigation path 603. Any or more of various other graphics, icons, indicators, or windows may also be included in GUI 600, such as an optimal fluoroscopy angle graphic 609, a driving force graphic 610, an orientation indicator 611, a target distance indicator 612, an anatomical structure boundary distance indicator 613, a tip bending radius graphic 614, or any combination thereof. The optimal fluoroscopy angle graphic 609 shows the recommended positioning angle of the fluoroscopy imaging system relative to the patient to optimize imaging of target 108 (e.g., minimizing obstructions). The optimal fluoroscopy angle graphic 609 may additionally or alternatively show the optimal imaging angle of any external imaging system, such as a fluoroscopy imaging system, a cone-beam CT imaging system, etc. The driving force graph 610 shows the force applied to navigate the extension device 607 to its current location (e.g., measured at the actuator or estimated at the distal tip of the extension device 607). The orientation indicator 611 shows the current orientation of the distal tip of the extension device 607 corresponding to the views provided by the virtual camera device view 604 and / or the physical camera device view 606. The target distance indicator 612 shows the current distance between the distal tip of the extension device 607 and the target 308. In the example shown, the target distance indicator 612 provides the distance from the distal tip of the extension device 607 to the nearest point on the target 108 and the distance from the distal tip of the extension device 607 to the farthest point on the target 108. In some examples, a single distance may be provided, such as the distance from the distal tip of the extension device 607 to the center point of the target 108 or a specific point of interest within the target 108. The anatomical boundary distance indicator 613 indicates the current distance between the distal tip of the elongation device 607 and the nearest anatomical element, such as the pleura, pulmonary fissure, or blood vessels, in the direction of the tip of the elongation device 607. The tip bending radius graph 614 indicates the current bending radius of the distal tip of the elongation device 607, which can be used during navigation to prevent excessive bending of the elongation device 607. Excessive bending of the elongation device 607 can damage the elongation device 607 or patient tissue.

[0063] Figure 6B The GUI 600 is shown after the target's location has been updated from the location of target 108 to the location of target 308. In some examples, the control system may determine an updated navigation path 605 to the updated location of target 308. Additionally, the location of at least a portion (e.g., the distal tip) of the elongation device 607 within the anatomical passage of model 601 may be updated. In some examples, such as Figure 6BAs shown, the updated positioning of target 308 can be within the field of view of the virtual camera device in virtual camera device view 604.

[0064] In some examples, the elongation device 607 may include a working channel, a medical device (e.g., Figure 8A The medical device 870 (which may be an ablation probe, biopsy needle, ultrasound transducer, etc.) can be inserted through the working channel. The segmentation of the elongation device 607 may also include the segmentation of the medical device. (The above refers to the segmentation of the elongation device.) Figure 2 The segmentation process discussed in process 218 can be similarly applied to segmenting medical devices. In this regard, the medical device can be identified separately from the elongation device 607 in the GUI 600. When the medical device extends or protrudes from the elongation device 607 during intraoperative imaging (e.g., from the distal tip of the elongation device 607), the control system can analyze the segmented target 308 and the segmented medical device to determine whether the medical device is in the desired deployment position. Additionally or alternatively, the control system can receive user input indicating whether the medical device is in the desired deployment position.

[0065] In some examples, Figure 2 The display processing 228 can be used Figure 2 The update process will be executed after step 226. Alternative location, Figure 2 The display processing 228 can be as described above regarding Figure 2 The process is performed after identifying the target in the intraoperative image data, as discussed in 220. Figure 7A The above instructions are shown. Figure 2 The various processing methods discussed use a GUI 700. GUI 700 may be the same as or similar to GUI 500. GUI 700 includes view windows 702, 704, and 706. Although three view windows are shown, more or fewer view windows may be displayed at any given time.

[0066] In some examples, view window 706 displays an oblique view of the patient's anatomy. The oblique view allows the user to more easily view the target 710, the elongation device 730, and the treatment area 740 (e.g., the ablation area) while scrolling through slices of CBCT image data, which will be discussed in further detail below.

[0067] As discussed above regarding GUI 500, the intraoperative image data displayed in GUI 700 may include an image of anatomical target 710 (e.g., target 108). Margins 712 (e.g., margin 515) may be overlaid on the intraoperative image data and displayed in one or more of view windows 702, 704, and 706. Figure 7BA close-up view of intraoperative image data is shown, including target 710 and the surrounding area of ​​patient anatomy structures very close to target 710. The following is about... Figure 7B Some or all of the features discussed can be displayed Figure 7A In the GUI 700 shown.

[0068] In some examples, the size and / or orientation of target 710 can be adjusted based on an image of target 710 displayed in intraoperative image data. For example, the control system may receive one or more user inputs to adjust the size and / or orientation of target 710 to generate an updated target 715. Alternatively or additionally, the control system may automatically adjust the size and / or orientation of target 710 to generate an updated target 715. In some examples, the control system may receive user input to select one or more adjustment icons 711. The user can interact with the GUI 700 by touching and dragging the adjustment icons 711 around the touchscreen of the display system. Adjusting the position of the adjustment icons adjusts the size and / or orientation of target 710 accordingly to generate an updated target 715. As discussed above, the user can scroll through slices of intraoperative image data to determine whether the updated target 715 contains an image of target 710 shown in the intraoperative image data. Alternatively or additionally, the control system can determine whether the updated target 715 contains an image of target 710 shown in the intraoperative image data by analyzing each slice of intraoperative image data as the user scrolls through the slices.

[0069] When adjustment icon 711 is moved to update the size and / or orientation of target 710, the control system can automatically adjust the size and / or orientation of margin 712. Alternatively or additionally, when the control system receives user input to adjust one or more adjustment icons (not shown) associated with margin 712, the size and / or orientation of margin 712 can be adjusted.

[0070] Reference Figure 2In processing 228, a graphic indicator 720 may be overlaid on the intraoperative image data and displayed in one or more view windows 702, 704, and 706. The graphic indicator 720 indicates the target location of the distal tip of a medical device (e.g., an ablation probe, biopsy needle, ultrasound transducer, etc.) within the patient's anatomy. As discussed above, the medical device may be inserted via an extension device 730 (e.g., extension device 104). The graphic indicator 720 may be overlaid on the intraoperative image data before the medical device is deployed from the extension device 730. The graphic indicator 720 can assist the operator in determining where to position the distal tip of the extension device 730 near the target 710 by allowing the operator to move the graphic indicator 720 to different positions. The graphic indicator 720 may additionally or alternatively indicate the target location of another part of the medical device (e.g., ablation probe, biopsy needle, ultrasound transducer, etc.) in the patient's anatomy, such as the midpoint of the medical device, the location of one or more reference points on the medical device, the location of one or more imagers (e.g., ultrasound transducer or other imagers) on the medical device, the location of one or more electrodes (e.g., microwave ablation electrode, electroporation electrode, etc.) on the medical device, and / or any other part of the medical device.

[0071] The positioning of the graphic indicator 720 in the intraoperative image data can be adjusted. In some examples, the control system can receive user input selecting the graphic indicator 720 on the GUI 700 via a touchscreen of the display system. User input may include touching the graphic indicator and dragging it to a new location in the intraoperative image data. Any other user input, such as clicking a mouse and dragging the cursor to a new location, can be used to update the positioning of the graphic indicator 720. The positioning of the graphic indicator 720 can be adjusted based on the location and / or orientation of the target 710 and any other anatomical or critical structures shown in the intraoperative image data to fine-tune the target position of the distal tip of the medical device.

[0072] In some examples, the graphical indicator 720 may be outside the view displayed in the view window 706. In these examples, an icon or other indicator (e.g., text, audible, tactile, etc.) may indicate the location of the graphical indicator 720. In some examples, the target 710 may be outside the view displayed in the view window 706. In these examples, an icon or other indicator (e.g., text, audible, tactile, etc.) may indicate the location of the target 710.

[0073] In examples where the medical device is an ablation probe, the treatment area 740 can be the ablation zone. The treatment area 740 can be overlaid on intraoperative image data and displayed in one or more view windows 702, 704, and 706. In some examples, the size and / or orientation of the treatment area 740 is adjusted when the positioning of the graphic indicator 720 is adjusted.

[0074] Treatment area 740 represents the predicted area to be treated (e.g., ablated) by a medical device (e.g., an ablation probe) during a single energy delivery over a single, uninterrupted duration during a single treatment procedure, such as an ablation procedure. The area covered by treatment area 740 may be an ablation area. Additional uninterrupted energy deliveries at different times and / or in different locations may create additional separate treatment areas covering the separate treatment area (e.g., a separate ablation area). In some cases, separate treatment areas and treatment areas may be used, as will be described in more detail below.

[0075] The predicted size and shape of the treatment zone 740 are based on the design and construction of the ablation probe, the amount of energy applied to the probe, the duration of energy application to the probe, and one or more tissue characteristics of the target 710 in which the ablation probe is deployed. Tissue characteristics of the target 710 may include density, stiffness, percentage of emphysema surrounding the target 710, fibrosis, necrotic tissue, proximity to critical structures, or any other physical characteristics of the target 710 or the surrounding anatomical structures. Therefore, ablation zones of various shapes and sizes can be predicted within anatomical structures with different tissue characteristics and by varying the power and duration of energy delivery at different ablation probe transducer locations.

[0076] In some examples, the treatment area 740 can be determined during the preoperative planning process. The preoperatively determined treatment area can be overlaid on intraoperative image data and displayed in one or more view windows 702, 704, and 706. The size / orientation of the treatment area 740 can be adjusted as needed to account for any differences in patient anatomy (e.g., target, size and / or location of anatomical structures, etc.) that may have occurred since the preoperative determination of the size / or orientation of the treatment area.

[0077] In some examples, treatment parameters for the treatment to be applied by the medical device can be adjusted based on the real-time position and orientation of the medical device in intraoperative imaging data. For instance, when the medical device is an ablation probe, the size / orientation of the treatment area 740 can be adjusted by changing the power setting, duration setting, or positioning of the ablation probe, which may alter the center of the treatment area 740. Treatment parameters can be adjusted by the control system and / or the user.

[0078] Treatment area 740 may include one or more margins, such as margin 742 and margin 744. One or both of margins 742 and 744 may be overlaid on intraoperative image data and displayed in one or more view windows 702, 704, and 706. Margin 742 lies between margin 712 of treatment area 740 and target 710. Margin 744 surrounds treatment area 740. Margin 744 may indicate a safe margin for preventing treatment (e.g., ablation energy) from affecting critical structures or other anatomical structures that may be very close to target 710. A critical structure may be very close to target 710 when it is within a threshold distance from target 710 or a threshold distance from treatment area 740. Additional details regarding treatment area 740 will be discussed below. Margin 742 may indicate a buffer zone for ensuring that target 710 is fully treated by the medical device. For example, because the treatment area 740 surrounds the margin 742, and because the margin 742 surrounds the margin 712 of the target 710, the control system and / or the user can determine that the ablation treatment will completely treat the target 710.

[0079] In some examples, margin 742 and margin 712 represent the same margin. Margin 742 is calculated as the distance measured radially inward from treatment area 740. Margin 712 is calculated as the distance measured radially outward from target 710 and / or the later target 715. Both margin 742 and margin 712 indicate a buffer zone used to ensure that target 710 is fully treated by the medical device. One or both of margin 742 and margin 712 may be displayed in GUI 700.

[0080] In some examples, more than one ablation or other treatment may be required to completely treat target 710. In these examples, one or more additional graphic indicators may be overlaid on the intraoperative image data, and one or more additional treatment areas corresponding to the additional graphic indicators may also be overlaid on the intraoperative image data.

[0081] As discussed above, the positioning of the graphic indicator 720 can be adjusted. When the positioning of the graphic indicator 720 is adjusted, the size and / or orientation of the treatment area 740 can also be adjusted. For example, when the positioning of the graphic indicator 720 is adjusted, the treatment area 740 can move together with the graphic indicator 720. Figure 7C The initial positioning graphic indicator 720 is shown, and Figure 7D A graphical indicator 720 showing the updated positioning is displayed. (See figure.) Figure 7D As seen in the image, treatment area 740 has been adjusted based on the updated positioning of graphical indicator 720.

[0082] In some examples, the medical device can be deployed from the extension device 730 before or after the positioning of the graphic indicator 720 is finally determined. Figure 7E A view window 704 is shown, which may be the virtual camera device view 604 in the GUI 600 discussed above. In some examples, one, some, or all of the features discussed below that may be displayed in the view window 704 may be hidden from the view. For example, user input indicating that one or more specific features should be hidden from the view window 704 may be received.

[0083] exist Figure 7E In this view window, a graphic indicator 720 is included, as discussed above, which has been transformed into the elongation device reference frame 350. View window 704 includes an image of the graphic indicator 720, an image of the updated target 715, an icon 752 which may be a "distance to target" icon, and an icon 754 which may be a "distance to graphic indicator" icon. The "distance to target" icon 752 shows the distance 751 between the distal tip (or other reference portion) of the elongation device 730 and the near edge (e.g., the nearest edge) of the updated target 715. Figure 7E The distance 751 shown is 10 mm. This distance is only illustrative and may change as the elongation device 730 moves relative to the target 715. The “distance to target” icon 752 may also show the distance 753 between the distal tip (or other reference portion) of the elongation device 730 and the distal edge (e.g., the furthest edge) of the updated target 715. Figure 7E The distance 753 shown is 15 mm. This distance is merely illustrative and may change as the elongation device 730 moves relative to the target 715. In some examples, the view window 704 may include a virtual trajectory icon that may represent a trajectory planned using preoperative imaging data. The “Distance to Target” icon 752 may additionally or alternatively show one or more of the following: the distance between the distal tip (or other reference portion) of the elongation device 730 and the proximal edge (e.g., the nearest edge) of the updated target 715; the distance between the distal tip (or other reference portion) of the elongation device 730 and the distal edge of the treatment area 740; the distance between the distal tip (or other reference portion) of the elongation device 730 and the proximal edge of the treatment area 740; the distance between the distal tip (or other reference portion) of the elongation device 730 and the center of the updated target 715; or the distance between the distal tip (or other reference portion) of the elongation device 730 and any designated point within the updated target 715 or treatment area 740.

[0084] The “Distance to Graphic Indicator” icon 754 indicates the distance 755 between the distal tip (or other reference portion) of the elongation device 730 and the graphic indicator 720. Figure 7EThe distance 755 shown is 21 mm. This distance is merely illustrative and may change as the elongation device 730 moves relative to the target 715. In some examples where the graphic indicator 720 has a three-dimensional shape (e.g., a sphere or ellipsoid), in addition to or instead of distance 755, the "distance to graphic indicator" icon 754 may also show the distance between the distal tip (or other reference portion) of the elongation device 730 and the near edge (e.g., the nearest edge) of the graphic indicator 720 and / or the distance between the distal tip (or other reference portion) of the elongation device 730 and the distal edge (e.g., the farthest edge) of the graphic indicator 720. For example, the "distance to graphic indicator" icon 754 may show a distance 756 between the distal tip (or other reference portion) of the elongation device 730 and the near edge (e.g., the nearest edge) of the graphic indicator 720. Figure 7E The distance 756 shown is 11 mm. This distance is only illustrative and may change as the elongation device 730 moves relative to the graphic indicator 720. The “Distance to Graphic Indicator” icon 754 may also show the distance 757 between the distal tip (or other reference portion) of the elongation device 730 and the distal edge (e.g., the furthest edge) of the graphic indicator 720. Figure 7E The distance 757 shown is 31 mm. This distance is only illustrative and can be changed as the extension device 730 moves relative to the graphic indicator 720.

[0085] Based on the registration between the intraoperative image reference system 450 and the elongation device reference system 350, the graphic indicator 720 can be transformed from the intraoperative image reference system 450 to the elongation device reference system 350. Additionally or alternatively, any other key structures can be transformed from the intraoperative image reference system 450 to the elongation device reference system 350. During the navigation of the elongation device 730 toward the target 710, the transformed graphic indicator 720 and / or the transformed key structures can be displayed in the GUI 600. The ability to view the graphic indicator 720 and / or key structures in the elongation device reference system 350 can aid in navigating the elongation device 730 to its target location within the patient's anatomy.

[0086] Figure 7FMenu 780 and view window 708 of GUI 700 are shown. As discussed above, in addition to indicating the ablation zone, treatment area 740 can also be a reference area, which may additionally or alternatively indicate areas of radiation, reference, tissue, and other anatomical structures that may be affected by any other treatment or procedure, or focal areas within or around anatomical structures, such as target 715. Also as discussed above, treatment area 740 and any other reference areas can be determined during the preoperative planning process. The preoperatively determined treatment / reference areas can be overlaid on intraoperative image data and displayed in one or more view windows 702, 704, 706, and 708. The size / orientation of the treatment / reference area can be adjusted as needed to account for any differences in patient anatomy (e.g., target, size and / or location of anatomical structures, etc.) that may have occurred since the preoperative determination of the size / orientation of the treatment / reference area. Additionally, margins can be added to one or more reference areas or targets during the preoperative planning process. Preoperatively determined margins can be overlaid on intraoperative image data and displayed in one or more of view windows 702, 704, 706, and 708. The size / orientation of the margins can be adjusted as needed to account for any differences in patient anatomy (e.g., target, size and / or location of anatomical structures, etc.) that may have occurred since the preoperative determination of the margin size / orientation.

[0087] Menu 780 may include one or more of a “Direct Placement” icon 782, a “Trajectory Placement” icon 784, and / or a “Margin” icon 786. GUI 700 may include one or more view icons, such as an axial view icon 791, a coronal view icon 792, a sagittal view icon 793, a apical view icon 794, and / or a side view icon 795. The apical and side views may each display the patient's anatomy in view window 708 from a corresponding oblique view. The oblique view may be a view of the patient's anatomy displayed based on the viewing direction relative to the elongation device 730. For example, the apical view displays the patient's anatomy as shown from an angle including a viewing direction aligned with the longitudinal axis of the distal end 732 of the elongation device 730. The side view displays the patient's anatomy as shown from an angle showing the side of the elongation device 730 and including a viewing direction generally perpendicular to the longitudinal axis of the elongation device 730. The oblique view may additionally or alternatively be based on a trajectory between two points, such as the trajectory between the distal tip 732 of the elongation device 730 and a second point at any location on the elongation device 730. The oblique view may additionally or alternatively be based on a trajectory between two points, such as the trajectory between the distal tip 732 of the elongation device 730 and any other second point (e.g., the center of the treatment area 740, the center of the updated target 715, a reference point, etc.). For example, the tip view and the side view may represent a plane extending between the distal tip 732 and the second point. In some examples, the second point may be selected based on received user input. In contrast, the axial view, coronal view, and sagittal view show views based on the patient's anatomy displayed relative to the patient's viewing direction.

[0088] User input can be received at GUI 700 to indicate which of the reference regions and / or new reference regions determined during the preoperative planning process will be superimposed on the intraoperative imaging data displayed in view window 708. Reference regions can be placed directly on the intraoperative imaging data (e.g., by receiving user input at the "Direct Placement" icon 782) and / or by a trajectory showing the spatial relationship between the reference region and the distal end 732 of the elongation device 730 (e.g., by receiving user input at the "Track Placement" icon 784). One, some, or all of the reference regions and / or targets 715 may include margins. Margins can be displayed and adjusted by receiving user input at the margin icon 786.

[0089] Figure 7GView window 708 is shown in coronal view, in which reference region 850 is displayed. Reference region 850 is a directly placed reference region. For example, reference region 850 is displayed in view window 708 without any spatial relationship between the reference region shown and the distal end 732 of elongation device 730. Reference region 850 may indicate an ablation zone, radiation zone, reference zone, area of ​​tissue and other anatomical structures that may be affected by any other treatment or procedure, or a focal area near target 715. In some examples, the size and / or orientation of reference region 850 may be adjusted. For example, the control system may receive one or more user inputs to adjust the size and / or orientation of reference region 850 to generate an updated reference region. Additionally or alternatively, the control system may automatically adjust the size and / or orientation of reference region 850 to generate an updated reference region. In some examples, the control system may receive user input to select one or more adjustment icons 852. The user can interact with GUI 700 by touching and dragging adjustment icons 852 around the touchscreen of the display system. Adjusting the position of the adjustment icon 852 accordingly adjusts the size and / or orientation of the reference region 850 to generate the reference region. As discussed above, the user can scroll through slices of intraoperative image data to determine whether the updated reference region includes portions of the patient's anatomy shown in the intraoperative image data that the user wishes the reference region 850 to cover. Before or after adjusting the size / orientation of the reference region 850, user input can be received at the "Placement" icon 788 to finalize the overlay positioning of the reference region 850 on the intraoperative imaging data in the view window 708.

[0090] Figure 7H The view window 708 is shown in the side view, which displays reference regions 850 and 860. Reference region 860 is a trajectory placement reference region. For example, reference region 860 and trajectory marker 862, which shows the spatial relationship between reference region 860 and the distal end 732 of the elongation device 730, are displayed in view window 708. For example, trajectory marker 862, which may be a solid line, a dotted line, a dashed line, or any other symbol, extends between the distal end 732 of the elongation device 730 and the center of reference region 860.

[0091] Reference region 860 can indicate the ablation zone, radiation zone, reference zone, tissue and other anatomical structures that may be affected by any other treatment or procedure, or the focal region near target 715. In some examples, the size and / or orientation of reference region 860 can be adjusted in the same manner as described above for adjusting the size / orientation of reference region 850. For example, the diameter, length, distal margin offset, and other properties of reference region 860 can be adjusted via user input received at adjustment menu 865. The size and / or orientation of reference region 850 can be adjusted similarly using adjustment menus that are the same as or different from adjustment menu 865. Before or after adjusting the size / orientation of reference region 860, user input can be received at "Placement" icon 788 to finalize the superimposed positioning of reference region 860 on intraoperative imaging data in view window 708.

[0092] For example Figure 7H As shown, the reference region 860 includes a margin 864. As discussed above with respect to the treatment area 740, the margin 864 of the reference region 860 can extend radially inward and / or radially outward from the reference region 860. Figure 7H The margin 864 extends radially inward and radially outward from the reference region 860. In some examples, the size and / or orientation of the margin 864 can be adjusted in the same manner as described above for adjusting the size / orientation of the reference region 860.

[0093] Figure 7I The side view shows view window 708, which displays reference area 850, reference area 860, and the margin 712 of target 715. (As mentioned above regarding...) Figure 7B The discussed margin 712 can be adjusted in size / orientation. One or more margins associated with each of reference area 850, reference area 860, and target 715 can be hidden from the view. For example, in Figure 7I In the middle, the margins for reference regions 850 and 860 are hidden, while the margin 712 for target 715 is displayed.

[0094] Reference Figure 2 In processing 232, after the medical device is deployed, intraoperative imaging scanning can be performed during deployment. Based on the intraoperative imaging data captured during this intraoperative imaging scan, the real-time position of the treatment area 740 can be determined. The control system can automatically adjust the size and / or orientation of the treatment area based on the updated intraoperative imaging data to avoid any critical structures and / or completely surrounding the target. Additionally or alternatively, the control system can receive user input for adjusting the size and / or orientation of the treatment area.

[0095] Reference Figure 8AThe GUI 700 may include an intraoperative external image 760 and an icon menu 765. The intraoperative external image 760 may be received from an intraoperative external imaging system at a control system. In some examples, the intraoperative external imaging system may be a CBCT imaging system that generates intraoperative cone-beam CT (“CBCT”) image data; however, any suitable imaging technique, such as conventional CT or fluoroscopy, may be used without departing from the examples of this disclosure. The intraoperative external imaging data may be received at a control system or another processing platform associated with the elongation device 730. It is also contemplated that, in some examples, shape data associated with the elongation device 730 may be transferred to the imaging system, or both shape data and intraoperative external imaging data may be transferred to a common platform for processing. In this regard, the registration of the shape data of the elongation device 730 with the intraoperative external imaging data may be performed by the control system, the imaging system, or another platform operatively communicating with the intraoperative external imaging system and the control system. In some examples, receiving the intraoperative external imaging data may include receiving one or more timestamps associated with the intraoperative external imaging data. A first timestamp can indicate the start time of intraoperative external imaging, and a second timestamp can additionally indicate the stop time of intraoperative external imaging. Alternatively, timestamps can be associated with each instance of intraoperative external imaging data. To ensure accurate correlation, the clock of the control system of the elongation device 730 can be synchronized with the clock of the intraoperative external imaging system, and each instance of shape data can also be associated with a timestamp. In this respect, each timestamped instance of intraoperative external imaging data can be paired with a corresponding timestamped instance of shape data.

[0096] As discussed above, when the medical device 770 is deployed from the elongation device 730, the intraoperative external image 760 may include images of the elongation device 730 and the medical device 770 (e.g., an ablation probe). In some examples, the position and / or orientation of the medical device 770 may be adjusted.

[0097] like Figure 8BAs shown, GUI 700 may include an increase icon 802 and a decrease icon 804. When the control system receives user input selecting the increase icon 802, the insertion distance D1 of the medical device 770 can be increased, which can be indicated by the insertion distance icon 800. Similarly, when the control system receives user input selecting the decrease icon 804, the insertion distance D1 can be decreased. The insertion distance D1 can be increased or decreased in increments of 1 mm, but can also be increased or decreased in increments of 0.5 mm, 2 mm, 3 mm, or any other desired distance. To reset the insertion distance D1 to the default insertion distance, icon 806 can be selected. In some examples, the medical device 770 can be hidden and removed from the virtual navigation view 602. When the control system receives user input selecting the "Done" icon 808, any adjustments made to the insertion distance D1 and / or any other adjustments made to the medical device 770 can be confirmed. Confirmed adjustments can be reviewed and further adjusted at any time.

[0098] In some examples, the control system may receive user input that selects (e.g., touches or clicks) the distal end 772 of the medical device 770 and the distal end 732 of the extension device 730 to measure the insertion distance D1 of the medical device 770. Alternatively or additionally, the control system may use imaging analysis to determine the insertion distance D1 of the medical device 770. The insertion distance can be adjusted to align the medical device 770 with the treatment area 740. This can result in more efficient and accurate treatment of the target structure 715.

[0099] like Figure 8C As shown, medical device 770 is depicted bent away from the longitudinal axis A of elongation device 730 at an angle 810. Angle 810 can be measured from the longitudinal axis A of elongation device 730 to medical device 770. The degree measure of angle 810 can be shown in icon 812 of GUI 700. GUI 700 may also include an increase icon 814 and a decrease icon 816. When the control system receives user input selecting increase icon 814, angle 810 can increase. Similarly, when the control system receives user input selecting decrease icon 816, angle 810 can decrease. Angle 810 can increase or decrease in increments of 1° (i.e., one degree), but can also increase or decrease in increments of 0.5°, 2°, 3°, or any other desired degree. Figure 8C As shown, angle 810 can be 30°. In some examples, the medical device 770 can be bent to any angle within the deployment range of the extension device 730. Angle 810 can be adjusted to align the medical device 770 with the treatment area 740. This can result in more efficient and accurate treatment of the target structure 715.

[0100] Reference Figure 2In processing 234, during or after confirming the treatment area 740 and / or confirming the position / or orientation of the medical device, the extension device 730 may be retracted. The extension device 730 may be retracted to prevent damage to the extension device during a treatment procedure (e.g., an ablation procedure) performed by the medical device (e.g., an ablation probe). In some examples, the control system may receive user input instructing the extension device 730 to retract.

[0101] Figure 7E The distance 755 between the distal tip of the elongation device 730 and the graphic indicator 720 can be used to confirm that the elongation device 730 has retracted. For example, the distance 755 can be changed from 21 mm to 24 mm to indicate that the elongation device 730 has retracted 3 mm from the graphic indicator 720, and therefore 3 mm from the distal tip of the medical device. Any other retraction distance can be used, such as 5 mm, 4 mm, 2 mm, etc. The desired retraction distance can vary based on the type of treatment performed by the medical device, the intensity of the treatment, or any other similar factors.

[0102] In some examples, the registration techniques of this disclosure, such as those concerning... Figure 2 The registration techniques discussed in 210 and 222 can be used to process, for example, Figure 9 and Figure 10 The robot-assisted medical system shown is used in image-guided medical procedures. Figure 9A clinical system 10 is shown, comprising a robot-assisted medical system 900 and an intraoperative imaging system 918. The robot-assisted medical system 900 typically includes a manipulator assembly 902 for operating a medical device system 904 (including, for example, an extension device 104) while performing various procedures on a patient P positioned on a table T within a surgical environment 901. The manipulator assembly 902 can be a robot-assisted, non-assisted, or hybrid robot-assisted and non-assisted component, having selectable degrees of freedom of motion that can be motorized and / or robot-assisted, as well as selectable degrees of freedom of motion that can be non-motorized and / or non-assisted. A main assembly 906, which may be inside or outside the surgical environment 901, typically includes one or more control devices for controlling the manipulator assembly 902. The manipulator assembly 902 supports the medical device system 904 and may optionally include multiple actuators or motors that drive inputs on the medical device system 904 in response to commands from a control system 912. The actuator may optionally include a drive system that, when coupled to the medical device system 904, can advance the medical device system 904 into a natural or surgically created anatomical opening. Other drive systems can move the distal end of the medical device system 904 in multiple degrees of freedom, including three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of rotational motion (e.g., rotation about the X, Y, Z Cartesian axes). Additionally, the actuator may be used to actuate an engageable end effector of the medical device system 904 to grasp tissue in the jaws of a biopsy device, etc.

[0103] The robot-assisted medical system 900 also includes a display system 910 (which may be the same as display system 100), which displays images or representations of the surgical site and medical device system 904, generated by sensor system 908 and / or endoscopic imaging system 909. Display system 910 and main component 906 can be oriented such that an operator O can control medical device system 904 and main component 906 through remotely presented perception.

[0104] In some examples, medical device system 904 may include components for surgery, biopsy, ablation, illumination, irrigation, or aspiration. Optionally, medical device system 904, together with sensor system 908, may be used to collect (e.g., measure) a set of data points corresponding to a location within the anatomical passage of a patient (e.g., patient P). In some examples, medical device system 904 may include components of imaging system 909, which may include an imaging endoscope assembly or imaging instrument that records simultaneous or real-time images of the surgical site and provides the images to an operator or operator O via display system 910. The images may be, for example, two-dimensional or three-dimensional images captured by an imaging instrument positioned within the surgical site. In some examples, imaging system components may be integrally or detachably coupled to medical device system 904. However, in some examples, a separate endoscope attached to a separate manipulator assembly may be used in conjunction with medical device system 904 to image the surgical site. The imaging system 909 may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, said one or more computer processors may include the processor of the control system 912.

[0105] The sensor system 908 may include a position / positioning sensor system (e.g., an electromagnetic (EM) sensor system) and / or a shape sensor system for determining the position, orientation, velocity, rate, attitude and / or shape of the medical device system 904.

[0106] The robot-assisted medical system 900 may further include a control system 912. The control system 912 includes at least one memory 916 and at least one computer processor 914, the computer processor 914 being used to implement control between the medical device system 904, the main component 906, the sensor system 908, the endoscopic imaging system 909, and the display system 910. The control system 912 also includes programming instructions (e.g., a non-transitory machine-readable medium storing instructions) for implementing some or all of the methods described according to the aspects disclosed herein, said instructions including instructions for providing information to the display system 910.

[0107] The control system 912 may optionally also include a virtual visualization system to provide navigational assistance to the operator O when controlling the medical device system 904 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system can be based on a reference to a preoperative or intraoperative dataset of the acquired anatomical access. The virtual visualization system processes images of the surgical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc.

[0108] Intraoperative imaging system 918 may be positioned within surgical environment 901, near patient P, to acquire images of patient P during the medical procedure. Intraoperative imaging system 918 may provide real-time or near-real-time images of patient P. In some examples, intraoperative imaging system 918 may be a mobile C-arm cone-beam CT imaging system for generating three-dimensional images. For example, intraoperative imaging system 918 may be a DynaCT imaging system from Siemens, Washington, D.C., or other suitable imaging system. In other examples, the imaging system may use other imaging techniques, including CT, MRI, fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc.

[0109] Figure 10 It shows a surgical reference frame (X). S Y S Z S Surgical environment 1000 is 250, in which patient P is positioned on table T. Patient P may be stationary in the surgical environment in the sense that the patient's overall movement is restricted by sedation, restraint, and / or other means. Periodic anatomical movements of patient P (including respiratory and cardiac movements) may continue unless the patient is asked to hold his or her breath to temporarily pause respiratory movements. Within surgical environment 1000, there is a medical device reference system (X). M Y M Z M A medical device 1004 (e.g., medical device system 904) of 350 is coupled to an instrument holder 1006. In this example, the medical device 1004 includes an extension device 1010, such as a flexible catheter, coupled to an instrument body 1012. The instrument holder 1006 is mounted to an insertion stage 1008 fixed within a surgical environment 1000. Alternatively, the insertion stage 1008 may be movable but has a known position within the surgical environment 1000 (e.g., via a tracking sensor or other tracking device). In these alternatives, the medical device reference frame is fixed or otherwise known relative to a surgical reference frame. The instrument holder 1006 may be part of a robot-assisted manipulator assembly (e.g., robot-assisted manipulator assembly 1002) coupled to the medical device 1004 to control insertion movements (e.g., movement along axis A) and, optionally, movement of the distal end 1018 of the extension device 1010 in multiple directions including yaw, pitch, and roll. The instrument holder 1006 or insertion stage 1008 may include an actuator, such as a servo motor (not shown), for controlling the movement of the instrument holder 1006 along the insertion stage 1008.

[0110] In this example, the sensor system (e.g., sensor system 908) includes a shape sensor 1014. The shape sensor 1014 may include an optical fiber extending within and aligned with the elongation device 1010. In one example, the optical fiber has a diameter of approximately 200 μm. In other examples, the scale may be larger or smaller. The optical fiber of the shape sensor 1014 forms an optical fiber bending sensor for determining the shape of the elongation device 1010. In an alternative, an optical fiber including a fiber Bragg grating (FBG) is used to provide strain measurements in one or more dimensions of the structure. Various systems and methods for monitoring the shape and relative position of optical fibers in three dimensions are described in U.S. Patent Application No. 11 / 180,389 (filed July 13, 2005) (disclosing "Fiber optic position and shape sensing device and method relating thereto"), U.S. Patent Application No. 12 / 047,056 (filed July 16, 2004) (disclosing "Fiber-optic shape and relative position sensing"), and U.S. Patent No. 6,389,187 (filed June 17, 1998) (disclosing "Optical Fiber Bend Sensor"), all of which are incorporated herein by reference in their entirety. In some examples, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In some examples, other techniques may be used to determine the shape of the conduit. For example, the history of the distal attitude of the elongation device 1010 may be used to reconstruct the shape of the elongation device 1010 over time intervals.

[0111] like Figure 10 As shown, the device body 1012 is coupled and fixed relative to the device holder 1006. In some examples, the fiber optic shape sensor 1014 is fixed at a proximal point 1016 on the device body 1012. In some examples, the proximal point 1016 of the fiber optic shape sensor 1014 may move with the device body 1012, but the location of the proximal point 1016 may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 1014 measures the medical device reference frame (X). M Y M Z M The shape of the extension device 1010 from the proximal point 1016 to another point (e.g., the distal end 1018 of the proximal point 1016).

[0112] The extension device 1010 includes a channel (not shown) sized and formed to receive a medical device 1022. In some examples, the medical device 1022 can be used for procedures such as surgery, biopsy, ablation, illumination, irrigation, or aspiration. The medical device 1022 can be deployed via the extension device 1010 and used at a targeted location within an anatomical structure. The medical device 1022 may include, for example, an image capture probe, a biopsy instrument, a laser ablation fiber, and / or other surgical, diagnostic, or therapeutic tools. The medical device 1022 can be advanced from the distal end 1018 of the extension device 1010 to perform the procedure and then retracted into the channel upon completion of the procedure. The medical device 1022 can be removed from the proximal end of the extension device 1010 or along the extension device 1010 from another optional instrument port (not shown).

[0113] The extension device 1010 may also accommodate cables, linkages, or other steering control (not shown) to controllably bend the distal end 1018. In some examples, at least four cables are used to provide independent "up-down" steering control to control the pitch of the distal end 1018 and "left-right" steering control to control the yaw of the distal end 1018.

[0114] The position measuring device 1020 provides information about the position of the instrument body 1012 as it moves along the insertion axis A on the insertion stage 1008. The position measuring device 1020 may include a rotary transformer, an encoder, a potentiometer, and / or other sensors that determine the rotation and / or orientation of the actuators controlling the movement of the instrument carriage 1006 and thus the movement of the instrument body 1012. In some examples, the insertion stage 1008 is linear, while in other examples, the insertion stage 1008 may be curved, or have a combination of curved and linear segments.

[0115] Intraoperative imaging system 1030 (e.g., imaging system 918) is positioned near patient P to acquire three-dimensional images of the patient as the elongation device 1010 extends within the patient's body. Intraoperative imaging system 1030 can provide real-time or near-real-time images of patient P. During preoperative and intraoperative imaging, one or more reference markers (not shown) can be positioned on patient P to improve registration.

[0116] In some examples, another component of medical device 1004 or a robot-assisted medical system registered with medical device 1004 may include device clock 1024. Imaging system 1030 may include imaging clock 1026. Clocks 1024 and 1026 may be time-synchronized according to a predetermined schedule or in response to a synchronization initiation event generated by a user, control system, or synchronization system. In some examples, clocks 1024 and 1026 may be components of a synchronization system, which may be a centralized or distributed system, and may also include servers, wired or wireless communication networks, communication devices, or other components for performing synchronization algorithms and protocols. In some examples, another component of medical device 1004 or a robot-assisted medical system registered with medical device 1004 may include communication device 1028. Imaging system 1030 may include communication device 1032.

[0117] Unless the context otherwise indicates, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well. Furthermore, the terms “comprises,” “comprising,” “includes,” “has,” etc., specify the presence of the stated feature, step, operation, element, and / or component, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be directly electrically or mechanically coupled, or they may be indirectly coupled via one or more intermediate components. Components described as coupled may be directly or indirectly communicatively coupled. The auxiliary verb “may” similarly implies that a feature, step, operation, element, or component is optional.

[0118] In the description, specific details describing some examples have been set forth. Numerous specific details have been set forth to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that some examples can be practiced without some or all of these specific details. The specific examples disclosed herein are intended to be illustrative and not restrictive. Those skilled in the art can implement other elements, though not specifically described herein, that fall within the scope and spirit of this disclosure.

[0119] Where feasible, elements described in detail with reference to an example, implementation, or application may optionally be included in other examples, implementations, or applications in which they are not specifically shown or described. For example, if an element is described in detail with reference to an example but not with reference to a second example, that element may still be considered included in the second example. Therefore, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with an example, implementation, or application may be incorporated into other examples, implementations, or aspects unless otherwise specifically described, unless the one or more elements would render the example or implementation ineffective, or unless two or more elements provide conflicting functionality.

[0120] Any changes and other modifications to the described apparatus, instruments, and methods, as well as any other application of the principles of this disclosure, are entirely contemplated as would normally occur to those skilled in the art related to this disclosure. Furthermore, the dimensions provided herein are for specific examples, and it is contemplated that the concepts of this disclosure can be realized using different sizes, scales, and / or ratios. To avoid unnecessary descriptive repetition, one or more components or actions described according to one illustrative example may be used or omitted as appropriate in other illustrative examples. For the sake of brevity, numerous iterations of these combinations will not be described separately. For simplicity, in some cases, the same reference numerals are used throughout the figures to refer to the same or similar components.

[0121] The systems and methods described herein are applicable to navigating and treating anatomical tissues via naturally or surgically created access channels in any of a variety of anatomical systems, including the lungs, colon, intestines, kidneys and renal calyces, brain, heart, and circulatory systems, including the vascular system. While some examples described herein relate to surgical procedures or instruments, or medical procedures and instruments, the disclosed techniques are applicable to non-medical procedures and non-medical instruments. For example, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial applications, general robotic applications, and sensing or manipulating non-tissue artifacts. Other example applications involve cosmetic improvements, imaging of human or animal anatomy, collecting data from human or animal anatomy, and training medical or non-medical personnel. Additional example applications include procedures for handling tissue removed from human or animal anatomy (without returning it to the human or animal anatomy) and for performing procedures on human or animal cadavers. Furthermore, these techniques can also be used in surgical and non-surgical medical treatment or diagnostic procedures.

[0122] Furthermore, although some of the examples presented in this disclosure discuss robot-assisted systems or remotely operable systems, the disclosed techniques are also applicable to computer-assisted systems that are partially or wholly moved directly and manually by an operator.

[0123] Additionally, one or more elements of the examples of this disclosure can be implemented in software to execute on a processor of a computer system, such as a control processing system. When implemented in software, the elements of the examples of this disclosure are essentially code segments that perform necessary tasks. The program or code segment can be stored in a processor-readable storage medium (e.g., a non-transitory storage medium) or device, and can be downloaded via a transmission medium or communication link via computer data signals embodied in a carrier wave. Processor-readable storage devices can include any medium capable of storing information, including optical, semiconductor, and magnetic media. Examples of processor-readable storage devices include: electronic circuitry; semiconductor devices, semiconductor memory devices, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. The code segment can be downloaded via a computer network such as the Internet, intranets, etc. Any of a variety of centralized or distributed data processing architectures can be employed. Programming instructions can be implemented as a number of individual programs or subroutines, or they can be integrated into many other aspects of the system described herein. In some examples, the control system may support wireless communication protocols such as Bluetooth, Infrared Data Association (IrDA), HomeRF, IEEE 802.11, Digital Enhanced Cordless Telecommunications (DECT), Ultra Wideband (UWB), ZigBee, and wireless telemetry.

[0124] A computer is a machine that follows programmed instructions to perform mathematical or logical functions on input information to produce processed output information. A computer includes logic units that perform mathematical or logical functions, and memory that stores programming instructions, input information, and output information. The term "computer" is similar to similar terms such as "processor," "controller," or "control system."

[0125] Note that the processing and display presented may be independent of any particular computer or other device itself, and various systems can be used with programs in accordance with the teachings herein. The necessary structures for the various systems discussed above will appear as elements in the claims. Furthermore, the examples in this disclosure are not described with reference to any particular programming language. It will be understood that the teachings of this disclosure as described herein can be implemented using various programming languages.

[0126] While certain exemplary examples of this disclosure have been described and shown in the accompanying drawings, it should be understood that such examples are merely illustrative of the broad conception of the disclosure and are not intended to limit the broad conception of the disclosure. Furthermore, the examples of this disclosure are not limited to the specific constructions and arrangements shown and described, as various other modifications will be apparent to those skilled in the art.

Claims

1. A system comprising: Display system; Elongation device; A medical device configured to extend within the elongating device; as well as The control system, communicatively coupled to the display system, is configured to: Intraoperative imaging data of the patient's anatomy received from an external imaging system is displayed in a graphical user interface (GUI) on the display system. Displaying a graphic indicator superimposed on the intraoperative imaging data, wherein the graphic indicator indicates the target location of the distal tip of the medical device in the patient's anatomy; Register the intraoperative imaging data with the elongation device; Based on the registration, the graphic indicator is transformed from the reference frame of the intraoperative imaging data to the reference frame of the elongation device; and Based on the target location of the distal tip of the medical device as displayed via the graphical indicator, navigation guidance is provided to guide the elongation device to a target location within the patient's anatomy, wherein, when the elongation device is located at the target location, the target location of the distal tip of the medical device is within the deployment range of the medical device.

2. The system according to claim 1, wherein, A portion of the intraoperative imaging data corresponds to the elongation device.

3. The system according to claim 2, wherein, The control system is also configured to segment the portion of the intraoperative imaging data corresponding to the elongation device.

4. The system according to claim 2, wherein, Registering the intraoperative imaging data with the elongation device includes comparing shape data received from the elongation device with the shape of the elongation device in the portion of the intraoperative imaging data corresponding to the elongation device.

5. The system according to claim 1, wherein, The control system is also configured to display a treatment area covering at least a portion of the anatomical target, wherein the orientation of the treatment area is determined based on the positioning of the graphical indicator in the GUI.

6. The system according to claim 5, wherein, The control system is also configured to receive user input that updates the location of the graphical indicator in the GUI.

7. The system according to claim 6, wherein, The control system is also configured to update the treatment area when the position of the graphic indicator changes.

8. The system according to claim 5, wherein, The control system is also configured to display the margins surrounding or within the treatment area.

9. The system according to claim 5, wherein, The control system is also configured to display the margins surrounding the treatment area.

10. The system according to claim 5, wherein, The control system is also configured to display the margins surrounding the anatomical target, wherein the treatment area surrounds the margins.

11. The system according to claim 5, wherein, The control system is also configured to receive user input for adjusting one or more treatment parameters.

12. The system according to claim 11, wherein, The control system is also configured to update the treatment area based on received user input that adjusts one or more treatment parameters.

13. The system according to claim 1, wherein, Providing navigation guidance includes determining the distance from the distal tip of the extension device to the graphic indicator.

14. The system according to claim 1, wherein, Providing navigation guidance includes determining the distance from the distal tip of the elongation device to the anatomical target.

15. The system according to claim 1, wherein, Providing navigation guidance includes providing user instructions for engaging the extension device to align the longitudinal axis of the extension device with the graphic indicator.

16. The system according to claim 15, wherein, The longitudinal axis of the elongation device is aligned with the center of the graphic indicator.

17. The system according to claim 1, wherein, The intraoperative imaging data includes oblique views showing the patient's anatomy.

18. The system according to claim 1, wherein, The intraoperative imaging data displayed includes showing one or more key structures in the patient's anatomy that are very close to the anatomical target.

19. The system according to claim 1, wherein, The control system is also configured to receive additional intraoperative imaging data of the patient's anatomy from the external imaging system, wherein the additional intraoperative imaging data includes images of the medical device extending from the elongation device.

20. The system according to claim 19, wherein, The control system is also configured to receive user input that adjusts the position of the graphic indicator.

21. The system according to claim 20, wherein, The control system is also configured to display the graphic indicator at an updated position based on received user input that adjusts the position of the graphic indicator.

22. The system according to claim 19, wherein, The control system is also configured to receive user input for adjusting one or more treatment parameters.

23. The system according to claim 22, wherein, The control system is also configured to: Displays a treatment area covering at least a portion of the anatomical target, wherein the orientation of the treatment area is determined based on the positioning of the graphical indicator in the GUI; and The treatment area is updated based on received user input adjusting one or more treatment parameters.

24. The system according to claim 19, wherein, The control system is also configured to receive user input for adjusting the treatment area, wherein the treatment area covers at least a portion of the anatomical target, and wherein the orientation of the treatment area is determined based on the positioning of the graphic indicator.

25. A system comprising: Display system; Elongation device; A medical device configured to extend within the elongating device; as well as The control system is configured to: Intraoperative imaging data of the patient's anatomy received from an external imaging system is displayed in a graphical user interface (GUI) on the display system. The treatment information is displayed overlaid on the intraoperative imaging data, wherein the treatment information shows one or more treatment parameters for treating anatomical targets in the patient's anatomy. Register the intraoperative imaging data with the elongation device; Based on the registration, the treatment information is transformed from the reference frame of the intraoperative imaging data to the reference frame of the elongation device; and Based on the treatment information, navigation guidance is provided to guide the elongation device to a target location within the patient's anatomy.

26. The system according to claim 25, wherein, A portion of the intraoperative imaging data corresponds to the elongation device.

27. The system according to claim 26, wherein, The control system is also configured to segment the portion of the intraoperative imaging data corresponding to the elongation device.

28. The system according to claim 26, wherein, Registering the intraoperative imaging data with the elongation device includes comparing shape data received from the elongation device with the shape of the elongation device in the portion of the intraoperative imaging data corresponding to the elongation device.

29. The system according to claim 25, wherein, The treatment information includes a treatment area covering at least a portion of the anatomical target, and wherein the orientation of the treatment area is determined based on the one or more treatment parameters.

30. The system according to claim 29, wherein, The control system is also configured to update the treatment area when one or more treatment parameters change.

31. The system according to claim 29, wherein, The control system is also configured to display the margins surrounding or within the treatment area.

32. The system according to claim 29, wherein, The control system is also configured to display the margins surrounding the anatomical target, wherein the treatment area surrounds the margins.

33. The system according to claim 25, wherein, Providing navigation guidance includes determining the distance from the distal tip of the elongation device to the anatomical target.

34. The system according to claim 25, wherein, Providing navigation guidance includes providing user instructions for engaging the extension device to align its longitudinal axis with a graphic indicator, wherein the graphic indicator indicates the target location of the distal tip of the medical device within the patient's anatomy.

35. The system according to claim 25, wherein, The intraoperative imaging data displayed includes showing one or more key structures in the patient's anatomy that are very close to the anatomical target.

36. The system according to claim 25, wherein, The control system is also configured to receive additional intraoperative imaging data of the patient's anatomy from the external imaging system, wherein the additional intraoperative imaging data includes images of the medical device extending from the elongation device.

37. The system according to claim 36, wherein, The control system is also configured to receive user input for adjusting one or more treatment parameters.

38. The system according to claim 37, wherein, The control system is also configured to: Displays a treatment area covering at least a portion of the anatomical target, wherein the orientation of the treatment area is determined based on the positioning of the graphical indicator in the GUI; and The treatment area is updated based on received user input adjusting one or more treatment parameters.

39. The system according to claim 36, wherein, The control system is also configured to receive user input for adjusting the treatment area, wherein the treatment area covers at least a portion of the anatomical target, and wherein the orientation of the treatment area is determined based on the one or more treatment parameters.

40. A method comprising: Intraoperative imaging data of the patient's anatomy received from an external imaging system is displayed, wherein the intraoperative imaging data is displayed in a graphical user interface (GUI) on the display system; Displaying a graphic indicator superimposed on the intraoperative imaging data, wherein the graphic indicator indicates the target location of the distal tip of the medical device in the patient's anatomy, wherein the medical device is configured to extend within an elongation device; Register the intraoperative imaging data with the elongation device; Based on the registration, the graphic indicator is transformed from the reference frame of the intraoperative imaging data to the reference frame of the elongation device; and Based on the target location of the distal tip of the medical device as displayed via the graphical indicator, navigation guidance is provided to guide the elongation device to a target location within the patient's anatomy, wherein, when the elongation device is located at the target location, the target location of the distal tip of the medical device is within the deployment range of the medical device.

41. The method according to claim 40, wherein, A portion of the intraoperative imaging data corresponds to the elongation device.

42. The method of claim 41, further comprising segmenting the portion of the intraoperative imaging data corresponding to the elongation device.

43. The method according to claim 41, wherein, Registering the intraoperative imaging data with the elongation device includes comparing shape data received from the elongation device with the shape of the elongation device in the portion of the intraoperative imaging data corresponding to the elongation device.

44. The method of claim 40, further comprising displaying a treatment area covering at least a portion of the anatomical target, wherein, The orientation of the treatment area is determined based on the location of the graphical indicator in the GUI.

45. The method of claim 44, further comprising receiving user input that updates the location of the graphical indicator in the GUI.

46. ​​The method of claim 45, further comprising updating the treatment area when the positioning of the graphic indicator changes.

47. The method of claim 44, further comprising receiving user input for adjusting one or more treatment parameters.

48. The method of claim 47, further comprising updating the treatment area based on received user input adjusting the one or more treatment parameters.

49. The method of claim 44, further comprising displaying the margins surrounding or within the treatment area.

50. The method of claim 44, further comprising displaying the margins surrounding the anatomical target, wherein, The treatment area surrounds the margin.

51. The method according to claim 40, wherein, Providing navigation guidance includes determining the distance from the distal tip of the extension device to the graphic indicator.

52. The method according to claim 40, wherein, Providing navigation guidance includes determining the distance from the distal tip of the elongation device to the anatomical target.

53. The method according to claim 40, wherein, Providing navigation guidance includes providing user instructions for engaging the extension device to align the longitudinal axis of the extension device with the graphic indicator.

54. The method according to claim 40, wherein, The intraoperative imaging data displayed includes showing one or more key structures in the patient's anatomy that are very close to the anatomical target.

55. The method of claim 40, further comprising receiving additional intraoperative imaging data of the patient's anatomy from the external imaging system, wherein, The additional intraoperative imaging data includes images of the medical device extending from the elongation device.

56. The method of claim 55, further comprising receiving user input to adjust the position of the graphic indicator.

57. The method of claim 56, further comprising displaying the graphical indicator at an updated position based on received user input adjusting the position of the graphical indicator.

58. The method of claim 55, further comprising receiving user input for adjusting one or more treatment parameters.

59. The method of claim 58, further comprising: Displays a treatment area covering at least a portion of the anatomical target, wherein the orientation of the treatment area is determined based on the positioning of the graphical indicator in the GUI; and The treatment area is updated based on received user input adjusting one or more treatment parameters.

60. The method of claim 55, further comprising receiving user input for adjusting the treatment area, wherein, The treatment area covers at least a portion of the anatomical target, and the orientation of the treatment area is determined based on the positioning of the graphic indicator.

Images