Systems and methods for lymph node assessment

Abstract

A system for providing navigation guidance for a medical procedure may include a processor and a memory operatively coupled to the processor, the memory to store instructions that when executed by the processor cause the system to perform operations. These operations may include receiving an anatomical model of an anatomical region, performing an analysis on the anatomical model to facilitate selection of a plurality of junction sites for the analysis, generating a sequence of procedures for the plurality of junction sites, providing navigation guidance to guide a medical device to a junction site of the plurality of junction sites in the sequence of procedures, and collecting local image data from the medical device at the junction site.

Description

[0001] Cross-referenced applications

[0002] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 341,958, filed on May 13, 2022, entitled “Systems and Methods for Lymph Node Assessment,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to systems, methods, and computer program products for lymph node assessment and staging to determine nodal metastases. Background Art

[0004] Minimally invasive medical techniques are intended to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. This minimally invasive technique can be performed through a natural orifice in the patient's anatomy or through one or more surgical incisions. Through these natural orifices or incisions, an operator can insert a minimally invasive medical tool to reach the target tissue location. Minimally invasive medical tools include instruments such as treatment, diagnosis, biopsy, and surgical instruments. Medical tools can be inserted into anatomical pathways and navigated toward an area of ​​interest in the patient's anatomy. Images of anatomical pathways can be used to assist navigation. Improved systems and methods are needed to plan and safely perform lymph node assessment and staging to assess node metastasis. Summary of the invention

[0005] Embodiments of the invention are best summarized by the claims appended to the description.

[0006] In one example, a system for providing navigation guidance for a medical procedure may include a processor and a memory operably coupled to the processor, the memory for storing instructions that, when executed by the processor, cause the system to perform operations. The operations may include receiving an anatomical model of an anatomical region, performing analysis on the anatomical model to facilitate selecting a plurality of knot sites for analysis, generating a procedure sequence for the plurality of knot sites, providing navigation guidance to guide a medical device to a knot site in the plurality of knot sites in the procedure sequence, and collecting local image data from the medical device at the knot site.

[0007] It should be understood that the above general description and the following detailed description are illustrative in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, through the following detailed description, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Many aspects of the present disclosure may be better understood with reference to the following drawings. The components in the drawings are not necessarily drawn to scale. Instead, emphasis is placed on clearly illustrating the principles of the present disclosure. The drawings should not be considered to limit the present disclosure to the specific embodiments depicted, but are for explanation and understanding only.

[0009] Figure 1A-Figure 1C Lymph node locations in the chest area of ​​a patient are shown according to some examples.

[0010] Figure 2 is a flow chart illustrating a method for planning and / or performing a lymph node clinical intervention.

[0011] Figure 3A-3B , Figure 4A-4D , Figure 5A-Figure 5B , Figure 6A-6B , Figure 7A-7B , Figure 8 A graphical user interface for generating a procedure sequence for imaging and / or performing clinical intervention at multiple lymph nodes is shown.

[0012] Figure 9A-9B , Figure 10A-10B , Figure 11A-11B A graphical user interface for providing navigation guidance for navigating a medical instrument to a plurality of lymph nodes in a procedural sequence is shown.

[0013] Figures 12A-12D , Figure 13A-13B , Figure 14A-14B , Figure 15A-15B A graphical user interface including intraoperative image data for evaluating a plurality of lymph nodes is shown.

[0014] Fig.16A and Fig. 16B A graphical user interface is shown displaying clinical intervention records stored for a plurality of lymph nodes in a procedural sequence.

[0015] Fig.17 is a schematic representation of a manipulator assembly, a medical instrument system, and an imaging system configured in accordance with various embodiments of the present technology.

[0016] Fig.18 is extended within the anatomical region of the patient according to various embodiments of the present technology Fig.17 Schematic representation of a part of a medical device system. DETAILED DESCRIPTION

[0017] The present disclosure relates to devices, systems, methods and computer program products for lymph node assessment and staging, for example, to determine nodal metastasis. In some embodiments, for example, an assessment is performed on a cancer patient to determine the anatomical extent of the cancer ("cancer staging"). The cancer staging process may include determining the amount of cancer in the patient's body, the location of the cancer and / or whether the cancer has spread from its original site (e.g., to the lymphatic drainage system, vascular system, other organ systems, etc.). For example, in the context of lung cancer (e.g., non-small cell lung cancer), the staging process may include imaging and biopsy of multiple lymph nodes and / or lymph node measurement points (i.e., clinically defined lymph node groupings) near the airways of the lungs. Accurate staging is important for assessing patient prognosis and selecting appropriate treatment options. For example, in some cases, surgery may be recommended for early lung cancer, but surgery may be contraindicated for advanced lung cancer. However, operators performing biopsy procedures may not be aware of clinically recommended staging guidelines (e.g., which lymph nodes and / or lymph node sites should be biopsied), may not know how to apply the guidelines to a particular patient's pathology, and / or may not know how to effectively perform the biopsy procedure while adhering to the guidelines.

[0018] Thus, the present technology can help an operator plan and / or perform a lymph node assessment by (i) identifying which lymph nodes and / or lymph node sites (collectively referred to as "lymph node sites") should be biopsied, and (ii) determining a sequence for biopsy of the selected lymph node sites. In some embodiments, for example, a three-dimensional (3D) model of an anatomical region of a patient is generated (e.g., the model is generated from preoperative image data) and segmented into components representing anatomical structures (such as airways, lungs, lymph nodes, blood vessels, and / or target lesions). The segmented model can be used to select lymph node sites to be biopsied (e.g., based on the location of the target lesion, the location of the lymph node site, lymphatic drainage pathways, clinical staging guidance, etc.). The model can also be used to determine a sequence for biopsy of the selected lymph node sites in an efficient manner while minimizing the risk of cross contamination. During the biopsy procedure, the selected lymph node site and the biopsy sequence can be displayed to provide visual guidance to the operator and facilitate navigation within the patient's anatomy. Current technology holds promise for increasing operator compliance with clinical staging instructions and improving the efficiency and accuracy of the staging process, which may contribute to better patient outcomes.

[0019] The present technology generally relates to planning and / or performing medical procedures, such as biopsy procedures, for diagnosing a disease or condition in a patient. In some embodiments, for example, the systems described herein are configured to plan a biopsy procedure for staging lung cancer (e.g., non-small cell lung cancer). The staging of lung cancer can be defined as follows:

[0020] Stage 0: The cancer has not spread from its original site ("in situ disease").

[0021] Stage I: A small primary tumor in only one lung that has not spread to any lymph nodes and has not metastasized.

[0022] Stage II: A larger primary tumor that has not spread to any lymph nodes, or a smaller tumor in the lung that has spread to nearby lymph nodes.

[0023] Stage III: The cancer has spread to the lymph nodes in the middle of the lungs and the middle of the chest ("locally advanced disease"). There are two subtypes of stage III: IIIA (cancer has spread only to the lymph nodes on the same side of the chest where the cancer started) and IIIB (cancer has spread to the opposite side of the chest and / or to the lymph nodes above the collarbone).

[0024] Stage IV: Cancer has spread to both lungs, to the fluid in the area around the lungs, or to other parts of the body, such as the liver or other organs ("advanced disease").

[0025] As described above, accurate lung cancer staging may be important for assessing patient prognosis and / or determining appropriate treatment options. For example, surgery may be recommended for patients with stage 0 or stage I cancer; for patients with stage II cancer, treatment (e.g., chemotherapy, radiotherapy, chemoradiotherapy, immunotherapy) may be recommended first, followed by surgery; for patients with stage III or stage IV cancer, treatment (e.g., chemotherapy, radiotherapy, chemoradiotherapy, immunotherapy) may be recommended without surgery.

[0026] In some embodiments, lung cancer staging involves obtaining tissue samples from one or more nodal sites within the chest region of a patient. As described above, the presence of cancer cells at certain nodal sites (e.g., lymph node sites in the middle of the chest, lymph node sites on the side of the chest opposite the original cancer site, etc.) can be associated with a more advanced stage of the cancer. Thus, the extent and severity of the cancer can be assessed by systematically sampling lymph nodes from different lymph node sites in the chest region.

[0027] Figure 1A-Figure 1C 1 shows lymph node measurement points in the chest area 100 of a patient. Figure 1A-Figure 1C As shown in Table 1 below, the lymph nodes in the chest region 100 can be grouped into 14 different lymph node measurement points (measurement points 1R-14). The lymph node measurement points can be grouped into 7 anatomical regions (102-114, Figure 1A-Figure 1C (shown by the dashed line).

[0028] Table 1: Chest lymph node measurement points

[0029]

[0030]

[0031] As described in more detail below, the systems described herein may be configured to select Figure 1A-Figure 1C and one or more of the nodal sites listed in Table 1 for biopsy during a medical procedure used to stage lung cancer.

[0032] Figure 2 200 is a flowchart illustrating a method 200 for planning and / or performing a clinical intervention for a lymph node (such as a biopsy procedure) according to various embodiments of the present technology. The method 200 is shown as a set of steps or processes 202-214. All or a subset of the steps of the method 200 may be implemented by a computing system or device, such as a workstation configured to perform preoperative planning of a medical procedure. Alternatively or in combination, all or a subset of the steps of the method 200 may be implemented by a control system of a medical device system or device, including various components or devices of a robot or remote control system, as described in more detail below. In some embodiments, one or more of the processes may be implemented at least in part in the form of executable code stored on a non-transient tangible machine-readable medium, which executable code may cause one or more processors to perform one or more of the processes when run by one or more processors (e.g., a processing unit of a control system). The methods described herein are shown as a set of operations or processes and continue to be described with reference to the attached drawings. Not all of the illustrated processes may be performed in all embodiments of the method. In addition, one or more processes not explicitly shown may be included before, after, between, or as part of the illustrated process.

[0033] At process 202, an anatomical model of an anatomical region may be received. Receiving the anatomical model may include generating the anatomical model, retrieving a stored anatomical model, obtaining the anatomical model from a network source, or otherwise acquiring the anatomical model. Process 202 may include receiving image data of an anatomical region of a patient. The image data may include, for example, computed tomography (CT) data, magnetic resonance imaging (MRI) data, fluoroscopic data, thermal imaging data, ultrasound data, optical coherence tomography (OCT) data, thermal image data, impedance data, laser image data, nanotube X-ray image data, and / or other suitable data representing the anatomical region in which a biopsy procedure is to be performed. The image data may correspond to a two-dimensional (2D), 3D, or four-dimensional (e.g., time-based or velocity-based) image. For example, in some embodiments, the image data includes 2D images from multiple perspectives that may be combined into a pseudo-3D image. The image data may be preoperative image data obtained before a biopsy procedure is performed on a patient. A 3D model of an anatomical region may be generated by segmenting the image data. The model can represent the anatomical region where the biopsy procedure is to be performed (e.g., the airways of a patient's lungs), and can represent the location, shape, and connectivity of pathways and other structures (e.g., lymph nodes, target lesions, blood vessels, etc.) within the region. In some embodiments, the model includes a plurality of segmented components corresponding to anatomical structures or features within the anatomical region. Examples of anatomical structures or features that may be included in the model include one or more of the following: airways (e.g., trachea, aorta, left mainstem bronchus, right mainstem bronchus, and / or subsegmental bronchi), lymph nodes (e.g., Figure 1A-Figure 1C and any nodal sites described in Table 1), blood vessels (e.g., aorta, superior vena cava, pulmonary trunk), lungs, and / or target lesions (e.g., tumors or other tissue sites known or suspected to be cancerous).

[0034] A 3D model can be generated by segmenting graphical elements that represent or otherwise correspond to anatomical structures or features in image data. During the segmentation process, pixels or voxels generated from the image data can be segmented into segments or elements, and / or marked to indicate that they share certain characteristics or computational properties, such as color, density, intensity, and texture. The segments or elements associated with the patient's anatomical features are then converted into segmented anatomical models generated in a model or image reference system. In order to represent the model, the segmentation process can outline a set of voxels representing an anatomical region, and then apply a function such as a marching cube function to generate a 3D surface surrounding the voxels. The model can be made by generating a grid, volume, or voxel map. Additionally or alternatively, the model can include a centerline model that includes a set of interconnected line segments or points that extend through the center of the modeling path. In the case where the model includes a centerline model containing a set of interconnected line segments, these line segments can be converted into a cloud or a set of points. By converting the line segments, the desired number of points corresponding to the interconnected line segments can be selected manually or automatically. Various systems and methods for segmenting anatomical structures from image data are described in more detail in U.S. Patent Application Publication No. 2020 / 0030044 (filed on April 18, 2018) (disclosing a graphical user interface for a planning procedure); and U.S. Patent No. 10,373,719 (filed on September 3, 2015) (disclosing systems and methods for preoperative modeling); the entire contents of both are incorporated herein by reference.

[0035] At process 204, analysis can be performed on the anatomical model. Analysis of the anatomical model can facilitate selection of multiple node locations for analysis. Optionally, process 204 can include one or more of processes 204a-204f. At process 204a, multiple node measurement points can be anatomically located on the anatomical model. As described above, the lymph nodes of the chest region 100 can be grouped into 14 different lymph node measurement points. The anatomical model can be segmented to correspond to the lymph node measurement points. Reference Figure 3A In some examples, a branched anatomical model 300 including a target lesion 309 (e.g., a suspected lung tumor) can be displayed in a graphical user interface 301 on a display system 302. Lymph node locations can be indicated graphically on the user interface 301 (such as by color, shading, borders, alphanumeric text, or other graphical indicators of different node locations in the anatomical region). Figure 3AIn the example of FIG. 3 , measuring point 1 (303), measuring point 2 (304), measuring point 4 (305), measuring point 10 (306), and measuring point 7 (307) can be displayed as shaded areas near the anatomical model 300. The user interface 301 can allow a user to visualize the approximate location of the measuring points relative to the anatomical model. Interactive elements 308 (such as menus, selectable elements, or information tabs associated with one or more of the displayed measuring points 303-307) can be displayed in the user interface 301.

[0036] Figure 3B Another example of a user interface 301 that may be used is provided. Figure 3B In the example of FIG. 4 , a branched anatomical model 400 including a target lesion 409 (e.g., a suspected lung tumor) can be displayed in a graphical user interface 301 on a display system 302. Lymph node locations can be indicated graphically on the user interface 301 (such as by color, texture, shading, borders, alphanumeric text, or other graphical indicators of different node locations in the anatomical region). Figure 3B In the example of , measuring point 2 (403) can be indicated by a dark blue area. Measuring point 4 (404) can be indicated by a red area. Measuring point 7 (406) can be indicated by a light blue area. Measuring point 10 (408) can be indicated by a yellow area. Further distinction of measuring point 10 can be provided by indicating the measuring point number and a text label (411) on the right or left side. Measuring point 11 (410) can be indicated by a green area. User interface 301 can allow a user to visualize the approximate location of the measuring points relative to the anatomical model. Interactive elements 308 (such as menus, selectable elements, or information tabs associated with one or more of the displayed measuring points 403-410) can be displayed in user interface 301 and can be color coded to match the colored areas of measuring points 403-440.

[0037] At process 204b, a plurality of lymph nodes may be identified in the anatomical model. Process 204b may include segmenting the plurality of lymph nodes in the image data. Segmentation of the lymph nodes may include, for example, analyzing the image data to identify graphical elements corresponding to the lymph nodes (rather than corresponding to other anatomical structures such as airways, blood vessels, etc.). In some embodiments, the lymph nodes are identified based on characteristics such as shape (e.g., oval or circular, not tubular), size (e.g., approximately 1 cm in diameter), and / or location (e.g., near an airway and / or within an anatomical region corresponding to a lymph node measurement point). Once identified, the lymph nodes may be segmented into individual model components as described above.

[0038] The lymph node segmentation procedure can be performed in various ways, such as automatically (e.g., without requiring any operator input to identify lymph nodes), semi-automatically (e.g., with some operator input), or manually by an operator. For example, automatic lymph node segmentation can be performed using a machine learning algorithm, such as a deep learning algorithm (e.g., a convolutional neural network or other type of neural network) that is trained to identify and segment individual lymph nodes from CT scans or other image data of a patient's anatomy. Training of the machine learning algorithm can be performed, for example, via supervised learning techniques using large image data sets in which lymph nodes have been identified. Once trained, the machine learning algorithm can automatically identify graphical elements in the image data that may correspond to lymph nodes, and can segment these graphical elements to create individual model components, as described above. Optionally, the machine learning algorithm can also be trained to automatically identify and segment other anatomical structures (e.g., airways, lesions, blood vessels, etc.).

[0039] The semi-automatic lymph node segmentation process may involve some steps that are automatically performed and some steps that are performed based on input from an operator. For example, an operator may select one or more locations in the image data that include lymph nodes and / or correspond to lymph node points, and a computing system may analyze the selected locations to identify and segment the lymph nodes at these locations. In some embodiments, the operator provides input indicating the selected location (e.g., via a suitable graphical user interface), such as by clicking or otherwise marking a point corresponding to a lymph node, drawing a border around the edge or surface of a lymph node, selecting an area of ​​the image data that includes lymph nodes and / or lymph node points, or any other suitable process. The system may then use the input from the operator as a starting point to automatically detect one or more lymph nodes in the image data. For example, the system may use an edge detection algorithm, a machine learning algorithm, or the like to search for objects that may correspond to lymph nodes in the image locations indicated by the operator. The results may be displayed to the operator for approval, rejection, modification, or other feedback. Alternatively or in combination, the system may automatically provide an initial selection of potential lymph nodes, and the operator may accept, reject, or modify these selections. Alternatively, if identifying individual lymph nodes is challenging (e.g., due to low signal-to-noise ratios in the image data), the system and / or operator may instead identify locations in the image data that correspond to lymph node measurements (e.g., as described above with respect to Figure 1A-Figure 1C Rather than identifying and segmenting individual lymph nodes, the present invention can identify and segment lymph nodes based on the lymph node measurement points and / or anatomical regions described by the present invention.

[0040] At process 204c, one, more, or all of the identified lymph nodes may be associated with one of a plurality of anatomical points. In some examples, the identified lymph nodes may be displayed as part of the anatomical model 300, thereby providing a user with a visual indication of the nodes and their associated points. For example, an image of a segmented lymph node may be overlaid on an image of the anatomical points of the anatomical model. Figure 4A Each of the segmented lymph nodes can be assigned to and overlaid on a lymph node measurement point (e.g., based on the location of the lymph node relative to other anatomical structures). Figure 4A As shown in the user interface 301 of FIG. 1 , knot 01 can be assigned to measuring point 1 ( 303 ). Knot 02 and knot 03 can be assigned to measuring point 2 ( 304 ). Knot 04 and knot 05 can be assigned to measuring point 4 ( 305 ). Other knots can be assigned to measuring points in a similar manner. Figure 4A As shown, the interactive element 308 may include a displayed list view of nodes grouped by the assigned measurement points. The list view may include the size and / or shape of the listed lymph nodes. The user interface 301 may include segmented nodes (e.g., node 01-node 05) overlaid on corresponding measurement points (e.g., measurement points 1, 2, and 4).

[0041] Figure 4B Another example of a user interface 301 that may be used is provided. Figure 4B In the example of FIG. 301 , knots 02 and 03 may be assigned to point 2 ( 304 ). Knots 04-09 may be assigned to point 4 ( 305 ). Interactive elements 308 (e.g., menus, selectable elements, or information tabs associated with one or more of the displayed points 304-305 ) may be displayed in user interface 301 , and the knots listed in interactive elements 308 may be color coded or shaded to match the knots displayed with the model.

[0042] Figure 4C Another example of a user interface 301 that may be used is provided. Figure 4C In the example of , a segmented knot can be displayed with characteristics derived from image data (e.g., CT data). For example, knot 04 can be displayed with a model having a size (e.g., short axis and / or long axis) and / or shape (e.g., roundness) determined from the CT image data. Further details of the derived characteristics can be displayed with the knot information in interactive element 308. For example, the size of the long and short axes of knot 04, a measure of the roundness of knot 04, and a PET standardized uptake value (SUV) can be displayed. Figure 4D Provide with Figure 4CSame user interface 301, but in this example, selected features of the knots can be highlighted. The selected features can be selected by the clinician, or can be selected by the control system based on importance to the procedure or other criteria. In this example, knots 04 and 08 displayed with the model can be highlighted with color, brightness, or shading. Text describing the characteristics of interest in interactive element 308 can also be highlighted with a color, brightness, or shading corresponding to the displayed knots. In this example, the roundness and PET SUV of knots 04 and 08 can be highlighted in interactive element 308. In some examples, these knots can be highlighted if a staging threshold is met. As shown in FIG. Figure 4D As shown, if the staging threshold is met, the threshold element 323 can be highlighted with the corresponding node. For example, the staging threshold for the short axis can be a size greater than 5 mm. The staging threshold for the long axis can be a size less than 10 mm. The staging threshold for roundness can be greater than 0.7. The staging threshold for PET SUV can be greater than 1.2. Figure 4D In the example shown, knot 04 and knot 08 may be highlighted because their roundness exceeds 0.7 and their PET SUV exceeds 1.2. On the other hand, knot 07 may not be highlighted because its roundness and PET SUV do not exceed the respective thresholds. As another example, if the user or the control system selects only the short axis threshold, knot 04, knot 07, and knot 08 may be highlighted in the GUI because their short axes exceed the short axis threshold.

[0043] At process 204d, the anatomical model data can be enhanced with supplemental imaging data. In some examples, at process 204d, performing analysis on the anatomical model can include overlaying imaging data (such as positron emission tomography (PET) imaging data) on the displayed anatomical model and lymph nodes. For example, supplemental imaging data in the form of fluorodeoxyglucose (FDG)-PET imaging data can be generated for relevant regions of the patient's anatomy. The FDG-PET imaging data can be used to enhance the anatomical model data to provide additional information about metabolically active malignant regions in the imaged anatomical region. In some examples, the supplemental imaging data can be displayed as an overlay or integrated with the anatomical model. For example, referring to Figure 5A , regions 310 of high FDG affinity can be displayed on user interface 301 as an overlay of anatomical model 300. Regions of high FDG affinity can be associated with measurement points or specific junctions with measurement points. For example, interactive element 308 can provide information indicating that measurement points 4 and 11 are associated with activity in the supplemental imaging data that may indicate malignancy, such as highlighted element 311. Figure 5BAnother example of a user interface 301 that may be used is provided. In this example, three areas 310 of high FDG affinity may be displayed on the user interface 301 as an overlay of the anatomical model 300. The interactive element 308 may provide information, such as highlighted element 311, indicating that the measurement points 4, 7, and 11 are associated with activity in the supplemental imaging data that may indicate malignancy.

[0044] At process 204e, critical or fragile anatomy can be identified from the anatomical image data. In some examples, at process 204e, performing analysis on the anatomical model can also include displaying critical anatomy located near at least one node site. Such critical or fragile tissue can include the pulmonary pleura, vasculature (e.g., aorta, pulmonary artery, pulmonary vein), heart, and / or bullae in areas that should be avoided during interventional procedures to prevent rupture or tearing. In some examples, critical or fragile anatomy can be segmented out from the anatomical image data and displayed as part of the anatomical model. For example, referring to Fig. 6A , anatomy 312 can be displayed on user interface 301 along with anatomical model 300. Displayed anatomy 312 can include graphical elements representing various anatomical structures, such as vasculature (e.g., ascending aorta-descending aorta, superior vena cava, pulmonary artery, pulmonary vein). The above process can be used to automatically, semi-automatically, or manually segment image data to create a 3D model of the chest region. In some examples, display of anatomy 312 can be turned on or off depending on the assessed stage and the usefulness of the information to the observer. Information 313 about anatomy 312 can also be displayed on user interface 301. Figure 6B Another example of a user interface 301 that can be used is provided. In this example, the anatomy 312 can be color-coded. For example, arteries can be color-coded with red, while veins can be color-coded with blue.

[0045] At process 204f, a node site may be selected for analysis. A node site may be an area including the location of a lymph node, and may include an area close to a location of a lymph node (including a location close to a location of a lymph node in an anatomical pathway). As described above, the selected node site may be imaged, biopsied, or otherwise analyzed to determine the stage of the patient's cancer, for example, by assessing whether the cancer has spread from an initial site (e.g., target lesion 309) to the node site. For example, if the malignant cells of a node site located near the target lesion test negative, the cancer may be an early-stage cancer. Conversely, if the malignant cells of a node site located away from the target test positive, the cancer may be an advanced-stage cancer. The number of positive node sites may also be related to the extent of the cancer, for example, if few or no node sites that have been biopsied test positive, the cancer may be early-stage, and if most or all of the node sites that have been biopsied test positive, the cancer may be advanced. Therefore, process 204f may involve selecting which node sites should be biopsied in order to accurately stage the patient's cancer. Process 204f may involve selecting one or more individual lymph nodes, one or more lymph node measurement points, or a combination thereof. In some embodiments, for example, one or more lymph node sites are selected without specifying any specific lymph nodes to be biopsied within the site. In other embodiments, one or more specific lymph nodes may be selected with or without specifying corresponding lymph node sites.

[0046] The selection of nodal sites can be based on a particular patient's pathology, such as the location of a target lesion within the patient's body. For example, process 204f can include selecting one or more nodal sites (e.g., sentinel lymph nodes) that are located downstream along a lymphatic drainage path of a target lesion. The drainage path within an anatomical region of a patient can be determined based on clinical guidance or research, the patient's specific anatomy and physiology, and / or any other appropriate considerations. For example, in the thoracic region, a drainage path may generally run from a lymph node site in a peripheral portion of the lungs to a lymph node site in the mediastinum between the lungs. The drainage path generally does not cross a lung lobe or reach the other side of a peripheral lung. Thus, if a target lesion is located in a peripheral portion of the lungs (e.g., near point 13R— Figure 1A ), process 204f may involve selecting some or all of the nodal sites (eg, sites 12R, 11R, and 10R— Figure 1A ) and mediastinal node sites (e.g., points 4R and 7— Figure 1A ). Optionally, the node sites upstream of the target lesion may be omitted (eg, measurement site 14R - Figure 1A ) and / or nodal sites that are not located along the drainage path from the target lesion (e.g., measurement points 14L, 13L, 12L— Figure 1A). The drainage path may vary to some extent from patient to patient and may skip lymph node sites and / or pass between lymph node sites. Thus, the selection process 204f may be customized to each patient's specific anatomy and / or other clinical considerations.

[0047] Alternatively or in combination, a node site may be selected based on its location, such as its proximity to a target lesion (e.g., a lung tumor). Proximity may be assessed quantitatively (e.g., based on a measured distance between the target lesion and the node site) and / or qualitatively (e.g., whether the node site is on the same side of the chest as the target lesion, in the middle of the chest, or on the opposite side of the chest from the target lesion). In some embodiments, process 204f may involve selecting node sites at different proximity to the target lesion. For example, the selection may include at least one node site on the same side of the chest as the target lesion (ipsilateral or "N1 node"), at least one node site in the middle of the chest ("N2 node"), and at least one node site on the opposite side of the chest from the target lesion (contralateral or "N3 node"). For example, if the target lesion is near measuring point 13R ( Figure 1A ), then step 230 may involve selecting ipsilateral node sites (e.g., measuring points 12R, 11R, 10R, and 4R— Figure 1A ) and / or contralateral nodal sites (e.g., 12L, 11L, 10L, and 4L— Figure 1A ). Optionally, process 204f may involve selecting more nodal sites that are close to the target lesion (e.g., N1 nodes) and fewer nodal sites that are far from the target lesion (e.g., N2 nodes, N3 nodes). Nodal sites that are very far from the target lesion or are unlikely to have metastases may be excluded.

[0048] In some examples, nodal sites can be selected based on predictive modeling. For example, predictive modeling can be used to predict which nodal sites are likely to have metastases based on the location of the target lesion, and can be performed using a statistical model, a machine learning model, or any other suitable technique. The predictive model can generate a risk score for each nodal site that represents the probability that the cancer has metastasized to a particular site. Nodal sites associated with higher risk scores can be selected for biopsy, while nodal sites associated with lower risk scores can be excluded.

[0049] Alternatively or in combination, the nodal site can be selected based on other parameters, such as one or more of the following: accessibility to imaging or biopsy equipment (e.g., lymph nodes located near airways that are too narrow, too tortuous, or inaccessible to biopsy equipment can be excluded), lymph node size (e.g., lymph nodes with a diameter greater than 1 cm and / or a short axis diameter greater than 5 mm can be selected), lymph node shape (e.g., lymph nodes with abnormal shapes can be selected), proximity to fragile anatomical structures (e.g., nodal sites that are too close to major blood vessels, pulmonary pleura, bullae, etc. can be excluded), spatial relationships between nodal sites and other anatomical structures (e.g., airways, lungs, lung nodules), patient-specific physiology, clinical guidance or research (e.g., related to cancer staging procedures), etc.

[0050] Any suitable number and combination of nodal sites may be selected. In some embodiments, process 204f involves selecting at least one, two, three, four, five, or more different nodal sites to be biopsied (e.g., at least one, two, three, four, five, or more different lymph node sites). Alternatively, process 204f may involve selecting a certain number of lymph nodes (e.g., at least one, two, three, or more lymph nodes) for each site to be biopsied. In some embodiments, certain nodal sites are always selected, such as mediastinal lymph nodes (e.g., some or all of lymph node sites 2L, 2R, 3A, 3P, 4L, 4R, 8L, 8R, 9L, 9R— Figure 1A ).

[0051] The selection of node positions can be performed automatically, semi-automatically or manually. For example, the system can automatically analyze the location of the target lesion, lymph nodes and / or other segmented anatomical structures in the 3D model, and apply any of the above-mentioned selection parameters to select a subset of lymph nodes for biopsy. The selection parameters can be determined by the system (for example, encoded in the system software), or can be manually set by the operator (for example, the operator can select which selection parameters should be applied). Once the system has selected the node position, the selection can be output to the operator for approval, rejection or modification. Optionally, the operator can provide user input, which indicates which node positions should be biopsied (for example, indicated via a graphical user interface). For example, the operator can manually select certain anatomical areas or regions, and the system can then identify and select the node positions within these regions. The operator can also manually select a specific node position to be biopsied.

[0052] At process 208, a program sequence for analyzing the selected node position can be determined. The sequence can indicate the order in which the selected node position should be imaged and / or biopsied during the procedure. The order of the nodes and the path through the branch anatomical region can be determined based on the position of the lesion and / or lymph node, the direction of lymph flow, the constraints of the navigation instrument (e.g., bending radius restrictions) and / or the travel efficiency (e.g., the distance between consecutive positions and avoiding the use of instruments to backtrack). In some embodiments, the sequence is configured to reduce the possibility of cross contamination between the node positions (e.g., malignant cells are delivered to non-cancerous lymph nodes). Therefore, the sequence can include a node position that is less likely to be positive for malignancy (e.g., a position located near and / or downstream of the target lesion) before a node position that is more likely to be positive (e.g., a position located near and / or downstream of the target lesion) is biopsied. The biopsy sequence can also include sampling of the peripheral node position before the central node position.

[0053] In some embodiments, the sequence is based at least in part on a trajectory for navigating a medical device (e.g., an imaging device or a biopsy device) toward a selected plurality of nodes and / or target lesions. The trajectory can be a predetermined route for traversing an anatomical pathway to reach a target lesion, and can be automatically, semi-automatically, or manually generated during preoperative planning of the procedure. In such an embodiment, if the trajectory passes near one or more of the selected node sites, imaging can be performed or biopsy samples can be collected from these sites in the order in which they are encountered (e.g., as the biopsy device moves toward the target lesion). Alternatively or in combination, the sequence can also be determined based on any of the following considerations: reducing backtracking, reducing the total distance traversed by the biopsy device, reducing the total time of the biopsy procedure, avoiding pathways that are inaccessible or otherwise difficult to navigate for the biopsy device, and / or avoiding passing the biopsy device through areas close to fragile anatomical structures (e.g., major blood vessels, lung pleura, bullae).

[0054] In some examples, process 208 may also include generating at least one proposed path for navigating a biopsy device within an anatomical region to reach each of the sequence knot sites. The path may be configured to traverse the anatomical pathways between the sequence knot sites so that the medical device reaches these sites in the correct order only by following the path. Optionally, the path may also include a route for navigating the medical device to the target lesion. The path may be generated in various ways, such as automatically, semi-automatically, or manually. For example, an operator may manually create some or all of the paths by selecting a pathway (e.g., an airway) in a model via a suitable graphical user interface. Alternatively or in combination, the system may automatically generate some or all of the paths. For example, the system may use a model to identify and select a pathway (e.g., having a sufficiently large diameter) that is located near the selected knot site and accessible to the biopsy device. In some embodiments, the system automatically generates a proposed path, and the operator may approve the path or manually modify the path (e.g., by adding, deleting, or otherwise modifying some parts of the path). Conversely, an operator may manually create a path, and the system may automatically modify the path or propose modifications for operator approval.

[0055] refer to Fig. 7A , the user interface 301 can show an ordered sequence of nodes AF and a navigation path 320 that traverses anatomical pathways extending near these sequence nodes. The sequence nodes AF along the path 320 can be assigned to regions such as region N1, region N2, and / or region N3. These regions can be associated with distances from the target lesion 309. For example, region N1 can be the region closest to the lesion 309, and region N3 can be farthest from the lesion 309. The information panel 324 can provide information about the sequence knots, including the regions and measurement points to which the knots are assigned. For example, node A can be located in region N3, node B and node C can be located in region N2, and node D and node E can be located in region N1. In this example, paths 319 or path segments that have not yet been traversed can be indicated with a different color, dashed pattern, or other distinguishing feature than the paths 317 that have already been traversed. For example, as Fig. 7A As shown, a first portion of the path between nodes A and B can be a solid line indicating the portion of the path that the medical device has already reached, and a second portion of the path between nodes B and F can be a dashed line indicating the portion of the navigable path that the medical device has not yet reached. Figure 7B Another example of a user interface 301 that may be used is provided. Figure 8 In some examples, user interface 301 may provide sequence images 330 - 334 showing step-by-step navigation paths between sequence nodes.

[0056] Optionally, the navigation path may include imaging locations near each sequence knot site.The imaging locations may provide a preferred vantage point from which to obtain intraoperative images of the selected knot site (eg, using ultrasound imaging techniques).

[0057] Optionally, the navigation path may include an exit location near each selected knot site. The exit location may correspond to the point where the biopsy device leaves the passage to reach the knot site (e.g., by piercing the lumen of the passage at the exit location). For example, the exit location may be the point closest to the site in the passage. The path may also include a path segment (referred to herein as an "exit segment") connecting the exit location to the knot site. The length of the exit segment may be configured to be less than or equal to the maximum insertion depth of the biopsy device. For example, some biopsy needles may not be able to perform a biopsy on a target that is more than 3 cm away from the exit location.

[0058] In some embodiments, the navigation path is configured to avoid one or more fragile anatomical structures, such as blood vessels, lung pleura, bullae, etc. For example, puncturing the lung pleura during a biopsy procedure may cause pneumothorax and / or other dangerous conditions for the patient. Therefore, the path, exit location, and / or exit segment can be constrained to avoid fragile anatomical structures. For example, this can be achieved by defining one or more danger fences around fragile anatomical structures, which are used to indicate locations where the path cannot touch and / or overlap. Danger fences can be created automatically, semi-automatically, or manually by an operator. Additional techniques for creating paths within anatomical regions are described in more detail in U.S. Patent Application Publication No. 2020 / 0030044 (filed on April 18, 2018) (disclosing a graphical user interface for a planning program), the entire contents of which are incorporated herein by reference. An image representing the position and shape of the traversed medical device 356 can also be displayed on the user interface 301.

[0059] The output of process 208 (e.g., 3D model, selected node site, biopsy sequence and / or path) can be saved as part of the plan for the biopsy procedure (e.g., saved as one or more digital files). In embodiments where the plan is created on a preoperative planning workstation, the plan can be transferred to a medical device system that will be used to perform an intraoperative assessment of the selected node site, which may include local imaging and / or a biopsy procedure. After planning, during intraoperative imaging and biopsy procedures, the 3D model, selected node site and / or biopsy sequence can be displayed to the operator (e.g., via a graphical user interface) to provide visual guidance and instructions for navigating to the selected biopsy site, which is described in more detail below.

[0060] At process 210, navigation guidance may be provided to guide a medical instrument to a node location in a procedural sequence. A node location may be an area including the location of a lymph node in a lymph node sequence, and may include an area close to a location of a lymph node (including a location close to a location of a lymph node in an anatomical pathway). In some examples, the instrument may be a medical instrument for performing intraoperative imaging in the region of a first sequence node, a last sequence node, or any node between a first sequence node and a last sequence node. In some examples, the instrument may be a medical instrument for performing clinical intervention (such as a biopsy) on a first sequence node. In some examples, the instrument may be configured to perform multiple functions, including intraoperative imaging and biopsy. For example, the medical instrument may be an endobronchial ultrasound / transbronchial fine needle aspiration (EBUS-TBNA) instrument that may perform both ultrasound imaging at a first location and a biopsy on the first lymph node in a sequence.

[0061] Optionally, process 210 may include one or more of processes 210a-210d. At process 210a and with reference to Fig. 9A , providing navigation guidance may include displaying an image of an anatomical model. For example, display system 302 may display at least a portion of anatomical model 300. For example, anatomical model 300 may be displayed in graphical user interface 350 in a first pane of display system 302. Figure 4A and Figure 4B As shown, providing navigation guidance may also include displaying an image of a segmented knot on top of a knot measurement point and / or displaying interactive elements 308 (including a list view). Fig. 9A The navigation guidance may also or alternatively include an indicator indicating whether the node site is to be sampled by the medical device (e.g. Fig.11A 380), an indicator indicating whether a knot site has been sampled by a medical instrument (e.g., a check mark indicator 358), and / or an indicator indicating whether a knot site has not been sampled by a medical instrument (e.g., a dashed mark indicator 360). The navigation guidance can be registered and transformed for display in any of a variety of displayed views. For example, any of the disclosed navigation guidance can be provided with a view of a three-dimensional image of an anatomical model (e.g., model 300), a virtual endoscopic image (e.g., virtual endoscopic image 364), and / or a real-time endoscopic image (e.g., real-time endoscopic image 366).

[0062] At process 210b and with further reference to Fig. 9A, providing navigation guidance may include displaying a representation of a navigable path of a program sequence through multiple knot positions. For example, the user interface 350 may display a navigable path 320 between sequence knots. The user interface 350 may also include a representation of a medical device 356 along the path 320. The user interface 350 may also include an information panel 321 that provides information about sequence knots (including the regions and measurement points to which the knots are assigned). During navigation, the information panel 321 may also provide information about the status of the program. For example, a check mark 358 located next to a listed knot may indicate that a program (e.g., an imaging program and / or a biopsy program) has been completed at the knot. As another example, a dotted mark 360 located next to a listed knot may indicate that the knot is omitted from the sequence and the program is not performed. As another example, a dot mark 362 may indicate a knot in the sequence that has not yet been encountered by the instrument 356. In some examples, the graphical user interface 352 in the second pane of the display 302 may display a virtual endoscopic image 364 (e.g., a virtual fly-through image) of the model 300. Image 364 can show the position and path 320 of knot E that anatomical wall 365, instrument 356 are advanced toward it.Display 302 can also show real-time endoscopic image view 366 obtained by medical instrument 356 in anatomical passage.Display 302 can also show navigation panel 368, and this navigation panel shows the information about the distance from the far end of instrument 356 to knot E.Display system 302 can be identical with the display system for planning process 202-208, or can be the different display systems used by the clinician of execution clinical procedure.The graphic information shown in user interface 350,352 can be displayed in one or more display panes on one or more display screens with any configuration.Some in graphic information can be selectively displayed based on user preference. Fig. 9B Another example of a user interface 350, 352 that may be used is provided.

[0063] At process 210c and with reference to Fig. 10A , providing navigation guidance may include displaying representations of key anatomy along the navigable path. For example, an outline, semi-transparent illustration, or other graphical representation of a critical or vulnerable anatomy 312 may be displayed in any of the user interface panels 350, 364, 366 along with the model 300 and the navigable path 320. The display of the representation of the anatomy 312 may be turned on or off to help the navigating clinician understand key structures near each node in the sequence, such as the vasculature. Navigation guidance may be provided by a user interface orientation element 370 in the user interface 352, which may include a pathway wall 365, an instrument 356, a representation of the key anatomy 312, and / or a direction indicator 372 of an imaging element (e.g., an ultrasound image transducer) carried by the instrument 356. Fig. 10BAnother example of a user interface 350, 352 that may be used is provided.

[0064] At process 210d, providing navigation guidance may include displaying a representation of a knot marker adjacent to one or more of the plurality of knot sites. The knot marker may be displayed with the anatomical model. The knot marker may be used to specify identification information about the associated knot. For example, the knot marker may indicate that the knot has been imaged by a medical device. Additionally or alternatively, the knot marker may indicate that the knot should be biopsied based on an assessment of one or more information sources (e.g., segmentation information, intraoperative ultrasound imaging, supplemental PET imaging). For example, reference Fig.11A , knots in the sequence (e.g., knot A, knot B, knot E) can be marked to indicate that they should be biopsied. Panel 321 can also include knot markers 380 associated with knot information, which knot information has been generated based on, for example, intraoperative imaging (e.g., size and composition information), clinical intervention on the knot (e.g., biopsy information), graphic segmentation (e.g., size and distance information), and / or supplemental imaging (e.g., FDG affinity information). For example, the knot information for the marked knot A includes the diameter of the knot (11 mm) and whether the knot is FDG-affinity (yes). Fig. 11B Another example of a user interface 350, 352 that may be used is provided. Characteristics of the knot marking (such as presence or absence, color, or orientation) may provide further indication that the knot is intended to be sampled.

[0065] At process 212, local image data can be collected from a medical device in an anatomical pathway located near a first node site. For example, the medical device can be an EBUS-TBNA device that is used to collect intraoperative ultrasound images of lymph nodes at a first site. Optionally, process 212 can include one or more of processes 212a-212d. At process 212a, local image data can be displayed. For example and with reference to Fig. 12A , the instrument 356 (e.g., an EBUS-TBNA instrument) can have a field of view area 390 including node B. The field of view area 390 can be in the form of a navigation guide for the medical instrument. In the user interface 352, the panel 384 can display the local image data as an ultrasound image 386 including node B in the field of view area 390. The image 386 can provide the size, shape and other information about the node B, which can be recorded and used for subsequent positioning / juxtaposition and orientation of the same or different biopsy instruments for biopsy of the node B. The panel 388 can provide a cross-sectional view of the instrument 356, which indicates the direction of the transducer 389 relative to the field of view area 390 displayed in the ultrasound image 386. Fig. 12B Another example of a user interface 350, 352 that may be used is provided.

[0066] Fig. 12CAnother example of a user interface 350, 352 that may be used is provided. Fig. 12C In an example of , a staging sequence window 351 can be provided in the user interface 350. The staging sequence window 351 can include a list of nodes in the staging sequence, and the current node (e.g., node 05) can be highlighted in the model 300 and the window 351 (e.g., with shading, brightness, or color). In this example, the user interface 352 can include a node information window 353, which can include imaging (e.g., CT and / or EBUS) derived features of the node. For example, the node information window can include the distance to the node and node features, such as short axis size, roundness, a measure of distinct margins, an indicator of central hilar structure, an indicator of heterogeneity, and / or coagulative necrosis factor. Fig.12D Can include Fig. 12C Same user interface 350, 352, but in this example, certain information items (eg, minor axis, circularity, hilar index, heterogeneity index) may be highlighted based on user interest, importance to the procedure, or other criteria.

[0067] Various types of image annotations can be displayed in various combinations with local image data (e.g., ultrasound image 386). Additionally or alternatively, image annotations can be registered and transformed for display in any of a variety of display views. For example, model annotations based on transformed image annotations can be overlaid on an image of an anatomical model (e.g., three-dimensional model 300). For example, orientation annotations based on transformed image annotations can be overlaid on a user interface orientation element (e.g., user interface orientation element 370). Similarly, annotations from other views can be transformed for display with local image data.

[0068] At process 212b, the local image data may be displayed together with the lymph node annotations. Fig.13A , the image annotation may include a boundary annotation 391, which may be made by a clinician or graphic analysis software to indicate the boundary of node B. The boundary annotation may be overlaid on the ultrasound image 386. The image annotation may also or alternatively include a distance indicator 392, which may provide a size dimension of node B that may be displayed in panel 393. The distance indicator 392 may provide a measurement between the proximal and distal sides of a lymph node visible in the ultrasound image 386. The distance indicator 392 may be overlaid on the ultrasound image 386. Fig. 13B Another example of a user interface 350, 352 that may be used is provided.

[0069] At process 212c, the local image data may be displayed together with the supplemental imaging annotations. Fig.14A, the image annotation may include Doppler information 395 , which may be overlaid on the ultrasound image 386 in the panel 384 to further annotate the image 386 . Fig. 14B Another example of a user interface 350, 352 that may be used is provided.

[0070] At process 212d, the local image data may be displayed along with key anatomy annotations. Fig.15A , image annotations in the form of key anatomy indicators 312 and boundary annotations 391 of node A may be overlaid on the ultrasound image 386 in panel 384 . Fig. 15B Another example of a user interface 350, 352 that may be used is provided.

[0071] Any annotations, including the boundary annotations 391 , distance indicators 392 , Doppler information 395 , or key anatomy 312 , may be interchangeably or selectively displayed with or without the ultrasound image 386 , user interface elements 370 , and / or anatomical model 300 .

[0072] The process 210, 212 may be repeated at each node in the lymph node sequence. In some procedures, a clinical intervention, such as a biopsy procedure, may be performed immediately after the guidance and local imaging process 210, 212 is performed at the node site before advancing to the next node in the sequence. In some procedures, the guidance and local imaging process 210, 212 may be performed for each node in the sequence before a clinical intervention is performed on any node in the sequence.

[0073] At process 214, optionally, information from clinical interventions at sequence nodes can be recorded and archived. The staging history information can include automatic archiving of node information, procedural steps, and biopsy passes. For example, clinical interventions, such as biopsy procedures, can be performed at one or more node sites in the order of the sequence, and information about the biopsy can be recorded and associated with each node site. The biopsy information can include, for example, the number of biopsy passes, pathological results of the sample, the depth of the biopsy needle, the volume of material removed, or other information about the biopsy procedure. For example and with reference to Fig.16A , biopsy information can be recorded and stored for each biopsied knot. The knot information displayed in panel 321 can include, for example, the number of biopsy stages. For example, three biopsy stages are recorded for knot A, three biopsy stages are recorded for knot B, and zero biopsy stages are recorded for knot C. Fig. 16B Another example of a user interface 350, 352 that may be used is provided.

[0074] Fig.17is a schematic representation of a robotic or teleoperated medical system 500 ("medical system 500") configured in accordance with various embodiments of the present technology. Medical system 500 may be similar to the medical system described above with respect to Figure 1A - any of the procedures or methods described in FIG. 16. For example, as previously described, the medical system 500 can be used to plan and / or perform imaging and / or biopsy procedures on one or more node sites. As shown, the medical system 500 includes a manipulator assembly 502, a medical device system 504, a main control assembly 506, and a control system 512. The manipulator assembly 502 supports the medical device system 504 and drives the medical device system 504 under the guidance of the main control assembly 506 and / or the control system 512 to perform various medical procedures on a patient 503 placed on an operating table 507 in a surgical environment 501. In this regard, the main control assembly 506 generally includes one or more control devices that can be operated by an operator 505 (e.g., a physician) to control the manipulator assembly 502. Additionally or alternatively, the control system 512 includes a computer processor 514 and at least one memory 516 to implement control between the medical device system 504, the main control assembly 506, and / or other components of the medical system 500. The control system 512 may also include programming instructions (e.g., a non-transitory computer-readable medium storing instructions) to implement any one or more of the methods described herein, including instructions for providing information to the display system 510 and / or processing data to register the medical device system 504 with the anatomical model of the patient 503 (this is described in more detail below). The manipulator assembly 502 may be a teleoperated, non-teleoperated, or hybrid teleoperated and non-teleoperated assembly. Thus, all or a portion of the master control assembly 506 and / or all or a portion of the control system 512 may be located inside or outside the surgical environment 501.

[0075] To assist the operator 505 in controlling the manipulator assembly 502 and / or the medical device system 504 during an image-guided medical procedure, the medical system 500 may also include a position sensor system 508, an endoscopic imaging system 509, an imaging system 518, and / or a virtual visualization system 515. In some embodiments, the position sensor system 508 includes a positioning sensor system (e.g., an electromagnetic (EM) sensor system) and / or a shape sensor system to capture position sensor data (e.g., position, orientation, speed, velocity, posture, shape, etc.) of the medical device system 504. In these and other embodiments, the endoscopic imaging system 509 includes one or more image capture devices (not shown) that record endoscopic image data including concurrent or real-time images (e.g., video, still images, etc.) of the patient's anatomy. The images captured by the endoscopic imaging system 509 may be, for example, 2D or 3D images of the patient's anatomy captured by an image capture device located within the patient 503, and are hereinafter referred to as "real navigation images".

[0076] In some embodiments, the medical device system 504 may include components of the position sensor system 508 and / or components of the endoscopic imaging system 509. For example, the components of the position sensor system 508 and / or components of the endoscopic imaging system 509 may be integrally or removably coupled to the medical device system 504. Additionally or alternatively, the endoscopic imaging system 509 may include a separate endoscope (not shown) attached to a separate manipulator assembly (not shown) that may be used in conjunction with the medical device system 504 to image the patient's anatomy. The position sensor system 508 and / or the endoscopic imaging system 509 may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, such as the computer processor(s) 514 of the control system 512.

[0077] In some examples, the medical device system 504 may include intraoperative imaging capabilities, such as ultrasound imaging, for imaging tissue that is not visible to the endoscopic imaging system. Thus, the medical device system 504 may include an ultrasound imaging system 511 that includes an ultrasound transducer for imaging tissue (such as lymph nodes) outside the walls of the anatomical passageway in which the device system extends.

[0078] The imaging system 518 of the medical system 500 can be arranged in the surgical environment 501 near the patient 503 to obtain real-time and / or near real-time images of the patient 503 before, during and / or after the medical procedure. In some embodiments, the imaging system 518 includes a mobile C-arm cone beam CT imaging system for generating 3D images. For example, the imaging system 518 may include a Siemens DynaCT imaging system or another suitable imaging system. In these and other embodiments, the imaging system 518 may include other imaging technologies, including MRI, fluoroscopy, thermal imaging, ultrasound, OCT, thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, etc.

[0079] When controlling the medical device system 504 during an image-guided medical procedure, a virtual visualization system 515 of the control system 512 provides navigation and / or anatomical interaction assistance to the operator 505. As described in more detail below, virtual navigation using the virtual visualization system 515 can be based at least in part on reference to an acquired preoperative or intraoperative data set of anatomical pathways of the patient 503 (e.g., based at least in part on reference to data generated by the position sensor system 508, the endoscopic imaging system 509, and / or the imaging system 518). For example, in some embodiments, the virtual visualization system 515 processes preoperative and / or intraoperative image data of an anatomical region of the patient 503 captured by the imaging system 518 to generate an anatomical model (not shown) of the anatomical region. The virtual visualization system 515 then aligns the anatomical model with the position sensor data generated by the position sensor system 508 and / or the endoscopic image data generated by the endoscopic imaging system 509 to (i) map the tracked position, orientation, posture, shape and / or movement of the medical device system 504 within the anatomical region to a correct position within the anatomical model, and / or (ii) determine a virtual navigation image of a virtual patient anatomy of the anatomical region from the viewpoint of the medical device system 503 at a position within the anatomical model corresponding to the position of the medical device system 504 within the patient 503.

[0080] Display system 510 (e.g., display system 302) can display various images or representations of patient anatomy and / or medical instrument system 504 generated by position sensor system 508, endoscopic imaging system 509, imaging system 518, and / or virtual visualization system 515. In some embodiments, display system 510 and / or master control assembly 506 can be oriented so that operator 505 can control manipulator assembly 502, medical instrument system 504, master control assembly 506, and / or control system 512 through telepresented perception.

[0081] As described above, the manipulator assembly 502 drives the medical device system 504 under the guidance of the master assembly 506 and / or the control system 512. In this regard, the manipulator assembly 502 may include selected degrees of freedom of motion that may be motorized and / or remotely operated, and selected degrees of freedom of motion that may be non-motorized and / or non-remotely operated. For example, the manipulator assembly 502 may include a plurality of actuators or motors (not shown) that drive inputs on the medical device system 504 in response to commands received from the control system 512. The actuator may include a drive system (not shown) that, when coupled to the medical device system 504, may advance the medical device system 502 into a natural orifice or an anatomical orifice created by a surgical procedure. Other drive systems may move a distal portion (not shown) of the medical device system 504 in multiple degrees of freedom, which may include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of rotational motion (e.g., rotation around the X, Y, and Z Cartesian axes). Additionally or alternatively, the actuator may be used to actuate an articulatable end effector of the medical instrument system 504 (eg, for grasping tissue in the jaws of a biopsy device or the like).

[0082] Fig.18 is within a surgical environment 501 and configured according to various embodiments of the present technology Fig.17 Schematic representation of manipulator assembly 502, medical device system 504, and imaging system 518. Fig.18 As shown, the surgical environment 501 has a surgical reference system (X S , Y S , Z S ), wherein the patient 503 is located on the operating table 507, and the medical instrument system 504 has a medical instrument reference system (X) within the surgical environment 501 M , Y M , Z M ). During the medical procedure, the patient 503 can remain stationary within the surgical environment 501, to the extent that the patient's overall movement can be limited by sedation, restraints, and / or other means. In these and other embodiments, the cyclic anatomical movement of the patient 503 (including respiratory and cardiac movement) can continue, unless the patient 503 is required to hold his breath to temporarily suspend respiratory movement.

[0083] The manipulator assembly 502 includes an instrument carriage 626 mounted to an insertion table 628. In the illustrated embodiment, the insertion table 628 is linear, while in other embodiments the insertion table 628 is curved or has a combination of curved and linear sections. In some embodiments, the insertion table 628 is fixed within the surgical environment 501. Alternatively, the insertion table 628 can be movable within the surgical environment 501, but have a known position within the surgical environment 501 (e.g., known via a tracking sensor (not shown) or other tracking device). In these alternatives, the medical instrument reference frame (X) is a linear system. M , Y M , Z M ) relative to the surgical reference frame (X S , Y S , Z S ) is fixed or otherwise known.

[0084] Fig.18 The medical device system 504 includes an elongated device 631, a medical device 632, an instrument body 635, at least a portion of the position sensor system 508, and at least a portion of the endoscopic imaging system 509. In some embodiments, the elongated device 631 is a flexible catheter or other biomedical device that defines a channel or lumen 644. The channel 644 can be sized and shaped to receive the medical device 632 (e.g., via a proximal end 636 of the elongated device 631 and / or an instrument port (not shown)) and facilitate delivery of the medical device 632 to a distal portion 638 of the elongated device 631. The elongated device 631 is coupled to the instrument body 635, which in turn is coupled to and fixed relative to the instrument bracket 626 of the manipulator assembly 502.

[0085] In operation, the manipulator assembly 502 can control insertion movement (e.g., proximal and / or distal movement along axis A) of the elongated device 631 into the body of the patient 503 via a natural orifice or a surgically created anatomical orifice of the patient 503 to facilitate navigation of the elongated device 631 through an anatomical pathway of an anatomical region of the patient 503 and / or to facilitate delivery of the distal portion 638 of the elongated device 631 to or near a target location within the body of the patient 503. For example, the instrument bracket 626 and / or the insertion stage 628 can include an actuator (not shown), such as a servo motor, that facilitates controlling movement of the instrument bracket 626 along the insertion stage 628. Additionally or alternatively, in some embodiments, the manipulator assembly 502 can control movement of the distal portion 638 of the elongated device 631 in multiple directions, including yaw, pitch, and roll rotational directions (e.g., to navigate the patient anatomy). To this end, the elongated device 631 can house or include cables, linkages, and / or other steering controls (not shown) that the manipulator assembly 502 can use to controllably bend the distal portion 638 of the elongated device 631. For example, the elongated device 631 can house at least four cables that the manipulator assembly 502 can use to provide (i) independent "up and down" steering to control the pitch of the distal portion 638 of the elongated device 631, and (ii) independent "side to side" steering of the elongated device 631 to control the yaw of the distal portion 638 of the elongated device 631.

[0086] The medical device 632 of the medical device system 504 can be used for medical procedures, such as for investigation of anatomical pathways, surgery, biopsy, ablation, illumination, irrigation, and / or aspiration. Thus, the medical device 632 can include an image capture probe, a biopsy instrument or device (e.g., a biopsy needle, an endobronchial ultrasound (EBUS) probe), a laser ablation fiber, and / or other surgical, diagnostic, and / or therapeutic tools. For example, the medical device 632 can include an endoscope or other biomedical device having one or more image capture devices 647 located at a distal portion 637 of the medical device 632 and / or at other locations along the medical device 632. In these embodiments, when the medical device 632 is located within the anatomical region of the patient 503, the image capture device 647 can capture one or more real navigation images or videos (e.g., a sequence of one or more real navigation image frames) of the anatomical pathway and / or other real patient anatomy.

[0087] As described above, the medical device 632 can be deployed and / or delivered to a target location in the patient 503 via the channel 644 defined by the elongated device 631. In embodiments where the medical device 632 comprises an endoscope or other biomedical device having an image capture device 647 at a distal portion 637 thereof, the image capture device 647 can be advanced to the distal portion 638 of the elongated device 631 before, during, and / or after the manipulator assembly 502 navigates the distal portion 638 of the elongated device 631 to the target location in the patient 503. In these embodiments, the medical device 632 can be used as an investigative device to capture realistic navigation images of anatomical pathways and / or other real patient anatomy, to capture intraoperative ultrasound images of tissue outside of anatomical pathways, and / or to assist an operator (not shown) in navigating the distal portion 638 of the elongated device 631 through an anatomical pathway to a target location to perform an imaging procedure or clinical intervention.

[0088] As another example, after the manipulator assembly 502 positions the distal portion 638 of the elongated device 631 near the target location in the patient 503, the medical device 632 can be advanced out of the distal portion 638 of the elongated device 631 to perform a medical procedure at the target location. Continuing with this example, after all or part of the medical procedure at the target location is completed, the medical device 632 can be retracted into the elongated device 631 and, additionally or alternatively, can be removed from the proximal end 636 of the elongated device 631 or from another instrument port (not shown) along the elongated device 631.

[0089] like Fig.18 As shown, the position sensor system 508 of the medical device system 504 includes a shape sensor 633 and a position measurement device 639. In these and other embodiments, the position sensor system 508 can include other position sensors (e.g., accelerometers, rotary encoders, etc.) in addition to or in lieu of the shape sensor 633 and / or the position measurement device 639.

[0090] The shape sensor 633 of the position sensor system 508 includes an optical fiber extending within and aligned with the elongated device 631. In one embodiment, the optical fiber of the shape sensor 633 has a diameter of approximately 200 μm. In other embodiments, the diameter of the optical fiber can be larger or smaller. The optical fiber of the shape sensor 633 forms an optical fiber bend sensor for determining the shape, orientation and / or posture of the elongated device 631. In some embodiments, an optical fiber with a fiber Bragg grating (FBG) can be used to provide strain measurements in a structure in one or more dimensions. Various systems and methods for monitoring the shape and relative position of an optical fiber in three dimensions are described in more detail in U.S. Patent Application Publication No. 2006 / 0013523 (filed July 13, 2005) (disclosing optical fiber position and shape sensing devices and related methods); U.S. Patent No. 7,781,724 (filed September 26, 2006) (disclosing optical fiber position and shape sensing devices and related methods); U.S. Patent No. 7,772,541 (filed March 12, 2008) (disclosing optical fiber position and / or shape sensing based on Rayleigh scattering); and U.S. Patent No. 6,389,187 (filed June 17, 1998) (disclosing optical fiber bend sensors), the entire contents of which are incorporated herein by reference. In these and other embodiments, sensors of the present technology may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In these and other embodiments, the shape of the elongated device 631 may be determined using other techniques. For example, the history of the posture of the distal portion 638 of the slender device 631 can be used to reconstruct the shape of the slender device 631 over a period of time.

[0091] In some embodiments, the shape sensor 633 is fixed at a proximal point 634 on the device body 635 of the medical device system 504. For example, in operation, the shape sensor 633 measures the shape of the medical device reference frame (X). M , Y M , Z M ) to another point along the optical fiber, such as a distal portion 638 of the elongated device 631. The proximal point 634 of the shape sensor 633 can be movable with the instrument body 635, but the position of the proximal point 632 can be known (e.g., via a tracking sensor (not shown) or other tracking device).

[0092] The position measurement device 639 of the position sensor system 508 provides information about the position of the instrument body 635 as it moves along the insertion axis A on the insertion stage 628 of the manipulator assembly 502. In some embodiments, the position measurement device 639 includes a resolver, an encoder, a potentiometer, and / or other sensor that determines the rotation and / or orientation of an actuator (not shown) that controls the movement of the instrument bracket 626 of the manipulator assembly 502 and, thereby, the movement of the instrument body 635 of the medical device system 504.

[0093] The systems and methods described herein may be provided in the form of one or more tangible and non-transient machine-readable media (such as hard disk drives, hardware memories, etc.) on which instructions are recorded for execution by a processor or computer. An instruction set may include various commands that instruct a computer or processor to perform a specific operation (such as the methods and processes of the various embodiments described herein). An instruction set may be in the form of a software program or application. Computer storage media may include volatile and non-volatile media and removable and non-removable media for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media may include, but are not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid-state memory technology, CD-ROM, DVD or other optical storage, disk storage, or any other hardware medium that can be used to store desired information and can be accessed by components of the system. The components of the system may communicate with each other via wired or wireless communications. These components may be separated from each other, or various combinations of components may be integrated together into a monitor or processor, or included in a workstation with standard computer hardware (e.g., processor, circuit system, logic circuit, memory, etc.). The system may include processing devices such as microprocessors, microcontrollers, integrated circuits, control units, storage media, and other hardware.

[0094] Although many embodiments are described above in the context of navigating and performing medical procedures within a patient's lungs, other applications and other embodiments beyond those described herein are also within the scope of the present technology. For example, unless otherwise indicated or explicitly indicated in the context, the devices, systems, methods, and computer program products of the present technology can be used for various image-guided medical procedures, such as medical procedures performed on, within, or near hollow patient anatomy, and more specifically, in procedures for investigating, biopsy, ablating, or otherwise treating tissue within and / or near hollow patient anatomical structures. Thus, for example, the systems, devices, methods, and computer program products of the present disclosure can be used in one or more medical procedures associated with other patient anatomy, such as a patient's bladder, urinary tract, GI system, and / or heart.

[0095] As used herein, the term "operator" should be understood to include any type of person who may be performing or assisting in a medical procedure, and thus includes physicians, surgeons, doctors, nurses, medical technicians, other personnel or users of the technology disclosed herein, and any combination thereof. Additionally or alternatively, the term "patient" should be deemed to include human and / or non-human (e.g., animal) patients on whom a medical procedure is being performed.

[0096] It can be understood from the above that the specific embodiments of the present technology are described herein for the purpose of illustration, but in order to avoid unnecessarily obscuring the description of the embodiments of the present technology, the well-known structures and functions are not shown or described in detail. If any material incorporated herein by reference conflicts with the present disclosure to some extent, the present disclosure shall prevail. Where the context permits, singular or plural terms may also include plural or singular terms, respectively. In addition, unless the term "or" is explicitly limited to refer only to a single item other than other items when referring to a list of two or more items, the use of "or" in such a list should be interpreted as including (a) any single item in the list, (b) all items in the list, or (c) any combination of items in the list. As used herein, the phrase "and / or" as in "A and / or B" refers to A alone, B alone, and both A and B. Where the context permits, singular or plural terms may also include plural or singular terms, respectively. In addition, the terms "include", "comprise", "have" and "with" are always used to indicate that at least the recorded (one or more) features are included, so that any greater number of the same features and / or additional types of other features are not excluded.

[0097] In addition, as used herein, the term "substantially" refers to the degree or extent of complete or nearly complete action, characteristic, property, state, structure, item, or result. For example, an object that is "substantially" enclosed means that the object is either completely enclosed or nearly completely enclosed. In some cases, the exact degree of deviation allowed from absolute completeness may depend on the specific context. However, in general, near completion will produce the same overall result as absolute and complete completion. When used in a negative connotation, the use of "substantially" is equally applicable to referring to the complete or nearly complete lack of an action, characteristic, property, state, structure, item, or result.

[0098] The above detailed description of the embodiments of the present technology is not intended to be exhaustive or to limit the present technology to the precise form disclosed above. Although specific embodiments and examples of the present technology are described above for illustrative purposes, those skilled in the relevant art will recognize that various equivalent modifications may be made within the scope of the present technology. For example, although the steps are presented in a given order, alternative embodiments may perform the steps in a different order. As another example, the various components of the technology may be further divided into sub-components, and / or the various components and / or functions of the technology may be combined and / or integrated. In addition, although the advantages associated with certain embodiments of the present technology are described in the context of these embodiments, other embodiments may also exhibit these advantages, and not all embodiments must exhibit these advantages to fall within the scope of the present technology.

[0099] It should also be noted that other embodiments in addition to the embodiments disclosed herein are also within the scope of the present technology. For example, embodiments of the present technology may also have different configurations, components, and / or processes in addition to those shown or described herein. In addition, it will be understood by those of ordinary skill in the art that these and other embodiments may not have several configurations, components, and / or processes shown or described herein without departing from the present technology. Therefore, the present disclosure and associated technology may cover other embodiments not explicitly shown or described herein.

Claims

1. A system for providing navigation guidance for a medical procedure, the system comprising: processor; as well as a memory operably coupled to the processor and storing instructions that, when executed by the processor, cause the system to perform operations comprising: receiving an anatomical model of the anatomical region; performing an analysis on the anatomical model to facilitate selection of a plurality of node sites for analysis; generating a program sequence for the plurality of knot sites; providing navigation guidance to guide a medical instrument to a knot site among the plurality of knot sites during the procedural sequence; and Local image data is collected from the medical device at the knot site. 2 . The system of claim 1 , wherein performing the analysis comprises identifying a plurality of anatomical points of the anatomical model.

3. The system of claim 1, wherein performing the analysis comprises identifying a plurality of lymph nodes in the anatomical model.

4. The system of claim 3, wherein the operations performed further comprise displaying an image of the anatomical model and images of segmented lymph nodes of the plurality of lymph nodes on a graphical user interface.

5. The system of claim 4, wherein the image of the segmented lymph node is overlaid on an image of an anatomical point of the anatomical model. 6 . The system of claim 5 , wherein the operations performed further include highlighting the image of the staged lymph nodes if a staging threshold is met.

7. The system of claim 4, wherein the operations performed further include overlaying PET imaging information on the image of the anatomical model.

8. The system of claim 3, wherein the navigation guidance comprises a displayed list view of the plurality of lymph nodes grouped by a plurality of anatomical points of the anatomical model.

9. The system of claim 8, wherein the list view includes a size of at least one lymph node of the plurality of lymph nodes.

10. The system of claim 8, wherein the list view includes a shape of at least one lymph node of the plurality of lymph nodes.

11. The system of claim 3, wherein the operations performed further include displaying critical anatomy proximate at least one of the plurality of knot sites.

12. The system of claim 1, wherein the navigation guidance comprises an image of the anatomical model and a representation of a navigable path through the procedure sequence for the plurality of node locations.

13. A system according to claim 12, wherein the representation of the navigable path includes a first portion and a second portion, the first portion indicating a portion of the navigable path that the medical device has reached, and the second portion indicating a portion of the navigable path that the medical device has not yet reached, and wherein the first portion is visually distinguishable from the second portion.

14. The system of claim 12, wherein the navigation guidance further comprises displayed representations of key anatomy along the navigable path.

15. The system of claim 12, wherein the navigation guidance further comprises a displayed knot marker adjacent to one or more knot locations of the plurality of knot locations.

16. The system of claim 1, wherein the navigation guidance comprises an indicator configured to indicate that at least one of the plurality of knot sites has been sampled by the medical device.

17. The system of claim 1, wherein the navigation guidance comprises an indicator configured to indicate that at least one of the plurality of knot sites has not yet been sampled by the medical device.

18. The system of claim 1, wherein the navigation guidance comprises an indicator configured to indicate a knot site among the plurality of knot sites to be sampled by the medical device.

19. The system of claim 18, wherein the indicator is displayed together with the three-dimensional image of the anatomical model.

20. The system of claim 18, wherein the indicator is displayed with a virtual endoscopic image view.

21. The system of claim 18, wherein the indicator is displayed with a real-time endoscopic image view.

22. The system of claim 1, wherein providing navigation guidance comprises displaying user interface orientation elements including a direction indicator of an imaging element of the medical device and a representation of critical anatomy.

23. The system of claim 1, wherein providing navigation guidance comprises displaying a representation of a field of view area of ​​an imaging element of the medical device.

24. The system of claim 1, wherein the operations performed further include displaying the partial image data together with image annotations of the partial image data.

25. The system of claim 24, wherein the image annotation of the partial image data comprises a boundary annotation overlaid on the displayed partial image data.

26. The system of claim 24, wherein the image annotation includes a distance indicator between a proximal side and a distal side of a lymph node visible in the local image data.

27. The system of claim 24, wherein the image annotation comprises Doppler information overlaid on the local image data.

28. The system of claim 27, wherein the image annotation further comprises a boundary annotation overlaid on the local image data along with the Doppler information.

29. The system of claim 24, wherein the image annotation further comprises outlines of key anatomy overlaid on the local image data.

30. The system of any one of claims 24-29, wherein the operations performed further include displaying model annotations overlaid on the image of the anatomical model, wherein the model annotations are based on the image annotations.

31. The system of any one of claims 24-29, wherein the operations performed further include displaying an orientation annotation overlaid on a user interface orientation element, wherein the orientation annotation is based on the image annotation.

32. A method comprising: receiving an anatomical model of the anatomical region; performing an analysis on the anatomical model to facilitate selection of a plurality of node sites for analysis; generating a program sequence for the plurality of knot sites; providing navigation guidance to guide a medical instrument to a knot site among the plurality of knot sites in the procedural sequence; as well as Local image data is collected from the medical device at the knot site.

33. The method of claim 32, wherein performing the analysis comprises identifying a plurality of anatomical points of the anatomical model.

34. The method of claim 32, wherein performing the analysis comprises identifying a plurality of lymph nodes in the anatomical model.

35. The method of claim 34, further comprising displaying an image of the anatomical model and an image of segmented lymph nodes of the plurality of lymph nodes on a graphical user interface.

36. The method of claim 35, wherein the image of the segmented lymph node is overlaid on an image of an anatomical point of the anatomical model.

37. The method of claim 36, further comprising highlighting the image of the staged lymph nodes if a staging threshold is met.

38. The method of claim 35, further comprising overlaying PET imaging information on the image of the anatomical model.

39. The method of claim 34, wherein the navigation guidance comprises a displayed list view of the plurality of lymph nodes grouped by a plurality of anatomical points of the anatomical model.

40. The method of claim 39, wherein the list view includes a size of at least one lymph node of the plurality of lymph nodes.

41. The method of claim 39, wherein the list view includes a shape of at least one lymph node of the plurality of lymph nodes.

42. The method of claim 34, further comprising: Critical anatomy is displayed proximate at least one of the plurality of knot sites.

43. The method of claim 32, wherein the navigation guidance comprises an image of the anatomical model and a representation of a navigable path through the procedural sequence for the plurality of node locations.

44. A method according to claim 43, wherein the representation of the navigable path includes a first portion and a second portion, the first portion indicating a portion of the navigable path that the medical device has reached, and the second portion indicating a portion of the navigable path that the medical device has not yet reached, and wherein the first portion is visually distinguishable from the second portion.

45. The method of claim 43, wherein the navigation guidance further comprises displayed representations of key anatomy along the navigable path.

46. ​​The method of claim 43, wherein the navigation guidance further comprises displayed knot markers adjacent to one or more knot sites of the plurality of knot sites.

47. The method of claim 32, wherein the navigation guidance comprises an indicator configured to indicate that at least one of the plurality of knot sites has been sampled by the medical device.

48. The method of claim 32, wherein the navigation guidance comprises an indicator configured to indicate that at least one of the plurality of knot sites has not yet been sampled by the medical device.

49. The method of claim 32, wherein the navigation guidance comprises an indicator configured to indicate a knot site among the plurality of knot sites to be sampled by the medical device.

50. The method of claim 49, wherein the indicator is displayed together with a three-dimensional image of the anatomical model.

51. The method of claim 49, wherein the indicator is displayed with a virtual endoscopic image view.

52. The method of claim 49, wherein the indicator is displayed with a real-time endoscopic image view.

53. The method of claim 32, wherein providing navigation guidance comprises displaying user interface orientation elements including directional indicators of imaging elements of the medical device and representations of critical anatomy.

54. The method of claim 32, wherein providing navigation guidance comprises displaying a representation of a field of view area of ​​an imaging element of the medical device.

55. The method of claim 32, further comprising displaying the partial image data together with an image annotation of the partial image data.

56. The method of claim 55, wherein the image annotation of the partial image data comprises a boundary annotation overlaid on the displayed partial image data.

57. The method of claim 55, wherein the image annotation includes a distance indicator between a proximal side and a distal side of a lymph node visible in the local image data.

58. The method of claim 55, wherein the image annotation includes Doppler information overlaid on the local image data.

59. The method of claim 58, wherein the image annotation further comprises a boundary annotation overlaid on the local image data along with the Doppler information.

60. The method of claim 55, wherein the image annotation further comprises outlines of key anatomy overlaid on the local image data.

61. The method of any one of claims 55-60, further comprising displaying model annotations overlaid on the image of the anatomical model, wherein the model annotations are based on the image annotations.

62. The method of any of claims 55-60, further comprising displaying an orientation annotation overlaid on a user interface orientation element, wherein the orientation annotation is based on the image annotation.

Images