Graphical user interface for planning programs

By using a graphical user interface to display image data and receive user input in the medical program planning system, dynamically updating interactive images, solving the problems of intuitive planning and anatomical channel management in existing systems, achieving more efficient and safe minimally invasive medical operations .

CN119069085APending Publication Date: 2024-12-03INTUITIVE SURGICAL OPERATIONS INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411018672.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2017-10-27
Filing Date
2018-04-18
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing medical program planning systems are difficult to provide an intuitive graphical user interface, limiting the intuitive planning and execution of minimally invasive medical technologies in medical programs, especially in managing the insertion, retraction and steering of elongated devices.

Method used

Display image data through a graphical user interface, receive user input to identify anatomical channels and targets, dynamically update interactive images, provide multiple interactive windows to view paths and instrument locations, and support users to virtually control the location of the instrument within the anatomical channels and Path planning.

Benefits of technology

It improves the planning intuitiveness and accuracy of medical procedures, reduces operational complexity, enhances the interactiveness of anatomical structure marking and path planning, and improves the efficiency and safety of minimally invasive medical operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119069085A_ABST
    Figure CN119069085A_ABST
Patent Text Reader

Abstract

The invention relates to a graphical user interface for planning a program. A system and method of planning a program includes a planning workstation comprising: a display system; and a user input device. The planning workstation is configured to: display the image data via the display system; receiving a first user input via a user input device; displaying, via a display system, a target of the medical procedure within the displayed image data identified based at least on the first user input; displaying an interactive image via the display system, the interactive image comprising the image data, the plurality of connected anatomical channels, and the identified target; receiving a second user input via the user input device; displaying, via the display system, a trajectory between the target and an exit point along a nearest channel of the plurality of connected anatomical channels identified based at least on the second user input; receiving a third user input via the user input device; and adjusting the interactive image based at least on the defined trajectory and the third user input.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of Chinese patent application 2018800335548 (PCT / US2018 / 028199) filed on April 18, 2018 and entitled “Graphical User Interface for Planning Programs”.

[0002] Related Applications

[0003] The present disclosure claims priority to U.S. Provisional Patent Application No. 62 / 486,896, filed on April 18, 2017, and entitled “Graphical User Interface for Planning Procedures,” and U.S. Provisional Patent Application No. 62 / 578,189, filed on October 27, 2017, and entitled “Graphical User Interface for Labeling Anatomy,” which are incorporated herein by reference in their entirety. Technical Field

[0004] The present disclosure relates to systems and methods for performing image-guided procedures, and more particularly to systems and methods for analyzing, identifying and / or marking anatomy using a graphical user interface. Background Art

[0005] Minimally invasive medical technology aims to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort and harmful side effects. This minimally invasive technology can be performed through natural orifices in the patient's anatomy or through one or more surgical incisions. Through these natural orifices or incisions, clinicians can insert minimally invasive medical devices (including surgical instruments, diagnostic instruments, therapeutic instruments or biopsy instruments) to reach the target tissue position. One such minimally invasive technology uses a steerable flexible elongated device (such as a catheter) that can be inserted into an anatomical channel and navigated toward an area of ​​interest in the patient's anatomy. The control of this elongated device by medical personnel during image-guided procedures involves the management of several degrees of freedom, including at least managing the insertion and retraction of the elongated device and the steering or bending radius of the device. In addition, different operating modes can also be supported.

[0006] Therefore, it would be advantageous to provide a graphical user interface that supports intuitive planning of medical procedures, including minimally invasive medical techniques. Summary of the invention

[0007] Embodiments of the invention are best summarized by the claims appended to the description.

[0008] According to some embodiments, a method for planning a medical procedure using a graphical user interface may include displaying image data via the graphical user interface and receiving a first user input defining a target of the medical procedure within the displayed image data. The method may further include displaying an interactive image via the graphical user interface, the interactive image including the image data, a plurality of connected anatomical corridors detected by segmentation of the image data, and the defined target. The method may further include receiving a second user input defining a trajectory between the target and an exit point along a nearest corridor of the plurality of connected anatomical corridors, and receiving a third user input adjusting the interactive image based on the defined trajectory.

[0009] According to some embodiments, a method for planning a medical procedure using a graphical user interface may include displaying image data via a graphical user interface, receiving a first user input defining hazards within the displayed image data, and displaying an interactive image. The interactive image includes the image data, a plurality of connected anatomical pathways detected by segmentation of the image data, and the defined hazards.

[0010] According to some embodiments, a method for previewing a plan for a medical procedure using a graphical user interface may include providing a plurality of interactive windows for a user to view the plan for the medical procedure. Each of the plurality of interactive windows may display a different rendering of a model of an anatomical passage. The method may further include displaying a path through the anatomical passage to a target of the medical procedure, displaying a virtual image of an instrument within the anatomical passage, displaying a control point corresponding to a distal end of the instrument in at least one of the plurality of interactive windows, receiving user input defining an orientation of the control point, and in response to receiving the user input, dynamically updating the orientation of the instrument in each of the plurality of interactive windows to match the orientation of the control point.

[0011] According to some embodiments, the planning workstation may include a display system and a user input device. The planning workstation may be configured to: display image data via the display system; receive a first user input via the user input device, the first user input defining a target of a medical procedure within the displayed image data; display an interactive image via the display system, the interactive image including the image data, a plurality of connected anatomical channels detected by segmenting the image data, and the defined target; receive a second user input via the user input device, the second user input defining a trajectory between the target and an exit point along a nearest channel in the plurality of connected anatomical channels; and receive a third user input via the user input device, the third user input adjusting the interactive image based on the defined trajectory.

[0012] According to some embodiments, a non-transitory machine-readable medium may include a plurality of machine-readable instructions that, when executed by one or more processors associated with a planning workstation, are adapted to cause the one or more processors to perform a method. The method may include displaying image data via a graphical user interface, receiving a first user input defining a hazard within the displayed image data, and displaying an interactive image. The interactive image may include the image data, a plurality of connected anatomical pathways detected by segmentation of the image data, and the defined hazards.

[0013] According to some embodiments, a non-transitory machine-readable medium may include a plurality of machine-readable instructions that, when executed by one or more processors associated with a planning workstation, are adapted to cause the one or more processors to perform a method. The method may include providing a plurality of interactive windows for a user to view a plan for a medical procedure, displaying a path through an anatomical passage to a target of the medical procedure, displaying a virtual image of an instrument within the anatomical passage, displaying a control point corresponding to a distal end of the instrument in at least one of the plurality of interactive windows, receiving user input defining a position of the control point, and in response to receiving the user input, dynamically updating the position of the instrument in each of the plurality of interactive windows to match the position of the control point. Each of the plurality of interactive windows displays a different rendering of a model of the anatomical passage.

[0014] According to some embodiments, a method for planning a medical procedure may include receiving imaging data and providing a model of an anatomical passage based on the imaging data, the anatomical passage including a plurality of branches. An image of the model may be displayed via a graphical user interface. A first user input representing a selection of a first label may be received, and a second user input representing a selection of a first branch among a plurality of branches may be received. In response to the first user input and the second user input, the first branch may be labeled with a first label, and a representation of the first label applied to the first branch may be displayed via the graphical user interface.

[0015] According to some embodiments, a non-transitory machine-readable medium may include a plurality of machine-readable instructions. The instructions may cause one or more processors to: display a model of an anatomical passage including a plurality of branches via a graphical user interface; display a list of anatomical labels via a graphical user interface; receive a first user input selecting a first label from the list of anatomical labels; receive a second user input selecting a first branch from the plurality of branches; and / or apply the first label to the first branch.

[0016] According to some embodiments, the planning workstation may include a display system and a user input device. The planning workstation may be configured to: display an anatomical corridor including a plurality of branches; display a list of labels; receive a first user input selecting a first branch of the plurality of branches via the user input device; receive a second user input selecting a first label from the list of labels via the user input device; and in response to the first user input and the second user input, display a representation of the first label applied to the first branch via the display system.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, other aspects, features and advantages of the present disclosure will be apparent to those skilled in the art from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the receiving Office upon request and payment of the necessary fee.

[0019] Figure 1 is a simplified diagram of a teleoperated medical system according to some embodiments.

[0020] Figure 2A is a simplified diagram of a medical device system according to some embodiments.

[0021] Figure 2B is a simplified diagram of a medical device with an extended medical tool according to some embodiments.

[0022] Figure 3A and 3B is a simplified diagram of a side view of a patient coordinate space including a medical device mounted on an insertion assembly according to some embodiments.

[0023] Figure 4 is a simplified diagram of a graphical user interface in a data selection mode according to some embodiments.

[0024] Figure 5A-5G is a simplified diagram of a graphical user interface in a hybrid segmentation and planning mode according to some embodiments.

[0025] Figure 6 is a simplified diagram of a graphical user interface in preview mode according to some embodiments.

[0026] Figure 7 is a simplified diagram of a graphical user interface in save mode according to some embodiments.

[0027] Figure 8is a simplified diagram of a graphical user interface in management mode according to some embodiments.

[0028] Fig. 9 is a simplified diagram of a graphical user interface in review mode according to some embodiments.

[0029] Fig.10 is a simplified diagram of a method for planning a medical procedure according to some embodiments.

[0030] Fig.11 A simplified diagram of a method for modifying an anatomical representation (eg, a model) to provide access to a goal of a medical procedure, according to some embodiments.

[0031] Fig.12 A simplified diagram of a method for augmenting an anatomical representation (eg, a model) to provide access to a target of a medical procedure, according to some embodiments.

[0032] Fig.13 is a simplified diagram of a method for planning a medical procedure using a graphical user interface according to some embodiments.

[0033] Figure 14A-14F is a simplified diagram of a graphical user interface in branch marking mode according to some embodiments.

[0034] Fig.15A and Fig. 15B is a simplified diagram of a method of applying labels to a branching model of anatomical pathways according to some embodiments.

[0035] Fig.16 is a simplified diagram of a method for planning a medical procedure according to some embodiments.

[0036] Figure 17A-17N is a schematic diagram of a graphical user interface corresponding to the performance of a method for planning a medical procedure according to some embodiments.

[0037] The embodiments of the present disclosure and their advantages can be best understood by referring to the detailed description below. It should be understood that the same reference numerals are used to identify the same elements shown in one or more of the accompanying drawings, wherein the display in the accompanying drawings is for the purpose of illustrating the embodiments of the present disclosure and is not intended to limit the embodiments of the present disclosure. DETAILED DESCRIPTION

[0038] In the following description, the specific details of describing and according to some embodiments of the present disclosure are set forth. Many specific details are set forth in order to provide a thorough understanding of the embodiments. However, it is apparent to those skilled in the art that some embodiments may be implemented without some or all of these specific details. The specific embodiments disclosed herein are illustrative and not restrictive. Those skilled in the art may implement other elements, although not specifically described here, within the scope and spirit of the present disclosure. In addition, in order to avoid unnecessary repetition, one or more features shown and described in conjunction with one embodiment may be incorporated into other embodiments, unless otherwise specifically described or if one or more features render the embodiments inoperative.

[0039] In certain instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0040] The present disclosure describes various apparatuses and parts of apparatuses according to their states in three-dimensional space. As used herein, the term "position" refers to the position of an object or a portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or a portion of an object (three rotational degrees of freedom--e.g., roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or a portion of an object in at least one translational degree of freedom and the orientation of an object or a portion of an object in at least one rotational degree of freedom (up to six degrees of freedom). As used herein, the term "shape" refers to a set of poses, positions, or orientations measured along an object.

[0041] One general aspect of the present disclosure includes a method for planning a medical procedure, the method comprising: displaying image data via a graphical user interface; receiving a first user input through the graphical user interface; identifying at least a portion of a target within the displayed image data using the first user input; displaying an interactive image via the graphical user interface, the interactive image comprising the image data, a plurality of connected anatomical channels associated with the image data, and the identified target; receiving a second user input; identifying at least a portion of a trajectory between the target and an exit point along a nearest connected channel in the plurality of connected anatomical channels using the second user input; receiving a third user input; and adjusting the interactive image based at least on the identified trajectory and using the third user input. Implementations may include one or more of the following features. The method includes providing a line tool via the graphical user interface to receive the second user input. The method of adjusting the interactive image includes: determining a distance represented by the trajectory; determining whether the distance is greater than a predetermined threshold; receiving a fourth user input; identifying at least a portion of an unconnected channel that is closer to the target than the nearest connected channel using the fourth user input; and connecting the unconnected channel to the plurality of connected channels. The method of identifying an unconnected channel includes receiving a fifth user input and iteratively rotating the interactive image using the fifth user input to identify an unconnected channel in the interactive image. The method, wherein the interactive image is iteratively rotated about one or more user-defined rotation points. The method further includes identifying a rotation axis based at least on the one or more user-defined rotation points. The method, wherein the interactive image is iteratively rotated about the rotation axis. The method of adjusting the interactive image includes: determining an exit angle based on the trajectory; and adjusting the exit angle by changing the orientation of the exit point along the nearest connecting channel. The method includes providing a slider via a graphical user interface and receiving user input via the slider that changes the exit point. The method also includes receiving a fourth user input and using a sixth user input to identify a hazard of a medical procedure within the displayed image data. The method, wherein the hazard corresponds to a vulnerable portion of a patient's anatomy. The method, wherein the hazard corresponds to an excessive bend in one or more of a plurality of connected anatomical channels. The method includes displaying a hazard fence to represent the hazard. The method, wherein the hazard fence includes at least one of a disk, a conical hazard fence, and a hemispherical hazard fence. The method also includes receiving a fourth user input and using a seventh user input to identify at least a portion of a path within a plurality of connecting channels to a target. The method, wherein the first user input is received before completing the segmentation of the image data. The method of displaying an interactive image includes overlaying a plurality of connected anatomical channels on displayed image data, and wherein the plurality of connected anatomical channels are dynamically updated to reflect progress in segmentation of the image data. The method also includes receiving a fourth user input and identifying at least a portion of the plurality of connected anatomical channels that are disconnected from the plurality of connected anatomical channels.The method wherein the plurality of connected anatomical passages comprises lung airways.

[0042] One general aspect of the present disclosure includes a method for planning a medical procedure, the method comprising: providing a graphical user interface; displaying image data via the graphical user interface; receiving a first user input; identifying at least a portion of a hazard within the displayed image data using the first user input; and displaying an interactive image including the image data, a plurality of connected anatomical channels, and a representation of the identified hazard. Implementations may include one or more of the following features. The method, wherein the hazard comprises a vulnerable portion of a patient's anatomy. The method, wherein the hazard comprises an excessive bend within a plurality of connected anatomical channels. The method comprises displaying a hazard fence to represent the hazard. The method, wherein the hazard fence comprises at least one of a disk, a conical hazard fence, and a hemispherical hazard fence. The method, wherein the plurality of connected anatomical channels comprise a pulmonary airway. The method further comprises receiving a second user input, identifying at least a portion of a target of the medical procedure within the displayed image data, and wherein the hazard corresponds to a vulnerable portion of the patient's anatomy proximal to the target. The method, wherein the vulnerable portion of the patient's anatomy comprises at least one of a pulmonary pleura, a blood vessel, a large bullae, and a heart. The method, wherein the first user input is received before completing segmentation of the image data.

[0043] A general aspect of the present disclosure includes a method for previewing a plan for a medical procedure, the method comprising: providing a graphical user interface, the graphical user interface including multiple interactive windows displaying the plan of the medical procedure, wherein at least two different renderings of a model of an anatomical passage are displayed using the multiple interactive windows; displaying a path through the anatomical passage to a medical procedure target; displaying a virtual image of an instrument within the anatomical passage; displaying a control point corresponding to a distal end of the instrument in at least one of the multiple interactive windows; receiving user input; identifying an orientation of the control point using the user input; and in response to receiving the user input, dynamically updating the orientation of the instrument in at least two of the multiple interactive windows to match the orientation of the control point.

[0044] One general aspect of the present disclosure includes a planning workstation, comprising: a display system; and a user input device; wherein the planning workstation is configured to: display image data via the display system; receive a first user input via the user input device; display a target of a medical procedure within the displayed image data identified at least based on the first user input via the display system; display an interactive image via the display system, the interactive image comprising the image data, a plurality of connected anatomical channels, and the identified target; receive a second user input via the user input device; display a trajectory between the target and an exit point of a nearest channel of the plurality of connected anatomical channels identified at least based on the second user input via the display system; receive a third user input via the user input device; and adjust the interactive image based at least on the defined trajectory and the third user input. Implementations may include one or more of the following features. The planning workstation, wherein the user input device includes a touch screen of the display system. The planning workstation, wherein the display system adjusts the interactive image including: determining a distance represented by the trajectory; determining whether the distance is greater than a predetermined threshold; receiving a fourth user input via the user input; identifying an unconnected channel that is closer to the target than the nearest connected channel based at least on the fourth user input; and connecting the unconnected channel to the plurality of connected channels. The planning workstation, wherein the planning workstation is further configured to receive a fifth user input via a user input device, and to rotate the interactive image based on at least the fifth user input to identify an unconnected channel in the interactive image. The planning workstation, wherein the interactive image rotates around one or more user-defined rotation points. The planning workstation, wherein the planning workstation is further configured to identify a rotation axis based on the one or more user-defined rotation points. The planning workstation, wherein the display system adjusts the interactive image includes: determining an exit angle based on the trajectory; and adjusting the exit angle by changing the orientation of the exit point along the nearest connected channel. The planning workstation, wherein the planning workstation is further configured to receive a fourth user input via a user input device, and to display a hazard of the medical procedure within the displayed image data via the display system based on the fourth user input. The planning workstation, wherein the hazard corresponds to at least one of a vulnerable portion of the patient's anatomy and an excessive curvature in one or more of a plurality of connected anatomical channels. The planning workstation, wherein the hazard is displayed using a hazard fence, wherein the hazard fence includes at least one of a disk, a conical hazard fence, and a hemispherical hazard fence. The planning workstation, wherein the user input device is configured to receive a first user input before segmentation of the image data is completed.

[0045] One general aspect of the present disclosure includes a non-transitory machine-readable medium including a plurality of machine-readable instructions, which, when executed by one or more processors associated with a planning workstation, are adapted to cause the one or more processors to perform a method, the method comprising: displaying image data via a graphical user interface; receiving a first user input; identifying a hazard within the displayed image data based at least on the first user input; and displaying an interactive image, the interactive image comprising the image data, a plurality of connected anatomical channels detected by segmentation of the image data, and the identified hazard. Implementations may include one or more of the following features. The non-transitory machine-readable medium, wherein the hazard comprises a vulnerable portion of a patient's anatomy. The non-transitory machine-readable medium, wherein the hazard comprises an excessive bend within a plurality of connected anatomical channels. The non-transitory machine-readable medium, wherein the hazard is represented using a hazard fence. The non-transitory machine-readable medium, wherein the hazard fence comprises at least one of a disk, a conical hazard fence, and a hemispherical hazard fence. The non-transitory machine-readable medium, wherein the plurality of connected anatomical channels comprises a pulmonary airway. The non-transitory machine-readable medium, wherein the machine-readable instructions are adapted to cause the one or more processors to perform the method, the method further comprising: receiving a second user input, identifying a target of a medical procedure within the displayed image data using at least the second user input, and wherein the hazard corresponds to a vulnerable portion of a patient anatomy proximate to the target. The non-transitory machine-readable medium, wherein the vulnerable portion of the patient anatomy comprises at least one of a pulmonary pleura, a blood vessel, a bulla, and a heart. The non-transitory machine-readable medium, wherein the machine-readable instructions are adapted to cause the one or more processors to perform the method, the method comprising receiving the first user input before segmentation of the image data is completed.

[0046] A general aspect of the present disclosure includes a non-transitory machine-readable medium including a plurality of machine-readable instructions which, when executed by one or more processors associated with a planning workstation, are suitable for causing the one or more processors to perform a method comprising: providing a plurality of interactive windows for a user to view a plan for a medical procedure, wherein each of the plurality of interactive windows displays a different rendering of an anatomical corridor model; displaying a path through the anatomical corridor to a target of the medical procedure; displaying a virtual image of an instrument within the anatomical corridor; displaying a control point corresponding to a distal end of an instrument in at least one of the plurality of interactive windows; receiving user input; identifying a position of the control point based at least on the user input; and in response to receiving the user input, dynamically updating the position of the instrument in at least one of the plurality of interactive windows to match the position of the control point.

[0047] One general aspect of the present disclosure includes a method for planning a medical procedure, the method comprising: receiving a representation of an anatomical passage including a plurality of branches; displaying an image of the representation via a graphical user interface; receiving a first user input indicating a selection of a first label; receiving a second user input indicating a selection of a first branch of the plurality of branches; and in response to the first user input and the second user input: marking the first branch with a first label; and displaying a representation of the first label applied to the first branch via the graphical user interface. Implementations may include one or more of the following features. The method further comprises, based on marking the first branch with the first label: selecting a second label; and displaying an indication via the graphical user interface that the second label has been selected. The method, wherein the second label is selected based on an arrangement of the plurality of branches within the anatomical passage. The method further comprises: receiving a third user input indicating a selection of a second branch of the plurality of branches; marking the second branch with a second label; and displaying a representation of the second label applied to the second branch via the graphical user interface. The method further comprises: identifying a branch group from the plurality of branches including the first branch; and in response to the first user input and the second user input, marking the branch group with the first label. The method, wherein identifying the branch group comprises identifying a descendant branch of the first branch and including the descendant branch in the branch group. The method, wherein identifying a branch group includes identifying an antecedent descendant branch of a first branch and including the antecedent branch in the branch group. The method further includes: displaying an indication of a second branch of the plurality of branches that does not have an assigned label via a graphical user interface. The method further includes: receiving a third user input indicating a selection of a second label; marking the second branch with a second label; and displaying a representation of the second label applied to the second branch via a graphical user interface. The method, wherein the representation of the anatomical channel is based on imaging data of the patient. The method, wherein the imaging data includes an anatomical structure and an image showing the representation of the anatomical channel displays the anatomical channel and the anatomical structure. The method further includes: in response to the first user input and the second user input: assigning a color to the first branch; and displaying an image of the representation of the anatomical channel with the first branch colored with the assigned color. The method further includes: providing a cursor via the graphical user interface; detecting that the cursor is aligned with a first branch of the plurality of branches; and based on detecting that the cursor is aligned with the first branch, modifying the representation of the cursor. The method further includes: receiving a third user input indicating a rotation instruction; in response to the third user input, rotating the representation of the anatomical channel; and displaying an image of the rotated representation of the anatomical channel via the graphical user interface.

[0048] One general aspect of the present disclosure includes a non-transitory machine-readable medium including a plurality of machine-readable instructions that, when executed by one or more processors, cause the one or more processors to perform operations including: displaying a representation of an anatomical passage via a graphical user interface, wherein the anatomical passage includes a plurality of branches; displaying a list of anatomical labels via a graphical user interface; receiving a first user input selecting a first label from the list of anatomical labels; receiving a second user input selecting a first branch of the plurality of branches; and applying the first label to the first branch. Implementations may include one or more of the following features. The non-transitory machine-readable medium, wherein the first branch is included in a branch group, the non-transitory machine-readable medium including further instructions that cause the one or more processors to apply the first label to the branch group based on the second user input selecting the first branch of the plurality of branches. The non-transitory machine-readable medium including further instructions that cause the one or more processors to identify the branch group by identifying descendant branches of the first branch and adding the descendant branches to the branch group. The non-transitory machine-readable medium including further instructions that cause the one or more processors to perform operations including: when applying the first label to the first branch, selecting a second label based on the arrangement of the branches within the anatomical passage. The non-transitory machine-readable medium includes further instructions that cause one or more processors to perform operations including: receiving a third user input selecting a second branch from the plurality of branches; and applying a second label to the second branch. The non-transitory machine-readable medium includes further instructions that cause one or more processors to perform operations including: when applying each label from the list of anatomical labels: identifying an unlabeled second branch from the plurality of branches; and displaying, via a graphical user interface, an indication that the second branch is unlabeled. The non-transitory machine-readable medium includes further instructions that cause one or more processors to perform operations including: receiving a third user input selecting a second label from the list of anatomical labels; and applying the second label to the second branch. The non-transitory machine-readable medium includes further instructions that cause one or more processors to perform operations including: displaying, via a graphical user interface, a status indicator for the second label indicating that the second label is assigned to more than one branch. The non-transitory machine-readable medium includes further instructions that cause one or more processors to perform operations including: displaying, via a graphical user interface, a representation of the first label applied to the first branch.The non-transitory machine-readable medium includes further instructions that cause one or more processors to perform operations including, based on applying the first label to the first branch: assigning a color to the first branch; and displaying, via a graphical user interface, a representation of an anatomical passage with the first branch colored with the assigned color. The non-transitory machine-readable medium, wherein the first branch is included in a branch group, includes further instructions that cause one or more processors to perform operations including, based on applying the first label to the first branch: assigning a color to the branch group; and displaying, via a graphical user interface, a representation of an anatomical passage with the branch group colored with the assigned color. The non-transitory machine-readable medium includes further instructions that cause one or more processors to: provide a cursor via a graphical user interface; detect alignment of the cursor with one of the plurality of branches; and based on detecting the cursor alignment, modify the representation of the cursor. The non-transitory machine-readable medium includes further instructions that cause one or more processors to, based on a first user input selecting a first label and a second user input selecting a first branch among a plurality of branches: compare the first label and the first branch with a second label applied to the second branch to determine whether the first label and the second label conflict; and when it is determined that the first label and the second label do not conflict, apply the first label to the first branch.

[0049] A general aspect of the present disclosure includes a planning workstation, comprising: a display system; and a user input device; wherein the planning workstation can be configured to display an anatomical channel including multiple branches via the display system; display a label list via the display system; receive a first user input selecting a first branch of the multiple branches via the user input device; receive a second user input selecting the first label from the label list via the user input device; and in response to the first user input and the second user input, display a representation of the first label applied to the first branch via the display system. Implementations may include one or more of the following features. The planning workstation, wherein the planning workstation is also configured to select a second label from the label list based on the arrangement of the branches within the anatomical channel in response to the first label being applied to the first branch. The planning workstation, wherein the planning workstation is also configured to perform operations including the following operations: identifying an unlabeled second branch of the multiple branches; and displaying an indication that the second branch is unlabeled via the display system. The planning workstation, wherein the planning workstation is further configured to, in response to a first user input selecting a first branch, perform an operation including the following operations: identifying a branch group from a plurality of branches including the first branch; applying a first label to the branch group; and displaying, via a display system, a representation of the first label applied to the branch group. The planning workstation, wherein the planning workstation is further configured to, in response to a first user input selecting the first branch, perform an operation including the following operations: determining whether the first label conflicts with the second label; and when determining that the first label does not conflict with the second label, applying the first label to the first branch.

[0050] Figure 1 FIG. 1 is a simplified diagram of a teleoperated medical system 100 according to some embodiments. In some embodiments, the teleoperated medical system 100 may be suitable for, for example, a surgical procedure, a diagnostic procedure, a therapeutic procedure, or a biopsy procedure. Figure 1 As shown, the medical system 100 generally includes a remotely operated manipulator assembly 102 for operating a medical instrument 104 to perform various procedures on a patient P. The remotely operated manipulator assembly 102 is mounted on or near an operating table T. A master control assembly 106 allows an operator (e.g., Figure 1 The surgeon, clinician or physician (2) is shown observing the intervention site and controlling the remotely operated manipulator assembly 102.

[0051] The main control assembly 106 can be located at the surgeon's console, which is typically located in the same room as the operating table T, such as on the side of the operating table where the patient P is located. However, it should be understood that the physician O can be located in a different room from the patient P or in a completely different building. The main control assembly 106 typically includes one or more control devices for controlling the remotely operated manipulator assembly 102. The control device can include any number of various input devices, such as a joystick, trackball, data glove, trigger gun, manual controller, voice recognition device, body movement or presence sensor and / or the like. In order to provide the physician O with a strong sense of direct control of the instrument 104, the control device can be provided with the same degrees of freedom as the associated medical device 104. In this way, the control device provides the physician O with a remote presentation / remote presence or perception that the control device and the medical device 104 are integrated.

[0052] In some embodiments, the control device may have more or fewer degrees of freedom than the associated medical instrument 104 and still provide telepresence to the physician O. In some embodiments, the control device may optionally be a manual input device that moves with six degrees of freedom and may also include an actuatable handle for actuating the instrument (e.g., for closing a grasping jaw, applying an electrical potential to an electrode, delivering a medication, and / or the like).

[0053] The teleoperated manipulator assembly 102 supports the medical device 104 and may include a kinematic structure of one or more non-servo controlled links (e.g., one or more links that can be manually positioned and locked in place, generally referred to as an assembly structure) and a teleoperated manipulator. The teleoperated manipulator assembly 102 may optionally include a plurality of actuators or motors that drive input devices on the medical device 104 in response to commands from a control system (e.g., the control system 112). The actuator may optionally include a drive system that, when coupled to the medical device 104, can advance the medical device 104 into a natural or surgically created anatomical orifice. Other drive systems can move the distal end of the medical device 104 with multiple degrees of freedom, which may include three linear motions (e.g., linear motion along X, Y, Z Cartesian axes) and three rotational motions (e.g., rotation around X, Y, Z Cartesian coordinate axes). In addition, the actuator may be used to actuate an articulated end effector of the medical device 104 for grasping tissue in a clamp of a biopsy device and / or the like. Actuator position sensors such as decoders, encoders, potentiometers, and other mechanisms can provide sensor data describing the rotation and orientation of the motor shaft to the medical system 100. This position sensor data can be used to determine the motion of an object being manipulated by the actuator.

[0054] The teleoperated medical system 100 may include a sensor system 108 having one or more subsystems for receiving information about an instrument of the teleoperated manipulator assembly 102. Such subsystems may include a position / location sensor system (e.g., an electromagnetic (EM) sensor system); a shape sensor system for determining the position, orientation, speed, velocity, posture, and / or shape along the distal end and / or one or more segments of the flexible body that may comprise the medical instrument 104; and / or a visualization system for capturing images from the distal end of the medical instrument 104.

[0055] The teleoperated medical system 100 also includes a display system 110 for displaying images or representations of the surgical site and medical instrument 104 generated by the subsystems of the sensor system 108. The display system 110 and the master control assembly 106 can be oriented so that the physician O can control the medical instrument 104 and the master control assembly 106 using the perception of the telepresentation.

[0056] In some embodiments, the medical device 104 may have a visualization system (discussed in more detail below) that may include a viewing mirror assembly that records instant or real-time images of the surgical site and provides the images to the operator or physician O through one or more displays of the medical system 100 (such as one or more displays of the display system 110). The instant images may be, for example, two-dimensional or three-dimensional images captured by an endoscope located within the surgical site. In some embodiments, the visualization system includes an endoscope component that may be integrally or removably coupled to the medical device 104. However, in some embodiments, a separate endoscope attached to a separate manipulator assembly may be used with the medical device 104 to image the surgical site. The visualization system may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which may include a processor of the control system 112.

[0057] The display system 110 can also display images of the surgical site and medical instruments captured by the visualization system. In some examples, the teleoperation medical system 100 can configure the controls of the medical instruments 104 and the master control assembly 106 so that the relative orientation of the medical instruments is similar to the relative orientation of the eyes and hands of the physician O. In this way, the physician O can manipulate the medical instruments 104 and the manual controls as if observing the workspace in a substantially telepresence situation. By telepresence, it means that the presentation of the image is a true perspective image that simulates the viewpoint of the physician who is physically manipulating the medical instrument 104.

[0058] In some examples, display system 110 can present images of a surgical site recorded preoperatively or intraoperatively using image data from imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermography, impedance imaging, laser imaging, nanotube x-ray imaging, and / or the like. The preoperative or intraoperative image data can be presented as a two-dimensional, three-dimensional, or four-dimensional (including, for example, time-based or velocity-based information) image and / or as an image from a representation (such as a model) created from the preoperative or intraoperative image data set.

[0059] In some embodiments, the display system 110 can display a virtual navigation image in which the actual position of the medical device 104 is registered (i.e., dynamically referenced) with a preoperative or immediate image or representation (e.g., a model), typically for the purpose of an imaging-guided surgical procedure. Doing so can present a virtual image of the internal surgical site to the physician O from the viewpoint of the medical device 104. In some examples, the viewpoint can be from the tip of the medical device 104. An image of the tip of the medical device 104 and / or other graphical or alphanumeric indicators can be superimposed on the virtual image to assist the physician O in controlling the medical device 104. In some examples, the medical device 104 may not be visible in the virtual image.

[0060] In some embodiments, the display system 110 can display a virtual navigation image in which the actual position of the medical device 104 is registered with the preoperative or immediate image to present a virtual image of the medical device 104 within the surgical site to the physician O from an external viewpoint. An image of a portion of the medical device 104 or other graphic or alphanumeric indicator can be superimposed on the virtual image to help the physician O control the medical device 104. As described herein, a visual representation of a data point can be rendered to the display system 110. For example, the measurement data points, movement data points, registration data points, and other data points described herein can be displayed on the display system 110 in a visual representation. The data points can be visually represented in a user interface by a plurality of points or spots on the display system 110, or can be visually represented as a rendered representation (e.g., a rendered model), such as a grid or line model created based on a set of data points. In some examples, the data points can be color-coded according to the data they represent. In some embodiments, the visual representation can be refreshed in the display system 110 after each processing operation has been implemented to change the data point.

[0061] The remote operation medical system 100 may also include a control system 112. The control system 112 includes at least one memory and at least one computer processor (not shown) for implementing control between the medical device 104, the main control component 106, the sensor system 108, and the display system 110. The control system 112 also includes programming instructions (e.g., a non-transitory machine-readable medium storing instructions) for implementing some or all of the methods described according to the various aspects disclosed herein, including instructions for providing information to the display system 110. Although in Figure 1 In the simplified schematic diagram of the control system 112 is shown as a single block, but the system may include two or more data processing circuits, wherein a portion of the processing is optionally performed on or near the remotely operated manipulator assembly 102 and another portion of the processing is performed at the main control assembly 106 and / or the like. The processor of the control system 112 can execute instructions, which include instructions corresponding to the processes disclosed herein and described in more detail below. Any of a variety of centralized or distributed data processing architectures can be employed. Similarly, the programming instructions can be implemented as multiple separate programs or subroutines, or they can be integrated into many other aspects of the remote operation system described herein. In one embodiment, the control system 112 supports wireless communication protocols such as Bluetooth, IrDA (infrared data communication), HomeRF (home radio frequency), IEEE 802.11, DECT (digital enhanced wireless communication), and wireless telemetry.

[0062] In some embodiments, the control system 112 may receive force and / or torque feedback from the medical device 104. In response to the feedback, the control system 112 may send a signal to the master control assembly 106. In some examples, the control system 112 may send a signal instructing one or more actuators of the remotely operated manipulator assembly 102 to move the medical device 104. The medical device 104 may extend to an internal surgical site in the body of the patient P via an opening in the body of the patient P. Any suitable conventional and / or specialized actuator may be used. In some examples, one or more actuators may be separate from or integrated with the remotely operated manipulator assembly 102. In some embodiments, one or more actuators and the remotely operated manipulator assembly 102 are provided as part of a remotely operated cart positioned adjacent to the patient P and the operating table T.

[0063] The control system 112 may optionally further include a virtual visualization system to provide navigation assistance to the physician O when controlling the medical device 104 during the image-guided surgical procedure. Virtual navigation using the virtual visualization system can be based on reference to a preoperative or intraoperative data set of an acquired anatomical channel. The virtual visualization system processes images of the surgical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluoroscopy, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or similar techniques. Software that can be used in conjunction with a manual input device is used to convert the recorded images into a segmented two-dimensional or three-dimensional composite representation of a portion or entire anatomical organ or anatomical region. The image data set is associated with the composite representation. The composite representation and the image data set describe the various positions and shapes of the channel and its connectivity. The images used to generate the composite representation can be recorded preoperatively or intraoperatively during the clinical procedure. In some embodiments, the virtual visualization system can use a standard representation (i.e., not patient-specific) or a mixture of a standard representation and patient-specific data. The composite representation and any virtual images generated by the composite representation may represent the static posture of the deformable anatomical region during one or more phases of motion (eg, during the inspiration / expiration cycle of the lungs).

[0064] During the virtual navigation procedure, the sensor system 108 can be used to calculate the approximate position of the medical device 104 relative to the anatomy of the patient P. This position can be used to generate both a macroscopic level (external) tracking image of the anatomy of the patient P and a virtual internal image of the anatomy of the patient P. The system can implement one or more electromagnetic (EM) sensors, fiber optic sensors, and / or other sensors to register the medical device with the preoperatively recorded surgical image and display the medical device with the preoperatively recorded surgical image together. For example, U.S. Patent Application No. 13 / 107,562 (filed on May 13, 2011) (disclosing "Medical System Providing Dynamic Registration of a Model of an AnatomicStructure for Image-Guided Surgery") discloses such a system, which is incorporated herein by reference in its entirety. The remotely operated medical system 100 may also include optional operating and support systems (not shown), such as lighting systems, steering control systems, irrigation systems, and / or suction systems. In some embodiments, the remotely operated medical system 100 may include more than one remotely operated manipulator assembly and / or more than one master control assembly. The exact number of teleoperated manipulator assemblies will depend on the surgical procedure and the space limitations in the operating room, as well as other factors. The master control assemblies 106 may be collocated or they may be located at different locations. Multiple master control assemblies allow more than one operator to control one or more teleoperated manipulator assemblies in various combinations.

[0065] Figure 2A 2 is a simplified diagram of a medical instrument system 200 according to some embodiments. In some embodiments, the medical instrument system 200 can be used as a medical instrument 104 in an image-guided medical procedure performed using a teleoperated medical system 100. In some examples, the medical instrument system 200 can be used for non-teleoperated exploratory procedures or procedures involving traditional manually operated medical instruments (such as endoscopy). Optionally, the medical instrument system 200 can be used to collect (i.e., measure) a set of data points corresponding to locations within an anatomical passage of a patient (such as patient P).

[0066] Medical device system 200 includes an elongated device 202 coupled to a drive unit 204. Elongated device 202 includes a flexible body 216 having a proximal end 217 and a distal end or tip portion 218. In some embodiments, flexible body 216 has an outer diameter of approximately 3 mm. Other flexible body outer diameters may be larger or smaller.

[0067] The medical device system 200 also includes a tracking system 230 for determining the position, orientation, speed, velocity, posture, and / or shape of the flexible body 216 along the distal end 218 and / or one or more segments 224 of the flexible body 216 using one or more sensors and / or imaging devices, as described in further detail below. The entire length of the flexible body 216 between the distal end 218 and the proximal end 217 can be effectively divided into the segments 224. If the medical device system 200 is consistent with the medical device 104 of the teleoperated medical system 100, the tracking system 230. The tracking system 230 can optionally be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which computer processors may include Figure 1 A processor of the control system 112 in the system.

[0068] The tracking system 230 may optionally track the distal end 218 and / or one or more segments 224 using a shape sensor 222. The shape sensor 222 may optionally include an optical fiber aligned with the flexible body 216 (e.g., provided within an internal channel (not shown) or mounted externally). In one embodiment, the diameter of the optical fiber is approximately 200 μm. In other embodiments, the size may be larger or smaller. The optical fiber of the shape sensor 222 forms a fiber optic bend sensor for determining the shape of the flexible body 216. In an alternative, an optical fiber including a fiber Bragg grating (FBG) is used to provide strain measurements in the structure in one or more dimensions. Various systems and methods for monitoring the shape and relative position of an optical fiber in three dimensions are described in U.S. Patent Application No. 11 / 180,389 (filed on July 13, 2005) (disclosing "Fiber optic position and shape sensing device and method relating thereto"); U.S. Patent Application No. 12 / 047,056 (filed on July 16, 2004) (disclosing "Fiber-optic shape and relative positions sensing"); U.S. Patent No. 6,389,187 (filed on June 17, 1998) (disclosing "Optical Fiber Bend Sensor"), which are incorporated herein by reference in their entirety. In some embodiments, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In some embodiments, the shape of the flexible body 216 may be determined using other techniques. For example, a history of the distal posture of the flexible body 216 may be used to reconstruct the shape of the flexible body 216 over a time interval. In some embodiments, tracking system 230 can optionally and / or additionally track distal end 218 using orientation sensor system 220. Orientation sensor system 220 can be a component of an EM sensor system, wherein orientation sensor system 220 includes one or more conductive coils that can be subjected to an externally generated electromagnetic field. Each coil of EM sensor system 220 then generates an induced electrical signal having characteristics that depend on the orientation and orientation of the coil relative to the externally generated electromagnetic field. In some embodiments, orientation sensor system 220 can be configured and positioned to measure six degrees of freedom (e.g., three position coordinates X, Y, Z and three orientation angles of pitch, yaw, and roll indicating a base point) or five degrees of freedom (e.g., three position coordinates X, Y, Z and two orientation angles of pitch and yaw indicating a base point).Further description of orientation sensor systems is provided in U.S. Patent No. 6,380,732 (filed August 11, 1999) (disclosing "Six-Degree of Freedom Tracking System Having a Passive Transponder on the Object Being Tracked"), which is incorporated herein by reference in its entirety.

[0069] In some embodiments, tracking system 230 may alternatively and / or additionally rely on historical pose, position, or orientation data stored for known points of the instrument system along a cycle of alternating motion (such as breathing). This stored data may be used to develop shape information about flexible body 216. In some examples, a series of position sensors (not shown), such as electromagnetic (EM) sensors similar to those in position sensors 220, may be positioned along flexible body 216 and then used for shape sensing. In some examples, the history of data acquired from one or more of these sensors during a procedure may be used to represent the shape of elongated device 202, particularly where anatomical passages are typically static.

[0070] Flexible body 216 includes a passageway 221 sized and shaped to receive a medical device 226 . Figure 2B2 is a simplified diagram of a flexible body 216 with an extended medical device 226 according to some embodiments. In some embodiments, the medical device 226 can be used for procedures such as surgery, biopsy, ablation, illumination, irrigation, or suction. The medical device 226 can be deployed through the passage 221 of the flexible body 216 and used at a target location in the anatomy. The medical device 226 may include, for example, an image capture probe, a biopsy instrument, a laser ablation fiber, and / or other surgical, diagnostic, or therapeutic tools. The medical tool may include an end effector having a single working member, such as a scalpel, a blunt blade, an optical fiber, an electrode, and / or the like. Other end effectors may include, for example, forceps, graspers, scissors, clip appliers, and / or the like. Other end effectors may further include an electrically activated end effector, such as an electrosurgical electrode, a transducer, a sensor, and / or the like. In various embodiments, the medical device 226 is a biopsy instrument, which can be used to remove sample tissue or cell sampling from a target anatomical location. The medical device 226 may also be used with an image capture probe within the flexible body 216. In various embodiments, the medical device 226 can be an image capture probe that includes a distal portion having a stereo camera or a monoscopic camera at or near the distal end 218 of the flexible body 216, the stereo camera or the monoscopic camera for capturing images (including video images) that are processed by the visualization system 231 for display and / or provided to the tracking system 230 to support tracking of the distal end 218 and / or one or more segments 224. The image capture probe can include a cable coupled to the camera for transmitting captured image data. In some examples, the image capture instrument can be a fiber optic bundle coupled to the visualization system 231, such as a fiberscope. The image capture instrument can be single-spectrum or multi-spectrum, which is used to capture image data in one or more of the visible light, infrared light, and / or ultraviolet light spectrums, for example. Alternatively, the medical device 226 itself can be an image capture probe. The medical device 226 can be advanced from the opening of the passage 221 to perform the procedure and then retracted into the passage when the procedure is completed. The medical device 226 may be removed from the proximal end 217 of the flexible body 216 or from another optional instrument port (not shown) along the flexible body 216 .

[0071] The medical device 226 may further house a cable, linkage, or other actuation control (not shown) extending between its proximal and distal ends to controllably bend the distal end of the medical device 226. Steerable instruments are described in detail in U.S. Patent No. 7,316,681 (filed October 4, 2005) (disclosing "Articulated Surgical Instrument for Performing Minimally Invasive Surgery with Enhanced Dexterity and Sensitivity") and U.S. Patent Application No. 12 / 286,644 (filed September 30, 2008) (disclosing "Passive Preload and Capstan Drive for Surgical Instruments"), which are incorporated herein by reference in their entirety.

[0072] The flexible body 216 may also house cables, linkages, or other steering controls (not shown) extending between the drive unit 204 and the distal end 218 to controllably bend the distal end 218, such as shown by a dashed line depiction 219 of the distal end 218. In some examples, at least four cables are used to provide independent pitch and "up and down" steering for controlling the distal end 218 and "left and right" steering for controlling the yaw of the distal end 281. Steerable catheters are described in detail in U.S. Patent Application No. 13 / 274,208 (filed on October 14, 2011) (disclosing "Catheter with Removable Vision Probe"), which is incorporated herein by reference in its entirety. In embodiments where the medical device system 200 is actuated by a remotely operated assembly, the drive unit 204 may include a drive input that is removably coupled to a drive element (such as an actuator) of the remotely operated assembly and receives power therefrom. In some embodiments, the medical device system 200 may include gripping features, manual actuators, or other components for manually controlling the movement of the medical device system 200. The elongated device 202 may be steerable, or alternatively, the system may be non-steerable without an integrated mechanism for operator-controlled bending of the distal end 218. In some examples, one or more lumens are defined in the wall of the flexible body 216 through which a medical device may be deployed and used at a target surgical site.

[0073] In some embodiments, the medical device system 200 may include a flexible bronchial instrument, such as a bronchoscope or bronchial catheter, for inspection, diagnosis, biopsy, or treatment of the lungs. The medical device system 200 is also suitable for navigating and treating other tissues in various anatomical systems via natural or surgically created connecting passages, including the colon, intestines, kidneys and renal tubules, brain, heart, circulatory system (including vascular system), and / or the like.

[0074] Information from tracking system 230 may be sent to navigation system 232 where it is combined with information from visualization system 231 and / or preoperatively acquired representations (e.g., models) to provide real-time position information to a physician, clinician or surgeon or other operator. In some examples, the real-time position information may be displayed on a Figure 1 on the display system 110 for controlling the medical device system 200. In some examples, Figure 1 The control system 116 can use the position information as feedback for positioning the medical instrument system 200. Various systems for registering and displaying surgical instruments with surgical images using fiber optic sensors are provided in U.S. Patent Application No. 13 / 107,562 filed on May 13, 2011, which discloses "Medical System Providing Dynamic Registration of a Model of an Anatomic Structure for Image-Guided Surgery," which is incorporated herein by reference in its entirety.

[0075] In some examples, the medical device system 200 may be Figure 1 In some embodiments, Figure 1 The teleoperated manipulator assembly 102 of the apparatus may be replaced by direct operator controls. In some examples, the direct operator controls may include various handles and operator interfaces for handheld operation of the apparatus.

[0076] Figure 3A and 3B FIG. 4 is a simplified diagram of a side view of a patient coordinate space including a medical device mounted on an insertion assembly according to some embodiments. Figure 3A and 3BAs shown, the surgical environment 300 includes a patient P positioned on a platform 302. The patient's overall movement is limited by sedation, restraints and / or other means, in which sense, the patient P can be stationary in the surgical environment. Cyclic anatomical movement, including breathing and cardiac movement of the patient P, can continue unless the patient is required to hold his or her breath to temporarily suspend respiratory movement. Therefore, in some embodiments, data can be collected at a specific stage of breathing, and the data can be marked and identified with the stage. In some embodiments, the stage of collecting data can be inferred from the physiological information collected by the patient P. Within the surgical environment 300, a point collection instrument 304 is coupled to an instrument bracket 306. In some embodiments, the point collection instrument 304 can use an EM sensor, a shape sensor, and / or other sensor modalities. The instrument bracket 306 is mounted to an insertion table 308 fixed within the surgical environment 300. Alternatively, the insertion table 308 can be movable, but has a known position within the surgical environment 300 (e.g., via a tracking sensor or other tracking device). The instrument bracket 306 can be a component of a teleoperated manipulator assembly (e.g., teleoperated manipulator assembly 102) that is coupled to the point collection instrument 304 to control the insertion motion (i.e., motion along the A axis) and optionally control the motion of the distal end 318 of the elongated device 310 in multiple directions (including yaw, pitch, and roll). The instrument bracket 306 or insertion stage 308 can include an actuator, such as a servo motor (not shown), which controls the motion of the instrument bracket 306 along the insertion stage 308.

[0077] The elongated device 310 is coupled to an instrument body 312. The instrument body 312 is coupled and fixed relative to the instrument bracket 306. In some embodiments, a fiber optic shape sensor 314 is fixed at a proximal point 316 on the instrument body 312. In some embodiments, the proximal point 316 of the fiber optic shape sensor 314 can move with the instrument body 312, but the position of the proximal point 316 can be known (e.g., via a tracking sensor or other tracking device). The shape sensor 314 measures the shape from the proximal point 316 to another point (such as the distal end 318 of the elongated device 310). The point collection instrument 304 can be substantially similar to the medical instrument system 200.

[0078] The position measurement device 320 provides information about the position of the instrument body 312 as it moves along the insertion axis A on the insertion stage 308. The position measurement device 320 may include a decoder, an encoder, a potentiometer, and / or other sensor that determines the rotation and / or orientation of the actuator that controls the movement of the instrument bracket 306 and thus the movement of the instrument body 312. In some embodiments, the insertion stage 308 is linear. In some embodiments, the insertion stage 308 may be curved or have a combination of curved and linear sections.

[0079] Figure 3A The instrument body 312 and the instrument bracket 306 are shown in a retracted orientation along the insertion stage 308. In this retracted orientation, the proximal point 316 is located at an orientation L0 on the axis A. In this orientation along the insertion stage 308, the A component of the position of the proximal point 316 can be set to zero and / or another reference value to provide a baseline reference to describe the orientation of the instrument bracket 306 on the insertion stage 308 and therefore describe the orientation of the proximal point 316 on the insertion stage 308. With this retracted orientation of the instrument body 312 and the instrument bracket 306, the distal end 318 of the slender device 310 can be positioned exactly within the access orifice of the patient P. Also in this orientation, the orientation measurement device 320 can be set to zero and / or another reference value (e.g., I=0). Figure 3B 308, the instrument body 312 and the instrument bracket 306 have been advanced along the linear track of the insertion table 308, and the distal end 318 of the elongated device 310 has been advanced into the patient P. In this advanced position, the proximal point 316 is at the position L1 on the axis A. In some examples, encoders and / or other position data from one or more actuators that control the movement of the instrument bracket 306 along the insertion table 308 and / or one or more position sensors associated with the instrument bracket 306 and / or the insertion table 308 are used to determine the position Lx of the proximal point 316 relative to the position L0. In some examples, the position LX can further be used as an indicator of the distance or insertion depth of the distal end 318 of the elongated device 310 inserted into the passage of the anatomy of the patient P.

[0080] In illustrative applications, a medical device system such as medical device system 200 can include a robotic catheter system for a lung biopsy procedure. The catheter of the robotic catheter system provides a catheter for a tool, such as an endoscope, an endobronchial ultrasound (EBUS) probe, and / or a biopsy tool, which is delivered to a position in the airway where one or more targets of a lung biopsy, such as lesions, nodules, tumors, and / or the like, are present. When the catheter is driven through the anatomy, an endoscope is typically installed so that a clinician (such as surgeon O) can monitor the real-time camera feed at the distal end of the catheter. Real-time camera feed and / or other real-time navigation information can be displayed to the clinician via a graphical user interface. An example of a graphical user interface for monitoring a biopsy procedure is described in U.S. Provisional Patent Application No. 62 / 486,879, entitled "Graphical User Interface for Monitoring an Image-Guided Procedure and ", filed April 18, 2017, which is incorporated herein by reference.

[0081] Before using a robotic catheter system to perform a biopsy procedure, a preoperative planning step can be performed to plan the biopsy procedure. The preoperative planning step can include segmentation of image data (such as a patient CT scan) to create a 3D model of the anatomy, select a target within the 3D model, determine the airway in the model, expand the airway to form a connected airway tree, and plan the trajectory between the target and the connected tree. One or more of these steps can be performed on the same robotic catheter system used to perform the biopsy. Alternatively or additionally, planning can be performed on different systems (e.g., a workstation dedicated to preoperative planning). The planning of the biopsy procedure can be saved (e.g., as one or more digital files) and transmitted to the robotic catheter system used to perform the biopsy procedure. The saved planning can include the 3D model, identification of the airway, the target location, the trajectory to the target location, the route through the 3D model and / or the like.

[0082] An illustrative embodiment of a graphical user interface for planning medical procedures is provided below, including but not limited to the lung biopsy procedure described above. The graphical user interface can include multiple modes, including a data selection mode, a hybrid segmentation and planning mode, a preview mode, a save mode, a management mode, and a review mode. Some aspects of the graphical user interface are similar to the features described in U.S. Provisional Patent Application No. 62 / 357,217, entitled “Graphical User Interface for Displaying Guidance Information During and Image-Guided Procedure,” filed on June 30, 2016, and U.S. Provisional Patent Application No. 62 / 357,258, entitled “Graphical User Interface for Displaying Guidance Information in a Plurality of Modes During and Image-Guided Procedure,” filed on June 30, 2016, the entire contents of which are incorporated herein by reference.

[0083] Figure 4-Figure 9 4 is a simplified diagram of a graphical user interface 400 that may be displayed on a display system such as display system 110 and / or a display system of a stand-alone planning workstation according to some embodiments. Graphical user interface 400 displays information associated with planning a medical procedure in one or more views viewable by a user (such as surgeon O). Figure 4-9, it will be appreciated that graphical user interface 400 may display any suitable number of views in any suitable arrangement and / or on any suitable number of screens. In some examples, the number of views displayed simultaneously may be changed by opening and closing views, minimizing and maximizing views, moving views between the foreground and background of graphical user interface 400, switching between screens, and / or otherwise completely or partially obscuring views. Similarly, the arrangement of views (including their size, shape, orientation, ordering (in the case of overlapping views), and / or the like) may vary and / or may be user-configurable.

[0084] In some examples, graphical user interface 400 may include one or more headers, footers, sidebars, menus, message bars, pop-up windows, and / or the like. Figure 4-9 As depicted, graphical user interface 400 includes a dynamic header 410 that updates based on the mode of graphical user interface 400. In various examples, header 400 may include a drop-down control menu, a page title, navigation controls (e.g., a continue button and / or a back button), patient information, a search bar, and / or the like.

[0085] Figure 4 4 shows a graphical user interface 400 in a data selection mode according to some embodiments. The data selection mode is used to select a data source, patient, and / or image data to be used in planning a medical procedure. Thus, the graphical user interface 400 in the source selection mode may include a data source selector 420, a patient selector 430, and a data selector 440. Figure 4As depicted, data source selector 420 includes options for loading data from a USB device, a DVD, and / or a network. It should be understood that data can be loaded from various other sources (including external and / or local sources) (e.g., a local hard drive). Patient selector 430 includes a list of patients whose image data can be obtained from the selected data source. Various patient attributes can be displayed in the list, including patient name, gender, date of birth, unique patient ID, and / or the like. Data selector 440 includes a list of image data available for the selected patient from the selected data source. Various attributes of the data can be displayed in the list, including a description of the data, the date on which the data was acquired, and / or a suitability rating indicating the suitability of the image data for planning a medical procedure. Suitability ratings can be qualitative and / or quantitative, and can be assigned manually and / or automatically. Ratings can be presented as numerical scores, stars, percentiles, symbolic representations, and / or the like. In some examples, suitability ratings can be determined based on the quality of the imaging technology used to acquire the image data. Once the image data is selected, the user can continue to plan the medical procedure using the selected image data. For example, the user may click and / or tap a load button of navigation panel 450 to continue.

[0086] Figure 5A-5G A graphical user interface 400 in a hybrid segmentation and planning mode according to some embodiments is shown. Segmentation is the process of analyzing image data (such as image data selected in a data selection mode) and creating a 3D model from the data. An example of an automated technique for performing CT data segmentation is described in U.S. Patent Application No. 14 / 845,031, entitled "Systems and Methods for Pre-Operative Modeling," which is incorporated herein by reference in its entirety. The segmentation process typically occurs within a time period (e.g., one to three minutes), which may vary depending on many factors (including the quality of the CT image data, the size and / or complexity of the CT image data, the level of detail in the 3D model, available computing resources, and / or the like). In some examples, the hybrid segmentation and planning mode of the graphical user interface 400 may allow a user to plan a medical procedure based on the image data and / or the 3D model while the segmentation process is occurring and before the 3D model is completed. Therefore, the process of planning a medical procedure may be accelerated because the user is able to begin planning a medical procedure without waiting for the potentially lengthy segmentation process to complete.

[0087] In some embodiments, the graphical user interface 400 in the hybrid segmentation and planning mode may be divided into one or more frames. Figure 5A-5GAs shown, the graphical user interface 400 includes a control frame 510 and a canvas frame 520. The control frame 510 provides a set of controls and / or indicators for planning medical procedures. In some examples, the control frame 510 may provide controls for adding, viewing, modifying and / or deleting one or more features (such as targets, paths, airways, trajectories and / or hazards) of the model and / or plan. In some examples, the control frame 510 may provide controls to undo and / or redo the most recent changes to the plan. In some examples, the control frame 510 may provide a segmentation progress indicator 515 based on the degree of progress along the segmentation process. The segmentation progress indicator 515 may be formatted as a progress bar, an elapsed time indicator, an estimated remaining time indicator and / or any other suitable segmentation progress indicator. In some embodiments, when the segmentation is completed, the segmentation progress indicator 515 may disappear.

[0088] In some embodiments, the graphical user interface 400 in the hybrid segmentation and planning mode may include a canvas frame 520. Figure 5A-5G As shown, the canvas frame 520 provides a workspace 522 for selecting, viewing, and / or interacting with image data and / or model data. Illustrative functions that can be performed via the workspace 522 include adding, modifying, and / or deleting planned features (e.g., targets, paths, and / or hazards), manipulating the 3D model, and / or verifying the accuracy of the segmentation process. To accommodate these functions, the workspace 522 can be switched between multiple interactive views, including a selection view, one or more image views, and one or more model views.

[0089] In some examples, the canvas frame 520 may include a tool selector 524 that provides a list of available tools. Figure 5A-5GAs depicted, the tool list includes a move tool, a magnifying glass tool, a window / level tool, an object drawing tool, a line drawing tool, a trim tool, a hazard tool, an angle and / or distance measurement tool, an undo / redo tool, and / or similar tools. In some examples, certain tools may be enabled and / or disabled based on the current view displayed in the workspace 522. For example, tools that are not used by the current view may be hidden, grayed out, and / or otherwise unselectable. In some examples, clicking a tool may cause a menu to display a list of sub-tools. For example, an object drawing tool may include sub-tools for drawing various 2D and / or 3D objects (such as free-form objects, predefined 2D shapes (e.g., circles, rectangles, ellipses, etc.), 3D shapes (e.g., spheres, 3D ellipsoids, etc.), and / or the like). In some examples, the tool selector 524 may include tools for semi-automatically detecting objects in the underlying image data (e.g., clicking a point in the image data and automatically identifying the corresponding object using edge detection techniques). Although tool selector 524 is depicted as a sidebar, it should be understood that tool selector 524 can be positioned and / or displayed in a variety of formats, including a palette, a header, a footer, a drop-down menu, an auto-hide menu, and / or the like. In some embodiments, tool selector 524 can be omitted, such as when tool selection is performed using keyboard shortcuts.

[0090] Figure 5A An example of a selection view displayed in workspace 522 is shown. In the selection view, a set of selections 531-536 are presented as a grid of thumbnails. Each of the selections 531-536 corresponds to a different rendering of image data and / or representation (e.g., model) data. The renderings can vary based on their perspective, zoom level, data type, style, and / or the like. For example, renderings of image data can be provided from a variety of perspectives, including transverse, sagittal, coronal, and / or virtual endoscopic perspectives. Figure 5AIn the depicted example, the selections 535 and 536 corresponding to the rendered representation data display a waiting indicator because the segmentation is not yet complete and the representation diagram is not ready for display. On the other hand, the selections 531-534 corresponding to the rendering of the image data are filled with actual image data because the graphical user interface 400 allows interactive input of the image data to be displayed and / or received before the segmentation is completed. Once the selection of the rendering is received, the selected rendering, the received interactive input, or both can be displayed via an interactive window. For example, user input (such as a single click or click) can be received via an extended view button 537 of the selected rendering to continue. In some examples, the extended view button 537 can appear when a user or a user holding object hovering over the corresponding selection is detected, otherwise the extended view button 537 disappears. Although the selections 531-536 are depicted as being arranged in a thumbnail grid, various alternatives are possible, such as a list of selections.

[0091] Figure 5B-5F An example of an interactive window 541 displayed in workspace 522 is shown. Interactive window 541 displays a rendering selected using a selection view. In some examples, a selection sidebar 542 can be displayed alongside interactive window 541 to allow a user to change to a different rendering without returning to the selection view. For example, selection sidebar 542 can display a scrollable bar of thumbnail images generally corresponding to selections 531-536, with the current selection identified by a blue border.

[0092] like Figure 5B As depicted, raw image data 543 (e.g., CT image data) is displayed as a first color palette, such as grayscale, and segmentation data 544 (e.g., detected airways) is displayed as a contrasting color or shade, such as pink. In some examples, while segmentation is still in progress, segmentation data 544 can be dynamically updated to reflect the progress of segmentation. For example, as new airways are detected over time, new pink areas can dynamically appear in interactive window 541.

[0093] In some examples, the interactive window 541 can display one or more features of the plan of the medical procedure, such as goals, paths, and / or hazards. These features can include features based on user input, automatically extracted features, semi-automatically extracted features, and / or the like. According to some embodiments, changes to one or more features made in a particular rendering can be dynamically propagated to other renderings. For example, a target added in one rendering can automatically appear in other renderings, including selecting a thumbnail image of the sidebar 542.

[0094] like Figure 5B, the features include a target 550 identified using a circular tool on a lesion visible in the underlying image data. The size, shape, and / or orientation of the target 550 can be adjusted to capture the shape of the lesion at a desired level of accuracy. When the size, shape, and / or orientation of the target 550 is adjusted, statistics 552 corresponding to the target 550 are updated in the control frame 510. In some examples, the target 550 can be named, renamed, and / or deleted using controls provided in the control frame 510. In some examples, after a first target has been identified by detecting a corresponding user input via the control frame 510, the controls provided in the control frame 510 can identify additional targets. Additionally or alternatively, the tool selector 524 may include one or more tools for adding, modifying, and / or deleting targets. In some examples, identifying the target 550 in one rendering can automatically cause an updated representation of the target 550 to appear in other renderings. Thus, parameters of the target 550 can be adjusted from multiple perspectives based on detecting user input from available renderings that can be displayed switchably.

[0095] exist Figure 5C , the interactive window 541 includes a trajectory 560 between the target 550 and an exit position 562. The exit position 562 corresponds to the point at which the medical instrument leaves the anatomical channel detected by the segmentation process to reach the target 550. In some examples, the exit position 562 is the closest point from the closest anatomical channel to the target 550. The trajectory 560 represents the trajectory of the medical instrument located at the exit position 562 to perform one or more interventional steps at the target 550. For example, the instrument can pierce the lumen of the anatomical channel at the exit position. For example, the medical instrument can include a biopsy needle, an ablation tool, a chemical delivery system, an ultrasound probe, and / or the like. In some examples, the medical instrument can have a maximum trajectory length. For example, a biopsy needle may not be able to perform a biopsy at a target that is more than 3 cm away from the exit position 562. Therefore, when the length of the trajectory 560 is greater than 3 cm, the graphical user interface 400 can display an out-of-range warning 564. In some embodiments, the out-of-range warning 564 can be provided based on a threshold value, which can include a fixed threshold value and / or a variable threshold value, which is set based on, for example, the type of tool to be used to enter the target. For example, a biopsy needle may provide a different insertion depth than an ablation tool or imaging probe, in which case the threshold may vary accordingly. In another example, different types of biopsy needles may provide different insertion depths. The user may input the type of tool being used, or the system may automatically detect the tool. Figure 5C As depicted, out of range warning 564 is presented as a message in control frame 510 .

[0096] In some embodiments, multiple trajectories to a given target may be identified, such as alternative trajectories to be used if trajectory 560 is found to be unreachable and / or otherwise unsuitable for use during a medical procedure. Consistent with such embodiments, control frame 510 may include controls for adding alternative trajectories to target 550. Additionally or alternatively, tool selector 524 may include one or more tools for adding, modifying, and / or deleting trajectories.

[0097] exist Figure 5D-5F , the interactive window 541 includes a hazard fence 570. The hazard fence 570 is used to facilitate trajectory planning by identifying vulnerable portions of the anatomy located near a target location. Examples of vulnerable portions of the anatomy may include blood vessels, pulmonary pleura, bullae, and / or the like. For example, puncturing the pulmonary pleura during a medical procedure may present a patient with a dangerous pneumothorax. Consistent with such embodiments, an exit location and / or a trajectory between an exit location and a target location may be constrained to avoid vulnerable portions of the anatomy. For example, a trajectory may be invalid when the trajectory passes within a threshold distance of a vulnerable portion of the anatomy, destroys a vulnerable portion of the anatomy, and / or the like. In Figure 5D-5F In the depicted example, danger fence 570 provides a warning to protect a portion of the lung pleura near the target location from puncture when using the planned trajectory. Figure 5D-5F As depicted, hazard fence 570 is placed using the hazard tool of tool selector 524. Additionally or alternatively, hazard fences may be added, modified, and / or removed using controls presented in control frame 510.

[0098] Figure 5D-5F Each of the figures in FIG. 5 shows a different type of hazard barrier 570. Figure 5D , hazard fence 570 is shown as a planar hazard fence with a pair of control points 571 and 572 used to define a disk in three dimensions. To convey the three-dimensional aspect of the disk, the portion of the disk that protrudes from interactive window 541 may be rendered in a solid color, while the portion of the disk that is projected into interactive window 541 may be rendered in a faded color and / or a semi-transparent color. Figure 5E In FIG. 5 , hazard barrier 570 is shown as a conical hazard barrier, wherein a pair of outer control points 573 and 574 are used to define the three-dimensional disk as the base of the cone and a vertex control point 575 is used to define the cone height. Fig. 5F In FIG. 5 , the hazard fence 570 is shown as a hemispherical hazard fence, wherein three control points 576-578 are used to define the hemisphere. Figure 5E and 5F, interactive window 541 further includes target 550 and trajectory 560. When trajectory 560 is connected to an exit location that is not in the plane of the underlying image (i.e., when trajectory 560 is projected into interactive window 541 and / or interactive window 541 is highlighted), projection 579 is displayed to link trajectory 560 to the modeled channel.

[0099] Various other types of hazards can be identified and marked using suitable indicators (such as hazard fences 570). For example, an anatomical passage can produce a tight bend that cannot be passed through by certain medical instruments (such as a biopsy needle and / or a catheter). Therefore, a blocking mark can be used to indicate the bend so that the user knows to plan different routes to the target that avoid bends. Automatic, manual and / or semi-automatic techniques can be used to determine whether the planned route includes any bends that are too tight. For example, taking into account the known physical properties of various medical instruments used in medical procedures, a bend radius that is too tight can be automatically identified. Additionally or alternatively, the user can visually identify a bend that appears too tight, and / or measurements can be performed to confirm that the bend is too tight. In some examples, candidate routes can be automatically sorted based on user-defined route rules and / or route feasibility features (such as the length of the route, the most compact bend encountered in the route, the width of the channel along the route, the length of the trajectory between the end of the route and the target, and / or the like). Therefore, the user can select among the candidate routes based on the ranking.

[0100] exist Figure 5G In the example, interactive window 541 displays a representation (e.g., model) 580 corresponding to a 3D model generated by segmentation of image data. In some examples, representation 580 may not be available until segmentation is complete. In some examples, when segmentation is incomplete, a partial version of model 580 may be displayed and may be updated in real time to reflect the ongoing progress of segmentation. In some embodiments, border 581 may be displayed around model 580 as a semi-transparent pattern, a wireframe pattern, and / or the like. Figure 5G As depicted, boundary 581 corresponds to the lung pleura. The appearance (e.g., color, size, texture, and / or the like) of boundary 581 can be changed to identify various features. For example, boundary 581 can be colored red to indicate the orientation of danger fence 570. Various features can also be depicted in interactive window 541, including target 550, trajectory 560, exit position 562, and / or the like. According to some embodiments, interactive window 541 may include orientation icon 582 to identify the viewing angle of model 580 relative to the patient's body. In some examples, the appearance of the channels along the route to target 550 in model 580 can be changed to indicate that they are along the route. For example, the channels on the route can be colored blue, while other channels in model 580 can be colored gray.

[0101] In some examples, an exit angle selector 583 may be provided in the control frame 510. The exit angle selector 583 provides an adjustment control, such as a slider, to adjust the orientation of the exit location 562 along the anatomical passage. Adjusting the orientation of the exit location 562 causes a corresponding adjustment of the exit angle of the trajectory 560 relative to the anatomical passage. In some examples, it may be desirable to set the exit angle of the trajectory 560 based on various factors and / or metrics, such as a default or "rule of thumb" exit angle (e.g., 45 degrees), the distance between the exit location 562 and the target 550, and / or the distance between the exit location 562 and the hazard fence 570. Thus, the exit angle selector 583 can speed up the process of defining the trajectory 560 by allowing a user to quickly test the exit angle range and confirm that the relevant metrics fall within an acceptable range. For example, the exit angle selector 583 can display the value of the exit angle (e.g., 37 degrees in the example provided), the distance from the exit location 562 to the target 550 (e.g., 2.4 cm in the example provided), and / or the distance from the exit location 562 to the hazard fence 570 (e.g., 7.4 cm in the example provided). The appearance (e.g., color, texture, size, font, etc.) of the exit angle selector 583 can change to warn the user when one or more related metrics are not within a predetermined range and / or do not meet a predetermined threshold. In some examples, one or more values ​​of the angle adjustment slider can be disabled when it is determined that the value is outside an acceptable range.

[0102] Figure 6A graphical user interface 400 in preview mode is shown according to some embodiments. The preview mode is used to preview the plan of a medical procedure prepared in the hybrid segmentation and planning mode. The graphical user interface 400 in preview mode displays a simulated live endoscopic view 610, a matching virtual endoscopic view 620, a global anatomical representation (e.g., model) view 630, and a reduced anatomical representation (e.g., model) view 640. The simulated live endoscopic view 610 and the virtual endoscopic view 620 depict a rendering of a representation (e.g., model) from inside an anatomical passage. The rendering is from the perspective of a virtual endoscope following the route of the planned medical procedure. The simulated live endoscope view 610 and the virtual endoscope view 620 are generally similar, except that the simulated live endoscope view 610 includes realistic details (e.g., blood vessels in the anatomical lumen) to simulate an actual camera feed from an endoscope, while the virtual endoscope view 620 is enhanced with directional cues toward a target location (such as contour lines 621, route lines, arrows, and / or the like). In the event that an anatomical channel branches, the channel may be highlighted in the virtual endoscope view 620 to indicate the direction the user should turn. One or more of the simulated live endoscope view 610 and the virtual endoscope view 620 may display various trajectory metrics, such as the remaining distance to the target location and / or hazards.

[0103] The global anatomical model view 630 generally corresponds to Figure 5G 3D perspective view of the model 580 depicted in FIG. Figure 6 As depicted in , the global anatomical model view 630 includes a model rendering 631, which includes an anatomical model, a boundary, multiple target locations, a hazard fence, and an orientation icon. The anatomical model view 630 further includes a depiction of a catheter 632 as a green line. The endoscope views 610 and 620 provide matching views from the distal end of the catheter 632.

[0104] In some embodiments, the graphical user interface 400 in preview mode can display a reduced anatomical model view 640. Reduced anatomical model view 640 provides a simplified overview of the planned route of the medical procedure including the key anatomical features of the route. Route path 641 is represented as a straight line. The depiction of catheter 632 is overlaid on route path 641 to indicate the progress of catheter 632 along route path 641. Anatomical channel 643 is rendered as a 2D layered projection to provide a simplified indication of the width of channel 643. Branches 644 are rendered to show where they are connected to channel 643, but other details of branches 644, such as their various sub-branches, are not rendered. Target icons 645 indicating exit angles and / or nearby hazards are located at the distal end of route path 641. When the planning of the medical procedure includes multiple targets and / or paths, a selector 646 is included to switch between multiple targets and / or paths. An embodiment of a reduced anatomical representation (e.g., model) view is further discussed in U.S. Provisional Patent Application No. 62 / 486,879, which is incorporated herein by reference.

[0105] like Figure 6 , a zoomed-out anatomical model view 640 is used as a controller to allow a user to navigate through a preview of the route. In particular, the distal end of the catheter 632 includes a control point 650 that can be dragged back and forth along the route path 641. As the control point 650 is dragged back and forth, the endoscopic views 610 and 620 are updated to reflect the viewpoint of the catheter 632 at the updated position of the control point 650. In addition, the depiction of the catheter 632 in the global anatomical model view 630 is updated to reflect the shape of the catheter 632 at the updated position of the control point 650. Figure 6 In the depicted example, control point 650 is shown as a triangular pyramid that represents the projected view of the endoscope from the distal end of the catheter. In alternative embodiments, control point 650 may be a variety of shapes and sizes.

[0106] Once the planned one or more routes have been previewed, the clinician can continue to save the plan. For example, the clinician can click and / or tap the next button of the header 410 to continue. Alternatively, the clinician can return to the early stages of the planning process to make changes as needed.

[0107] Figure 7A graphical user interface 400 is shown in a save mode according to some embodiments. The save mode is used when the planned medical procedure is completed and / or ready to be transferred to a medical device to perform the medical procedure. A set of options 710 are presented via the graphical user interface 400. The options 710 may include a transfer option, a discard option, a delete option, and / or a save option. The save options may include saving the plan locally, saving to an external device, and / or sending the plan to, for example, a cloud storage facility via a network. One or more options may require the installation of an external storage device. For example, the transfer option may require a storage device (e.g., a USB device) that is compatible with the medical device to which the plan will be transferred. Accordingly, a message 720 may be displayed to notify the user of applicable storage device requirements.

[0108] Figure 8 The graphical user interface 400 is shown in a management mode according to some embodiments. The management mode is used to manage available plans. Available plans can be stored locally, can be stored on an external drive, and / or can be downloaded over a network. A selection grid 810 is displayed, which includes a thumbnail representation of the plan (e.g., a rendering of a representation (e.g., a model), a planned route, a target location, and / or the like). Additionally or alternatively, the selection grid 810 may include patient data, such as the patient's name, date of birth, and / or the like. In some examples, the program being planned can be reloaded and viewed using the preview mode of the graphical user interface 400. In some examples, the program being planned can be saved at any time and reloaded using the hybrid segmentation and planning mode of the graphical user interface 400 to continue working at a later time. Therefore, the selection grid 810 may include an indication of the status of each plan (e.g., transfer, planning, start, and / or the like) and / or the time when the plan was last saved. In some examples, the selected plan can be viewed, deleted, and / or transferred. In some embodiments, the selection grid 810 may include an option to create a new plan. When a new plan is selected, the graphical user interface 400 may advance to the Figure 4 The data selection mode described in .

[0109] Fig. 9A graphical user interface 400 in review mode according to some embodiments is shown. Review mode is used to review a record of a completed medical procedure. After a medical procedure is performed using a given plan, a record of the procedure can be saved and transmitted to a planning workstation. In some embodiments, the recorded procedure file can include a video of live endoscopic images captured during the procedure, associated virtual endoscopic images, an anatomical representation (e.g., a model) showing catheter movement during the procedure, notes made by a clinician during the procedure, and / or the like. Thus, a viewer 910 can be displayed that includes playback controls (e.g., play, pause, zoom, and / or the like), snapshot controls, annotation and / or bookmark controls, and / or the like.

[0110] Fig.10 is a simplified diagram of a method 1000 for planning a medical procedure according to some embodiments. Figure 1-Figure 9 Consistent with some embodiments, method 1000 may be used to operate a graphical user interface, such as graphical user interface 400, in a variety of modes, including a data selection mode, a hybrid segmentation and planning mode, a preview mode, a save mode, a management mode, and a review mode. In some embodiments, the graphical user interface is interactive and may receive user input via a mouse, keyboard, touch screen, stylus, trackball, joystick, voice commands, virtual reality interface, and / or the like.

[0111] At process 1010, data is selected via a graphical user interface in a data selection mode. According to some embodiments, selecting data includes selecting a data source using a data source selector (such as data source selector 420), selecting a patient using a patient selector (such as patient selector 430), and selecting data using a data selector (such as data selector 440). The selection may be confirmed by using a load button on the graphical user interface. The data may include imaging data, such as CT data and / or any other type of imaging or patient data.

[0112] At process 1020, a medical procedure is planned via a graphical user interface in a hybrid segmentation and planning mode. According to some embodiments, the data selected at process 1010 includes image data that is segmented to generate an anatomical representation (e.g., a model) based on the extracted channels. Simultaneously during segmentation, the medical procedure is planned by receiving user input that defines features of the plan (e.g., targets, hazards, and / or paths). In some examples, an interactive window such as interactive window 541 can provide a user with an interface for adding, modifying, and / or deleting features from the plan. When the segmentation has progressed so that the representation (e.g., a model) is ready for viewing, the interactive window can be used to view the representation (e.g., a model) and / or interact with the representation. In some examples, the target may not have any extracted channels that are close enough to draw a valid trajectory (e.g., a trajectory shorter than the maximum trajectory length) between the target and the extracted channels. Therefore, the user can manually identify the representation (e.g., a model) and add nearby channels to the representation. Refer to the following Fig.11 An exemplary method for manually adding a connection channel to a representation graph (eg, a model) is described in further detail.

[0113] At process 1030, the planned medical procedure is previewed via a graphical user interface in a preview mode. According to some embodiments, previewing the medical procedure may include viewing a live simulated endoscopic view (such as live simulated endoscopic view 610), a virtual endoscopic view (such as virtual endoscopic view 620), an anatomical model view (such as anatomical model view 630), and / or a simplified model view (such as simplified model view 640). According to some embodiments, the simplified model view may include control points (such as control points 650) for scrolling back and forth in the preview of the medical procedure.

[0114] At process 1040, the planned medical procedure is transferred to the medical device via the graphical user interface in a save mode. According to some embodiments, transferring the planned medical procedure can include installing a storage device compatible with the planning workstation and the medical device. In some examples, a message can be displayed via the graphical user interface to alert the user of the need for compatibility. In some examples, the planned medical procedure can be saved during process 1040. In some examples, the planned medical procedure can be transferred to the robotic catheter system. In some examples, after process 1040, method 1000 can proceed to process 1050 to perform the medical procedure according to the plan.

[0115] Fig.111 is a simplified diagram of a method 1100 for modifying an anatomical model to provide access to a target of a medical procedure according to some embodiments. According to some embodiments, the method 1100 may be performed in a hybrid segmentation and planning mode using a graphical user interface such as graphical user interface 400. In some examples, the method 1100 may generally be performed after segmentation is complete and the anatomical model is available for viewing and / or manipulation.

[0116] Typically, the channels of interest to the user are those that are navigated continuously by the instrument from a main channel (such as the trachea) through various branches to an exit point at a channel near the target. In some cases, automatic segmentation may not detect all such channels. Therefore, the set of channels connected to the model generated by segmentation is incomplete. When the initial model does not provide satisfactory access to the target (for example, when the closest exit point is not within a threshold distance, such as previously described), the target may be detected by the automatic segmentation. Figure 5C 3cm as described above), the user may desire to connect one or more channels that were not originally connected to the model. In some cases, automatic segmentation can detect channels that are not of interest to the user, such as channels that do not lead to the target. Method 1100 provides an example of a technique for identifying channels of interest that are not detected by automatic segmentation and connecting the unconnected channels to the model. Method 1100 further provides an example of a technique for pruning channels that are not of interest to the user from the model.

[0117] At process 1110, the distance between the target and the nearest connection channel is measured. According to some embodiments, the distance can be measured automatically, for example, in response to a user defining the target via a graphical user interface. In some examples, the distance can be measured via a graphical user interface by clicking on the target and the nearest connection channel.

[0118] At process 1120, it is determined whether the measured distance is greater than a predetermined threshold. In some examples, the predetermined threshold may correspond to the maximum range of a medical tool (such as a biopsy needle) used in a medical procedure. In some examples, the predetermined threshold may be a fixed value, such as 3 cm, and / or may vary based on factors such as the type and / or model of the medical tool used. At process 1130, it is determined that the measured distance is less than the predetermined threshold. The model may be saved and method 1100 may terminate at process 1130 because the existing model provides satisfactory access to the target. When the distance is greater than the predetermined threshold, method 1100 may proceed to process 1140 for identifying an unconnected channel close to the target and expanding the model to include the identified channel, as described below with reference to Fig.12 Once the identified channel has been connected to the model, method 1100 can repeat process 1110 and process 1120 until the channel has been connected to the model within a predetermined threshold distance from the target.

[0119] In some embodiments, the distance between the target and the nearest connecting channel may not be the only consideration for determining whether the model provides enough airways to reach the target. In some examples, other factors affecting the exit point from the closest airway to the target may be considered. These factors may include a satisfactory exit angle from the exit point, the presence of a compact radius bend that must be navigated through the connecting airway to reach the exit point, the diameter size of the anatomical channel and / or the potential danger between the exit point and the target. Different paths through different airways can be selected based on these considerations. Therefore, in alternative embodiments, optional processes 1150a, 1150b and 1150c may be completed to evaluate other factors about the selected channel to determine whether other factors are satisfactory, and select alternative channels when other factors are unsatisfactory.

[0120] Fig.12 A simplified diagram of a method 1200 for augmenting an anatomical model to provide access to a target of a medical procedure according to some embodiments. At process 1210, an interactive image is displayed via a graphical user interface such as graphical user interface 400. The interactive image depicts image data, connected channels within the image data, and a target within the image data. At process 1220, a user input is received to identify at least a portion of an unconnected channel that is closer to the target than the nearest connected channel. In one example, the method 1200 can be used to augment the anatomical model based on the identification of unconnected channels detected adjacent to or near the connected channels of the model and gradually working toward the target in a "forward" manner. In an alternative example, the method 1200 can be used to augment the anatomical model based on the identification of unconnected channels detected adjacent to or near the connected channels of the model and gradually working toward the connected channels of the model in a "backward" manner.

[0121] At process 1230, when a suitable unconnected channel is not identified in the interactive image initially displayed, the interactive image can be searched by iteratively rotating the interactive image and determining whether the unconnected channel is visible. At process 1240, a rotation point is defined in a graphical user interface by selecting a point in the image data (e.g., by double-clicking the point). In some examples, the rotation point is displayed by placing a crosshair on the interactive image. In one example, the rotation point is selected as a point along the connected airway closest to the target. In another example, the rotation point is selected as a point on the target. In other examples, the rotation point can be any point in the interactive image. At process 1250, the interactive image rotates around the rotation point. In some examples, the rotation point provides a 360-degree rotation in three dimensions around the rotation point. At process 1260, it is determined whether an unconnected channel is identified in the interactive image. When an unconnected channel is not identified, a new rotation point is selected at process 1270, and the interactive image rotates around the new rotation point to identify the unconnected channel. Processes 1240-1270 can be repeated until an unconnected channel is identified.

[0122] When an unconnected channel is identified in the interactive image at process 1230 or process 1260, a user input (e.g., a click and / or tap) identifying the unconnected channel can be received at process 1280 and the unconnected channel is connected to the model. In some embodiments, the unconnected channel can be automatically connected to the model by using segmentation software to track the channel to the connection point with the model. When process 1280 is completed, method 1200 can return to method 1100 to determine whether the newly connected channel provides satisfactory access to the target. Processes 1110-1150 can be repeated until satisfactory access to the target is achieved.

[0123] According to some embodiments, when the interactive image is rotated around the rotation point during process 1250, the unconnected channel is partially identified. After the interactive image is rotated to a state where the unconnected channel is partially identified, it may be helpful to limit the rotation to a rotation around the rotation axis rather than an unconstrained 3D rotation around the rotation point. The rotation axis is defined in the graphical user interface by drawing a line between the rotation point and a second point (such as a target position). Limiting the rotation to the rotation axis can enhance the usability relative to rotation only around the rotation point. Consistent with such an embodiment, the search for unconnected channels can be performed by iteratively repeating processes 1240-1270 using any combination of the rotation point (when unconstrained rotation is required) and the rotation axis (when limited rotation is required).

[0124] Starting with a rotation point may provide some of the advantages of starting with a rotation axis. For example, if the user initially provides an axis of rotation and a given unconnected channel is positioned in an orientation orthogonal to the axis of rotation, the user will not see any change in the appearance of the unconnected channel when rotating the interactive image, i.e., the unconnected channel will appear as a circle, and the unconnected channel will always appear as a circle when rotating about the selected axis of rotation. Using a single rotation point will provide a 360 degree rotation in three dimensions. Thus, regardless of the initial orientation of the unconnected channel, the user will eventually rotate the interactive image in a way that the unconnected channel will be visible. Additionally, selecting a new axis of rotation may be difficult. The user has no guidance as to how to draw the new line. However, changing the rotation point is simple because it only selects a single point. The rotation point may be selected at a location close to a connected channel, however, if the rotation point proves insufficient, the rotation point can be easily moved to the target.

[0125] In another example, again referring to Fig.11 In process 1140, when the user draws a line from the closest connected channel to the target, the model can be alternatively enlarged. Using automated techniques, structures that appear to correspond to unconnected channels can be detected, and the closest such structures can be connected to the model. The user can iteratively continue to draw lines from the latest connected channel to the target, while additional unconnected channels are detected and connected to the model until the closest connected channel falls within the threshold established in process 1120. In one example, when a channel that is unacceptable to the user is connected to the model, the user can select the airway and clear or delete the airway from the model.

[0126] At any time during segmentation and / or model expansion, the model can be pruned. In some examples, one or more connected channels can be determined to be irrelevant. For example, a connected channel can be determined to be far from the target and / or otherwise rarely relevant and / or irrelevant to the purpose of the medical procedure. In some examples, it may be desirable to disconnect and delete irrelevant channels from the model. For example, disconnecting irrelevant channels can reduce visual clutter and / or can improve performance (e.g., improve loading time, rendering time, and / or the like) by reducing the size of the model. Therefore, a pruning tool can be provided to the user via a graphical user interface. For example, the pruning tool can be selected via a tool selector (such as tool selector 524). When the pruning tool is enabled and a user input (e.g., a user click and / or tap) identifying an irrelevant channel is received, the channel can be disconnected and deleted from the model. In some examples, the channel can be disconnected at a point identified by the user. In some examples, the channel can be disconnected at the nearest connection point to the model. In some examples, the identified channel can be disconnected together with any sub-branches of the identified channel. According to some embodiments, pruning may be performed at any time during methods 1100 and / or 1200 , and / or as a separate process independent of methods 1100 and / or 1200 .

[0127] Fig.13 is a simplified diagram of a method 1300 for planning a medical procedure using a graphical user interface according to some embodiments. Figure 4-9 Consistent with some embodiments, method 1300 may be performed using graphical user interface 400 in a hybrid segmentation and planning mode, such as Figure 5A-5G In some examples, method 1300 and / or its various processes can be performed before, during, and / or after segmentation of image data to generate a model.

[0128] At process 1210, an interactive window, such as interactive window 541, is provided to a user to create a plan for a medical procedure. The interactive window may be displayed via a display system and may be interacted with via a user interface such as a mouse, trackball, joystick, touch screen, natural user interface (e.g., voice, gestures), augmented / virtual reality interface, and / or the like. According to some embodiments, the interactive window may be displayed in conjunction with one or more other views, such as a tool selector (e.g., tool selector 524), a selection sidebar (e.g., selection sidebar 542), a control frame (e.g., control frame 510), and / or the like.

[0129] At process 1310, image data is displayed via a graphical user interface. In some examples, the image data may correspond to raw image data (e.g., CT data) of the patient. The image data may be pre-selected in a data selection mode of the graphical user interface. In some examples, the image data may be displayed while the image data is segmented using a background segmentation process. The segmentation data generated by the segmentation process (e.g., airways detected in the image data) may be overlaid on the image data. For example, the image data may be displayed in a first color palette (such as grayscale), and the segmentation data may be displayed in a contrasting color (such as pink). As the segmentation of the image data proceeds, the displayed segmentation data may be updated to reflect the progress of the segmentation.

[0130] At process 1320, a first user input is received that defines one or more features of a plan within the displayed image data. According to some embodiments, the one or more features of the plan may include a target of the medical procedure, a hazard of the medical procedure, and / or the like. In some examples, the target may be defined using an object placement tool having a suitable shape provided by a graphical user interface (e.g., a circle tool and / or a 3D ellipse tool). In some examples, the hazard may be defined using a hazard fence placement tool having a suitable shape (e.g., a 3D disk, a conical hazard fence, and / or a hemispherical hazard fence) and / or suitable control points for defining the hazard fence. Examples of hazards may include vulnerable portions of anatomy (e.g., a lung pleura, a blood vessel, a bulla, and / or a heart) and / or an excessive bend in an anatomical passage (e.g., a bend that is too tight to accommodate the passage of a medical instrument (such as a biopsy needle)).

[0131] At process 1330, an interactive image is displayed via a graphical user interface. The interactive image includes image data, connected anatomical channels detected by segmentation of the image data, and one or more features defined during process 1320. The connected anatomical channels form a tree in which each branch is reachable from a main channel (such as a trachea). Therefore, a medical device inserted via the main channel can enter the connected anatomical channels. The user can interact with the interactive image via a user interface such as a mouse, trackball, joystick, touch screen, natural user interface (e.g., voice, gestures), augmented / virtual reality interface, and / or the like. According to some embodiments, the interactive image can be displayed in conjunction with one or more other views (such as a tool selector (e.g., tool selector 524), a selection sidebar (e.g., selection sidebar 542), a control frame (e.g., control frame 510), and / or the like).

[0132] At process 1340, a second user input is received identifying at least a portion of a trajectory of the medical procedure within the interactive image. In some examples, the trajectory can be identified by connecting the target to a closest channel in the connected anatomical channels. For example, the received second user input may include a line drawn between the target and the closest channel via a line tool provided by the graphical user interface.

[0133] At process 1350, a third user input is received to adjust one or more aspects of the interactive image based at least in part on the defined trajectory. According to some embodiments, process 1350 may generally correspond to method 1200 for enlarging the anatomical model, in which case the third user input may include one or more user inputs received during method 1200. For example, process 1350 may include determining a distance represented by the trajectory, for example, the distance between the closest channel and the target. Consistent with such an example, adjusting the interactive image may include connecting an unconnected channel to a connected channel when the distance is greater than a predetermined threshold (e.g., 3 cm). The unconnected channel may be connected by receiving a fourth user input identifying an unconnected channel closer to the target than the nearest channel and connecting the identified channel using an automatic technique. In some examples, adjusting the interactive image may include determining an exit angle based on the trajectory (e.g., the angle at which the medical device pierces the lumen of the channel when entering the target from the channel) and receiving a third user input to manipulate a control (such as a slider) provided by a graphical user interface for changing the orientation of the exit point along the channel. In some examples, the control may provide continuous control over the orientation of the exit point and / or may provide real-time updated metrics associated with the selected exit point, such as the corresponding exit angle.

[0134] Figure 14A-14F 14 is a further simplified diagram of graphical user interface 400 in branch labeling mode according to some embodiments. In this example, branch labeling mode is applied to lung anatomy, but in other examples, graphical user interface 400 is used to label any suitable anatomical structure. The lungs include a right lung and a left lung, wherein each lung is divided into lobes, which in turn may be divided into segments and lobules. Within each lobe are various anatomical structures, including a set of anatomical passages 1402, which may include a plurality of branches 1403. In Figure 14A-Figure 15BIn the example of , the branch labeling mode can be used to identify and label which lobe a single branch 1403 may belong to. In an alternative embodiment, the branch labeling mode is used to identify and label which lung the segments and / or lobules and individual branches 1403 may belong to. In the branch labeling mode, the graphical user interface 400 provides a mechanism for the user to assign labels to the branches 1403, reflecting which segment 1404 (e.g., lobe) of the lung the corresponding branch 1403 belongs to. In an exemplary embodiment, the branch labeling mode is used to help align the airway model to the human anatomy to provide navigation during image-guided biopsy procedures. Further details on alignment can be found in US62 / 486,879, which has been previously incorporated by reference.

[0135] Branch labeling can operate on a 3D model (such as the model described above) or any other suitable model of the anatomical passage 1402. The model can be created from imaging data including CT data, MRI data, OCT data, x-ray data, and / or other types of imaging or patient data as described above. In some embodiments, the 3D model includes a set of anatomical passages 1402 and other anatomical structures, such as ribs, blood vessels, tumors, lesions, and / or organs (e.g., heart). In the illustrated embodiment, the 3D model includes lung branches 1403 and lung pleura 1405, but further embodiments include any type of suitable anatomical passage 1402 and surrounding structures. The graphical user interface 400 displays elements of the 3D model, and the display of individual elements (e.g., lung pleura 1405) can be hidden or displayed to improve clarity.

[0136] In some examples, the graphical user interface 400 automates aspects of the labeling process, including label selection. Further, in some examples, the graphical user interface 400 automates aspects of label verification, including identifying unlabeled branches 1403 and / or identifying conflicts in user input. In these and other examples, the graphical user interface 400 can speed up the planning of medical procedures by providing the user with an accelerated process for identifying branches.

[0137] Fig.14A An example of a graphical user interface 400 in a branch marking mode before marking the segments / lobes 1404 of any branch 1403 is shown. In some embodiments, the graphical user interface 400 includes an interactive window 1406 for viewing, selecting, marking, and / or otherwise interacting with the model of the anatomical passage 1402. The interactive window 1406 can display an image of the model representing the anatomical passage 1402 and the surrounding anatomy (e.g., the lung pleura 1405) as a 3D rendering, wireframe, cross-section, and / or other suitable representation. In one such embodiment, the interactive window 1406 represents the branch 1403 by a centerline.

[0138] In some embodiments, graphical user interface 400 includes label tool 1408, which is used to select the label of the branch to be assigned to the model. Label tool 1408 can be represented as a color palette, header, footer, sidebar, menu, message bar, drop-down menu, pop-up window and / or other suitable representation diagram. In some embodiments, label tool 1408 displays a list 1410 of labels to be applied. Label tool 1408 can use highlighting, color, font, outline, emphasis and / or other suitable indicators to indicate the currently selected label. Label tool 1408 can also display status indicator 1412 for each label, which indicates the state such as whether the label has been applied. Status indicator 1412 can indicate whether the label has been applied to more than one branch, and in one such embodiment, status indicator 1412 displays a single check mark to indicate that the label has been applied to a single branch and displays two check marks to indicate that the label has been applied to more than one branch. Additionally or alternatively, the labeling tool 1408 can display a set of interactive objects 1414 (e.g., buttons, check boxes, radio buttons, text-based buttons, etc.) for setting properties of the corresponding labels. For example, the user can select the interactive object 1414 to indicate that the branch corresponding to the label is missing and does not exist in the model and / or anatomy.

[0139] The graphical user interface 400 is operable to receive user input and display a cursor 1416 in response to the user input. When the cursor 1416 is within the boundaries of the labeling tool, the graphical user interface 400 can receive user input to select a label from the list 1410, activate or deactivate the interactive object 1414, and / or take other suitable actions. When the cursor 1416 is within the interactive window 1406, the graphical user interface 400 can receive user input to select a branch of the model, manipulate (e.g., rotate, translate, zoom, etc.) the image of the model and / or the image data of the surrounding anatomy, and / or take other suitable actions. In an example, the graphical user interface 400 is responsive to a first mouse button within the interactive window 1406 for selection, and in response to a second mouse button, the perspective view shown in the interactive window is rotated by 90°. When rotating the perspective view, the graphical user interface 400 can also rotate the patient orientation indicator 1417, which represents the perspective view of the model and the surrounding anatomy displayed in the interactive window 1406 relative to the patient.

[0140] When a label is selected and cursor 1416 is positioned over branch 1403 in interactive window 1406, graphical user interface 400 may indicate that branch 1403 may be labeled by changing the representation of cursor 1416, the representation of branch 1403 or multiple branches extending from the selected branch in interactive window 1406, and / or providing other suitable indications. Additionally or alternatively, when a label is selected and cursor 1416 is positioned in interactive window 1406 but not over any branch 1403, graphical user interface 400 may indicate that the cursor is not over a branch by changing the representation of cursor 1416 and / or providing other suitable indications.

[0141] refer to Figure 14B-Figure 14F An example of a graphical user interface 400 responsive to user input is described. Fig. 14B An example of applying labels to branches 1403 of anatomical passage 1402 via graphical user interface 400 is shown. In some embodiments, labeling tool 1408 of graphical user interface 400 automatically selects a first label to be applied. Labeling tool 1408 can select the first label from those labels in list 1410 that have not been applied to at least one branch. Labeling tool 1408 can also provide an indication that the first label has been selected, such as those described above. The user can select a different label via labeling tool 1408 to override the automatic selection. Figure 14A-14F In the example shown, the label indicates a lung lobe. It should be understood that the label can indicate other sections of the lung, including the left or right lung, a lung segment, and / or a lung lobule.

[0142] Then, the graphical user interface 400 can receive a user selection of a branch 1403 via an interactive window 1406, and in response, the selected label can be assigned to the selected branch 1403. Selecting a single branch 1403 can make the graphical user interface 400 identify other branches 1403 connected to the selected branch, and assign labels to multiple branches 1403 as a whole. In some examples, a label indicator 1418 is displayed in the interactive window 1406, such as a sign, a mark, a text box and / or other suitable indicators representing the assigned label and the corresponding branch 1403 or multiple branches 1403. The representation diagram of the branch 1403 or multiple branches 1403 in the interactive window 1406 can be colored, outlined, emphasized, de-emphasized or otherwise modified to indicate that a label has been assigned. The graphical user interface 400 can also update the corresponding status indicator 1412 in the label tool 1408 to indicate that a label has been assigned to at least one branch. In the example shown, the first branch 1403 is selected by the user and is labeled as "upper right leaf". The graphical user interface 400 highlights all branches connected to the first branch 1403 up to the trunk (eg, the trachea of ​​the lung) in a first color. At this point, all highlighted branches are effectively identified and labeled as belonging to the "right upper lobe" segment / lobe 1404.

[0143] like Fig. 14C As depicted, when a label is applied to a branch 1403 or multiple branches 1403, a labeling tool 1408 can automatically select the next label to be applied in the list 1410. In some embodiments, the labeling tool 1408 selects the next label based on the arrangement of the branches in the anatomical channel 1402. For example, the middle lobe label can be selected after assigning the upper lobe label because the upper lobe is close to the middle lobe in the anatomical channel 1402. In some embodiments, the labeling tool 1408 selects from those labels in the list 1410 that have not been applied to any branch. For example, if the middle lobe label has been used to label a branch, the lower lobe label can be selected after assigning the upper lobe label. By automatically selecting the next label, in these and other embodiments, the user can continue to select branches without moving the cursor out of the interactive window 1406.

[0144] The process of selecting tags and branches can be repeated. Fig. 14B As explained in the context of FIG. 1 , in one example, the first branch 1403 is selected by the user and is labeled as “upper right leaf”. Therefore, all branches connected to the first branch 1403 up to the trunk are also labeled as “upper right leaf”. In response, the graphical user interface 400 selects the next label “middle right leaf”, such as Fig. 14C Reference Fig.14D, the user selects the second branch 1403 to assign the label "right middle lobe". The graphical user interface 400 assigns a second color to the label and changes the color of all branches 1403 that are descendants of the second branch (e.g., child branches that originate from the second branch or are distal to and connected to the second branch) with the second color, identifying and marking the descendant branches belonging to the right middle lobe. In one example, when the second branch 1403 is selected, all descendant branches are highlighted with the second color, and a portion of the proximal / parent connected branches reflects the change in color up to the trunk of the previously identified segment / lobe 1404. In one embodiment, multiple branches 1403 may overlap, possibly belonging to two separate segments / lobes 1404. Overlapping branches may be highlighted with a separate specific color reflecting the overlap. The process is repeated for the "right lower lobe" using a third color to identify and mark the distal branches of the third user-selected branch within the right lower lobe. The process may be repeated again for the "left upper lobe" and "left lower lobe" until all branches 1403 have been identified and marked as belonging to the segment / lobe 1404. Fig.14E Segment / lobe 1404 is shown.

[0145] like Fig.14E As depicted, to assist the user, in some embodiments, the graphical user interface 400 indicates in the interactive window 1406 those branches 1403 in the anatomical passage 1402 that do not yet have a label. The graphical user interface 400 may indicate the unlabeled branches 1403 using indicators 1420, such as markings, highlights, outlines, colors, line widths, and / or other suitable indicators. In one such example, the graphical user interface 400 displays a labeled question mark symbol with a connector that extends to the unlabeled branches 1403. In some embodiments, the graphical user interface 400 indicates the remaining unlabeled branches for the user to reference upon detecting that each label has been applied to at least one branch 1403. When the user selects a label and selects an unlabeled branch, the label may be applied to the branch 1403 and / or the segment / lobe 1404 to which it belongs, and the indicator 1420 may be removed, as shown. Fig.14F In the example of performing unlabeled branch detection when each label is applied to at least one branch, applying the label to the unlabeled branch may cause the label to be applied to more than one branch. Therefore, the graphical user interface 400 can update the corresponding status indicator accordingly. Fig.14E , the status indicator 1412A is updated to display a pair of check marks because the left upper lobe has been assigned to more than one segment / lobe 1404. In another example of performing unlabeled branch detection when all labels are applied to at least one branch, the graphical user interface 400 removes the unlabeled branch indicator 1420 when a label is deleted from a branch and it is no longer the case that all labels have been applied to at least one branch.

[0146] Fig.15A and Fig. 15B is a simplified diagram of a method 1500 for planning a medical procedure according to some embodiments. Figure 14A-14F Consistent with some embodiments, method 1500 may be adapted for use with a graphical user interface, such as graphical user interface 400, operating in a branch marking mode.

[0147] exist Fig.15A At process 1501, imaging data, such as CT data of a patient, is received. At process 1502, a model of the patient's anatomy is provided based on the image data. The model may include a set of anatomical channels 1402 and / or other anatomical structures, such as organs, blood vessels, tumors, lesions, etc. Specifically, the anatomical channels 1402 may include branches 1403 and / or other suitable structures.

[0148] At process 1504, an image of the model is displayed via graphical user interface 400. According to some embodiments, the image of the model is displayed in interactive window 1406, which includes a representation of the anatomical passage 1402 and surrounding anatomy, such as the lung pleura 1405. These elements of the surrounding anatomy may be displayed or hidden individually to provide a frame of reference and improve clarity.

[0149] At process 1506, a first tag is selected. The first tag may be selected automatically and / or in response to a first user input received by the tag tool 1408 of the graphical user interface. The user input may be provided via any suitable input mechanism, including a mouse, keyboard, touch, stylus, trackball, joystick, voice command, virtual reality interface, and / or the like. At process 1508, an indication is displayed by the tag tool 1408 of the graphical user interface that the first tag has been selected.

[0150] At process 1510, the graphical user interface 400 may receive a second user input, which selects the first branch 1403 for the first label. The second user input may select the first branch 1403 via an interactive window 1406 displaying the model. At process 1511, in some examples, the graphical user interface 400 identifies other branches 1403 connected to the first branch 1403, so that multiple branches 1403 can be labeled in a single process. In one example, this includes identifying those branches 1403 that are descendants of the first branch 1403 (e.g., sub-branches starting from the distal side of the first branch) and including descendants in the segment / lobe 1404. In this way, labels can be propagated downstream from the selected branch. In one example, this includes identifying the preceding branch 1403 (e.g., the parent branch, ancestor branch, etc. starting from the proximal side of the first branch) until the main branch. The main branch can be the trachea in the initial case or the main branch of the previously identified segment / lobe 1404. In this way, labels can be propagated upstream so that the user can label the subtree without having to select the root of the subtree.

[0151] At process 1512, graphical user interface 400 determines whether the label conflicts with the previously selected label. For example, the user can mark the child branch with the upper right leaf label. If the child branch includes the parent branch previously marked with the upper left leaf label, the algorithm can identify the conflict between the child branch and the parent branch. Graphical user interface 400 can refuse to apply the current label and illustrate the conflict by highlighting the current label, the currently selected branch, the previous label and / or the previously selected branch. The option for correcting the previous label or the current label can then be presented to the user. Once any conflict is resolved, at process 1513, the first branch 1403 and any other branches 1403 identified in process 1511 are marked with the first label.

[0152] At process 1514, the graphical user interface displays a representation of the first label applied to first branch 1403 and / or its corresponding segment / lobe 1404. In some such examples, the graphical user interface displays a banner, mark, text box, and / or other suitable indicator in interactive window 1406 to indicate the first label. In some examples, the representation of first branch 1403 and / or its segment / lobe 1404 in interactive window 1406 is colored, outlined, emphasized, de-emphasized, or otherwise modified to indicate that the label has been assigned.

[0153] Reference now Fig. 15B, at process 1516, it is determined whether the labeling tool 1408 has an additional unassigned label. If so, then at process 1518, the next label is selected. The next label can be automatically selected from a set of unassigned labels based on the arrangement of the branches 1403 in the anatomical channel 1402 and / or other suitable criteria. The next label can also be selected in response to a first user input received by the labeling tool 1408 of the graphical user interface. In some such examples, the user selection overrides the automatic selection. At process 1520, an indication that the first label has been selected is displayed by the labeling tool 1408 of the graphical user interface. At process 1522, the graphical user interface can receive user input of a branch of the anatomical channel for selecting a label. In response, at process 1524, the selected branch is marked with a label. At process 1526, the graphical user interface displays a representation of the label applied to the branch. Processes 1518-1526 can be performed substantially similar to processes 1506-1514.

[0154] At process 1528, the graphical user interface can identify branches 1403 of the anatomical channel 1402 that do not have assigned labels. This operation can be performed when it is determined in process 1516 that each label has been assigned to at least one branch. At process 1530, the graphical user interface displays an indicator identifying the unassigned branches in the interactive window 1406, such as a flag, highlight, outline, color, line width, and / or other suitable indicators. At process 1532, user input selecting a label is received. This can be performed substantially similar to processes 1506 and / or 1518. At process 1534, the selected label is applied to the unassigned branches and their corresponding segments / lobes, and at process 1536, the graphical user interface displays a representation of the labels applied to the branches and / or segments / lobes. Processes 1534-1536 can be performed substantially similar to processes 1511-1514. This can be repeated until a label is assigned to each branch.

[0155] Fig.16 is a simplified diagram of method 1600 for planning a medical procedure according to some embodiments. Figure 17A-17N is a corresponding diagram of graphical user interface 400 during execution of method 1600 according to some embodiments. Figure 1-15B Consistent with some embodiments, graphical user interface 400 may include a graphical user interface that generally corresponds to Figure 1-15B, such as interactive header 410, control frame 510, canvas frame 520, workspace 522, tool selector 524, interactive window 541, selection sidebar 542, image data 543, segmentation data 544, and / or the like. In some embodiments, graphical user interface 400 may include mode selector 1712 (illustratively placed within interactive header 410) and / or view selector 1714 (illustratively placed at the bottom of interactive window 541) for enabling and / or disabling various features of graphical user interface 400, including various features used during execution of method 1600.

[0156] At process 1610, a goal 1720 for a medical procedure is added via graphical user interface 400. Fig.17A and 17B An illustrative screen shot corresponding to process 1610 is depicted in FIG. In some embodiments, target 1720 may generally correspond to target 550. Fig.17A As depicted, during the addition of target 1720, new target menu 1722 may be displayed in control frame 510. New target menu 1722 may display instructions for identifying, placing, adjusting, and / or otherwise configuring target 1720, controls for confirming and / or cancelling the addition of target 1720, and / or the like. When target 1720 is added, target data 1724 may be displayed in control frame 510. Fig. 17B As depicted, target data 1724 may include metrics associated with target 1720 (eg, size metrics), controls (eg, controls for deleting, renaming, and / or editing target 1720), and / or the like.

[0157] At process 1620, the operator may optionally zoom to target 1720 (and / or other portions of image data 543) to view graphical user interface 400. Figure 17C-17E An illustrative screen shot corresponding to process 1610 is depicted in . Graphical user interface 400 may provide one or more options for zooming to target 1720, including asynchronous zooming, synchronous zooming, automatic zooming, and / or the like. Fig. 17C depicts zooming performed when the synchronized zoom feature of graphical user interface 400 is not enabled, and Fig.17D Depicted is a zoom performed when the synchronized zoom feature is enabled. When the synchronized zoom feature is not enabled, zooming in and / or out of the image displayed in the interactive window 541 is not accompanied by a matching zoom effect in the thumbnail image displayed in the selection sidebar 542. Conversely, when the synchronized zoom feature is enabled, zooming in and / or out of the image displayed in the interactive window 541 is accompanied by a matching zoom effect in the thumbnail image displayed in the selection sidebar 542. Fig.17EDepicted is a zoom performed using the auto-zoom feature of graphical user interface 400. Use of the auto-zoom feature causes target 1720 to be automatically centered and / or enlarged in the image displayed in interactive window 541. The auto-zoom feature may or may not be accompanied by selection of a matching zoom effect in a thumbnail image displayed in sidebar 542.

[0158] At process 1630, the operator may optionally edit the target 1720. Fig.17F An illustrative screen shot corresponding to process 1630 is depicted in FIG. Fig.17F As depicted, during editing of target 1720, edit target menu 1726 may be displayed in control frame 510. Edit target menu 1726 may display instructions for editing target 1720, controls for confirming and / or cancelling editing of target 1720, and / or the like. In some embodiments, an operator may modify properties of target 1720 (e.g., size, orientation, and / or the like) via interactive window 541 and / or via selection sidebar 542.

[0159] At process 1640, a path 1730 to a target 1720 is identified via graphical user interface 400. Figure 17G-17I An illustrative screen shot corresponding to process 1640 is depicted in FIG. Figure 17G As shown, path data 1732 corresponding to the selected path can be displayed in control frame 510. Path data 1732 can include metrics associated with path 1730 (e.g., distances between endpoints of path and target 1720, exit angles at endpoints of path, and / or the like), controls (e.g., controls to delete, rename, and / or edit path 1730), and / or the like. In some embodiments, one or more alerts associated with the path can be displayed via graphical user interface 400. For example, Figure 17G As depicted, when the distance between the endpoint of the path and the closest point of the target 1720 exceeds a predetermined threshold, an alert 1734 is displayed. Fig.17H As depicted, when the exit angle at the endpoint of the path exceeds a predetermined threshold, an alert 1736 is displayed. In some embodiments, an endpoint slider 1738 may be displayed in the control frame 510 to allow the operator to adjust the orientation of the endpoint along the path. In this regard, the operator may determine the orientation of the endpoint that meets a predetermined threshold associated with the distance to the target 1720 and / or the exit angle.

[0160] At process 1650, one or more pathways may be extended, optionally via graphical user interface 400. For example, when an acceptable path to target 1720 is not identified at process 1640, one or more pathways may be extended. Figure 17J-17MAn illustrative screen shot corresponding to process 1650 is depicted in FIG. Fig.17J As depicted, instruction panel 1740 may be displayed in control frame 510 to provide instructions for extending one or more channels. Figure 17K As depicted, the operator may draw a path extension 1742 via the interactive window 541. The path extension 1742 may then be rendered, as shown in FIG. Figure 17L In some embodiments, path extension 1742 may be rendered with a different color, texture, pattern, and / or the like to distinguish path extension 1742 from the portion of the path determined by segmentation. Fig.17M As depicted, the updated path to target 1720 may meet predetermined thresholds associated with the distance and / or exit angle to target 1720 such that alerts 1734 and / or 1736 are no longer displayed with path data 1732 .

[0161] At process 1660, the plan for the medical procedure is reviewed via graphical user interface 400. Fig.17N An illustrative screenshot corresponding to process 1660 is depicted in FIG. Fig.17N The features depicted in usually correspond to Figure 6 In accordance with these embodiments, Fig.17N A virtual endoscope view 620, a global anatomical model view 630, and a reduced anatomical model view 640 are depicted. Fig.17N As depicted, the global anatomical model view 630 includes controls, for example, for allowing an operator to pan, zoom, and / or rotate the model view and / or select between different types of images (e.g., CT images). Similarly, the reduced anatomical model view 640 includes controls for starting and / or pausing playback of a plan for a medical procedure, exporting the current plan, and / or the like. In some embodiments, an information panel 1750 may be displayed to provide information associated with the plan, such as metrics associated with the distance to the target 1720.

[0162] Some examples of control units, such as control unit 130, may include a non-transitory, tangible, machine-readable medium that includes executable code that, when executed by one or more processors (e.g., processor 140), may cause the one or more processors to perform the processes of methods 1000-1300 and / or method 1500 to render graphical user interface 400. Some common forms of machine-readable media that may include methods 1000-1300, the processes of method 1500, and / or instructions for rendering graphical user interface 400 are, for example, floppy disks, diskettes, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cassettes, and / or any other media suitable for reading by a processor or computer.

[0163] Although illustrative embodiments have been shown and described, extensive modifications, changes, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the various embodiments may be employed without the corresponding use of other features. Those of ordinary skill in the art will recognize many variations, substitutions, and modifications. Therefore, the scope of the present invention shall be limited only by the appended claims, and it shall be understood that the claims shall be interpreted broadly in a manner consistent with the scope of the embodiments disclosed herein.

Claims

1. A planning workstation, comprising: Display system; and user input device; Wherein the planning workstation is configured to, during preoperative planning of a procedure: displaying a plurality of interactive windows for a user to view a plan of a medical procedure, wherein each of the plurality of interactive windows displays a different rendering of a model of an anatomical passage; displaying a pathway through the anatomical passage to a target of the medical procedure; displaying a virtual image of the instrument within the anatomical passage; displaying, in at least one of the plurality of interactive windows, a control point corresponding to a distal end of the instrument; receiving user input via the user input device; identifying a position of the control point based at least on the user input; and In response to receiving the user input, the position of the instrument is dynamically updated in at least one of the plurality of interactive windows to match the position of the control point.

2. The planning workstation of claim 1, wherein the user input device comprises a touch screen of the display system.

3. A planning workstation according to claim 1, wherein the control point is displayed in a first interactive window among the multiple interactive windows, the first interactive window including a reduced anatomical model view, and receiving the user input regarding the reduced anatomical model view.

4. A planning workstation according to claim 3, wherein the reduced anatomical model view includes a simplified overview of the planned route of the instrument through the anatomical passage including key anatomical features.

5. The planning workstation of claim 3, wherein the position of the instrument is dynamically updated in a second interactive window of the plurality of interactive windows, the second interactive window comprising a simulated live endoscopic view.

6. The planning workstation of claim 3, wherein the position of the instrument is dynamically updated in a second interactive window of the plurality of interactive windows, the second interactive window comprising a virtual endoscopic view.

7. The planning workstation of claim 6, wherein the virtual endoscopic view includes at least one directional cue to guide the user toward the target.

8. A planning workstation according to claim 3, wherein the position of the instrument is dynamically updated in a second interactive window of the plurality of interactive windows, the second interactive window comprising a global anatomical model view.

9. The planning workstation of claim 8, wherein dynamically updating the position of the instrument in the second interactive window comprises updating a shape of the virtual image of the instrument.

10. A method for preoperative planning of a medical procedure, the method comprising: displaying a plurality of interactive windows on a display system for a user to view a plan of a medical procedure, wherein each of the plurality of interactive windows displays a different rendering of a model of an anatomical corridor; displaying a pathway through the anatomical passage to a target of the medical procedure; displaying a virtual image of the instrument within the anatomical passage; displaying, in at least one of the plurality of interactive windows, a control point corresponding to a distal end of the instrument; receiving user input via a user input device; identifying a position of the control point based at least on the user input; and In response to receiving the user input, the position of the instrument is dynamically updated in at least one of the plurality of interactive windows to match the position of the control point. The method of claim 10 , wherein the user input device comprises a touch screen of the display system.

12. The method according to claim 10, wherein the control point is displayed in a first interactive window among the multiple interactive windows, the first interactive window comprising a reduced anatomical model view, and receiving the user input regarding the reduced anatomical model view.

13. The method of claim 12, wherein the reduced anatomical model view includes a simplified overview of a planned route of the instrument through the anatomical passage including key anatomical features.

14. The method of claim 12, wherein the position of the instrument is dynamically updated in a second interactive window of the plurality of interactive windows, the second interactive window comprising a simulated live endoscopic view.

15. The method of claim 12, wherein the position of the instrument is dynamically updated in a second interactive window of the plurality of interactive windows, the second interactive window comprising a virtual endoscopic view.

16. The method of claim 15, wherein the virtual endoscopic view includes at least one directional cue to guide the user toward the target.

17. The method of claim 12, wherein the position of the instrument is dynamically updated in a second interactive window of the plurality of interactive windows, the second interactive window comprising a global anatomical model view.

18. The method of claim 17, wherein dynamically updating the position of the instrument in the second interactive window comprises updating a shape of the virtual image of the instrument.

19. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which, when executed by one or more processors associated with a planning workstation, are adapted to cause the one or more processors to perform a method of preoperatively planning a procedure, the method comprising: providing a plurality of interactive windows for a user to view a plan of a medical procedure, wherein each of the plurality of interactive windows displays a different rendering of a model of an anatomical passage; displaying a pathway through the anatomical passage to a target of the medical procedure; displaying a virtual image of the instrument within the anatomical passage; displaying, in at least one of the plurality of interactive windows, a control point corresponding to a distal end of the instrument; Receive user input; identifying a position of the control point based at least on the user input; as well as In response to receiving the user input, the position of the instrument is dynamically updated in at least one of the plurality of interactive windows to match the position of the control point.

20. The non-transitory machine-readable medium of claim 19, wherein the control point is displayed in a first interactive window of the plurality of interactive windows, the first interactive window comprising a reduced anatomical model view, and receiving the user input regarding the reduced anatomical model view.

21. A planning workstation, comprising: Display system; and user input device; Wherein the planning workstation is configured to, during preoperative planning of a procedure: receiving a representation of an anatomical passage including a plurality of branches; displaying the image representing the graph via a graphical user interface; receiving a first user input indicating a selection of a first tab; receiving a second user input indicating a selection of a first branch of the plurality of branches; In response to the first user input and the second user input: labeling the first branch with the first label; and displaying, via the graphical user interface, a representation of the first label applied to the first branch; and Based on labeling the first branch with the first label, a second label is automatically selected to be applied to at least a portion of the representation of the anatomical passage.

22. The planning workstation according to claim 21, further configured to: An indication is displayed via the graphical user interface that the second tab has been selected.

23. The planning workstation method of claim 22, wherein the second label is selected based on a placement of the plurality of branches within the anatomical passage.

24. The planning workstation according to claim 21, further configured to: receiving a third user input indicating a selection of a second branch of the plurality of branches; labeling the second branch with the second label; and A representation of the second label applied to the second branch is displayed via the graphical user interface.

25. The planning workstation according to claim 21, further configured to: identifying a branch group including the first branch from among the plurality of branches; and In response to the first user input and the second user input, the branch group is labeled with the first label.

26. The planning workstation of claim 25, wherein identifying the set of branches comprises identifying descendant branches of the first branch and including the descendant branches in the set of branches.

27. The planning workstation according to claim 21, further configured to: An indication of a second branch of the plurality of branches that does not have an assigned label is displayed via the graphical user interface.

28. The planning workstation according to claim 21, further configured to: In response to the first user input and the second user input: assigning a color to the first branch; and The image is displayed showing a representation of the anatomical passageway wherein the first branch is colored with an assigned color.

29. The planning workstation of claim 21, further configured to: providing a cursor via the graphical user interface; detecting that the cursor is aligned with the first branch of the plurality of branches; and A representation of the cursor is modified based on detecting that the cursor is aligned with the first branch.

30. The planning workstation according to claim 21, further configured to: receiving a third user input indicating a rotation instruction; In response to the third user input, rotating the representation of the anatomical corridor; and An image of a rotational representation of the anatomical corridor is displayed via the graphical user interface.

31. A method for preoperative planning of a medical procedure on a planning workstation, comprising: receiving a representation of an anatomical passage including a plurality of branches; displaying the image representing the graph via a graphical user interface; receiving a first user input indicating a selection of a first tab; receiving a second user input indicating a selection of a first branch of the plurality of branches; In response to the first user input and the second user input: labeling the first branch with the first label; as well as displaying, via the graphical user interface, a representation of the first label applied to the first branch; and Based on labeling the first branch with the first label, a second label is automatically selected to be applied to at least a portion of the representation of the anatomical passage.

32. The method of claim 31 , further comprising: An indication is displayed via the graphical user interface that the second tab has been selected.

33. The method of claim 32, wherein the second label is selected based on an arrangement of the plurality of branches within the anatomical passage.

34. The method of claim 31, further comprising: receiving a third user input indicating a selection of a second branch of the plurality of branches; labeling the second branch with the second label; as well as A representation of the second label applied to the second branch is displayed via the graphical user interface.

35. The method of claim 31, further comprising: identifying a group of branches including the first branch from among the plurality of branches; as well as In response to the first user input and the second user input, the branch group is labeled with the first label.

36. The method of claim 35, wherein identifying the set of branches comprises identifying descendant branches of the first branch and including the descendant branches in the set of branches.

37. The method of claim 31, further comprising: An indication of a second branch of the plurality of branches that does not have an assigned label is displayed via the graphical user interface.

38. The method of claim 31, further comprising: In response to the first user input and the second user input: assigning a color to the first branch; as well as The image is displayed showing a representation of the anatomical passageway wherein the first branch is colored with an assigned color.

39. The method of claim 31, further comprising: providing a cursor via the graphical user interface; detecting that the cursor is aligned with the first branch of the plurality of branches; as well as A representation of the cursor is modified based on detecting that the cursor is aligned with the first branch.

40. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which, when executed by one or more processors associated with a planning workstation, are adapted to cause the one or more processors to perform a method of preoperatively planning a procedure, the method comprising: receiving a representation of an anatomical passage including a plurality of branches; displaying the image representing the graph via a graphical user interface; receiving a first user input indicating a selection of a first tab; receiving a second user input indicating a selection of a first branch of the plurality of branches; In response to the first user input and the second user input: labeling the first branch with the first label; as well as displaying, via the graphical user interface, a representation of the first label applied to the first branch; and Based on labeling the first branch with the first label, a second label is automatically selected to be applied to at least a portion of the representation of the anatomical passage.

Citation Information

Patent Citations

  • Systems and methods for pre-operative modeling

    US20160070878A1

  • System for use in treatment of vertebral fractures

    US20180028199A1