Systems and methods for indicating proximity to anatomical boundaries

Abstract

A medical system includes a display system and a medical instrument. The system further includes a control system communicatively coupled to the display system. The control system is configured to display, via the display system, image data of a patient anatomy. The control system is further configured to determine that a distal portion of the medical instrument is within a threshold distance of a target location of the patient anatomy as the medical instrument navigates the patient anatomy. If the distal portion of the medical instrument is within the threshold distance of the target location, the control system is further configured to display, via the display system, an anatomical boundary that indicates a surface of an anatomical structure of the patient anatomy.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit and priority of U.S. Provisional Application No. 62 / 955,184, filed December 30, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to systems and methods for planning and executing image-guided procedures, and more specifically to systems and methods for automatically generating anatomical boundaries that can be viewed and / or manipulated via a graphical user interface. Background Technology

[0004] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby reducing patient recovery time, discomfort, and harmful side effects. Such techniques can be performed through natural openings in the patient's anatomy or through one or more surgical incisions. Through these natural openings or incisions, clinicians can insert minimally invasive medical instruments (including surgical, diagnostic, therapeutic, or biopsy instruments) to reach target tissue locations. Some minimally invasive techniques use medical instruments that can be inserted into anatomical channels and navigate toward regions of interest within the patient's anatomy. Control of such instruments during image-guided procedures may involve the management of several degrees of freedom of motion, including the insertion and retraction of elongated devices and their orientation. Improved systems and methods can be used to reduce the risk of patient injury by identifying boundaries when planning the navigation and deployment of instruments. Summary of the Invention

[0005] Consistent with some embodiments, a medical system is provided. The system includes a display system and a medical device. The system further includes a control system communicatively coupled to the display system. The control system is configured to display image data of a patient's anatomy via the display system. The control system is further configured to determine, while the medical device is navigating the patient's anatomy, that a distal portion of the medical device is within a threshold distance of a target location within the patient's anatomy. The control system is further configured to display an anatomical boundary via the display system if the distal portion of the medical device is within the threshold distance of the target location. The anatomical boundary indicates the surface of an anatomical structure of the patient's anatomy.

[0006] In another example, a non-transitory machine-readable medium is provided. The non-transitory machine-readable medium includes a plurality of machine-readable instructions that, when executed by one or more processors associated with a workstation, are adapted to cause the one or more processors to perform a method. The method includes displaying image data of a patient's anatomy via a display system. The method further includes determining, while navigating the patient's anatomy with a medical device, that a distal portion of the medical device is within a threshold distance of a target location within the patient's anatomy. The method further includes displaying an anatomical boundary via the display system if the distal portion of the medical device is within the threshold distance of the target location. The anatomical boundary indicates the surface of the anatomical structure of the patient's anatomy.

[0007] It should be understood that the foregoing general description and the following detailed description are illustrative and explanatory in nature and are intended to provide an understanding of this disclosure, without limiting its scope. Additional aspects, features, and advantages of this disclosure will be apparent to those skilled in the art from the following detailed description. Attached Figure Description

[0008] Figure 1 It is a simplified diagram based on some examples of patient anatomy.

[0009] Figure 2A This is a flowchart of a method for generating anatomical boundaries, based on some example illustrations.

[0010] Figure 2B This is a schematic diagram based on some examples of the input used to determine anatomical boundaries.

[0011] Figure 3 This is a simplified diagram of the graphical user interface during the execution of a method for generating anatomical boundaries, based on some examples.

[0012] Figure 4A It is a simplified diagram of guiding information associated with anatomical boundaries based on illustrations of some examples.

[0013] Figure 4B It is a simplified diagram of a graphical user interface that uses two-dimensional image data to present guidance information, based on some examples.

[0014] Figure 5 This is a schematic diagram based on some example illustrations used to determine the input for the area boundary.

[0015] Figure 6 It is a simplified diagram of guiding information associated with anatomical boundaries based on illustrations of some examples.

[0016] Figures 7A-7C This is a simplified diagram of the graphical user interface during the execution of a method for generating anatomical boundaries, based on some examples.

[0017] Figures 8A-8E It is a simplified diagram of a graphical user interface that presents guidance information associated with the navigation of medical devices, based on some examples.

[0018] Figure 9 It is a simplified diagram based on some examples of medical systems.

[0019] Figure 10 It is a simplified diagram of a patient coordinate space side view based on some examples, including a medical device mounted on an insertion component.

[0020] The embodiments of this disclosure and their advantages are best understood by referring to the following detailed description. It should be understood that the same reference numerals are used to identify the same elements illustrated in one or more of the figures, wherein the illustrations in the figures are for the purpose of illustrating embodiments of this disclosure and not for the purpose of limiting the examples of this disclosure. Detailed Implementation

[0021] During the planning and execution of medical procedures using steerable medical devices, anatomical boundaries, or virtual “hazard fences,” can be defined by identifying anatomical surfaces that the medical device must avoid during the procedure. Anatomical boundaries can shield vulnerable portions of the anatomical body near the target location or protect other anatomical structures of interest from unintentional penetration by the medical device. Structures of interest that include vulnerable anatomical structures or surfaces can include, for example, the pleura, lung fissures, large alveoli, and blood vessels. For example, puncturing the pleura during a medical procedure can lead to a dangerous pneumothorax in the patient. Generating anatomical boundaries corresponding to the pleura allows the operator to limit the path of the medical device to avoid vulnerable portions of the anatomical body. Candidate paths identified during procedure planning may be deemed invalid if they pass within a threshold distance of a vulnerable portion of the anatomical body or if they breach a vulnerable portion. An illustrative example of a graphical user interface for planning medical procedures (including, but not limited to, lung biopsy procedures) is provided below. The graphical user interface may include various modes, including data selection mode, mixed segmentation and planning mode, preview mode, save mode, management mode, and review mode. Some aspects of graphical user interfaces resemble features described in the following documents: U.S. Provisional Patent Application No. 62 / 357,217, filed June 30, 2016, entitled “Graphical User Interface for Displaying Guidance Information During and Image-Guided Procedure”, and U.S. Provisional Patent Application No. 62 / 357,258, filed June 30, 2016, entitled “Graphical User Interface for Displaying Guidance Information in a Plurality of Modes During and Image-Guided Procedure”, which are incorporated herein by reference in their entirety.

[0022] Figure 1 The illustration shows an elongated medical device 100 extending within a branching anatomical passage 102 of an anatomical region 104 (e.g., a human lung). The anatomical region 104 has an anatomical reference frame (X). A Y A Z AThe distal end 108 of the medical device 100 can advance through the anatomical passage 102 to perform medical procedures, such as biopsy procedures, at or near the target 106. The anatomical region 104 may also include fragile surfaces or surfaces of additional interest during the performance of the medical procedure. For example, the pleura 110 and lung fissures 112 may be surfaces of interest because damage to these surfaces during the medical procedure could harm the patient. Preoperative planning steps can be performed to plan the medical procedure prior to its execution. In some embodiments, a robot-assisted medical system can be used to plan and perform the medical procedure.

[0023] Figure 2A The illustration depicts a method 200 for generating anatomical boundaries during the planning of a medical procedure, based on several examples. For instance, planning a medical procedure typically includes planning an initial tool position and a trajectory between one or more anatomical targets. One or more method steps can be performed on the same robot-assisted medical system used to perform a biopsy or other medical procedure. Alternatively or additionally, planning can be performed on a different system, such as a workstation dedicated to preoperative planning. The planning for the medical procedure can be saved (e.g., as one or more digital files) and transferred to the robot-assisted medical system used to perform the biopsy procedure. The saved planning may include a 3D model, airway identification, target locations, trajectories to the target locations, routes through the 3D model, and / or the like.

[0024] Method 200 is illustrated as a set of operations or procedures 210 to 250, and continues to be referenced. Figure 3 The description includes illustrations of a graphical user interface 400 in planning mode during the execution of method 200, based on some examples. For instance... Figure 3 As shown, in the planning mode, a traversal path 425 for a medical device (e.g., device 100) can be planned through an anatomical region (e.g., anatomical region 104) between the patient's mouth (or any other insertion point of the medical device into the patient's body) and a target location 416 (e.g., the location of target 106). The traversal path 425 can be generated by a user, a remote operating control system, or a combination of manual and automatic input.

[0025] At step 210, image data of the anatomical region is displayed. For example, as shown... Figure 3 As shown, image data 410 corresponding to the patient's three-dimensional anatomical region 104 is displayed via a graphical user interface 400. The displayed image data 410 has an image reference frame (X). I Y I Z IImage data 410 may include, for example, computed tomography (CT) image data. In various alternative examples, image data may be generated using imaging techniques such as magnetic resonance imaging (MRI), fluorescence examination, thermal imaging, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or other imaging techniques like these. Figure 3 A 3D anatomical model is illustrated. Additionally or alternatively, image data 410 may include individual planes or “slices” of image data, as depicted in a thumbnail view 412 of the graphical user interface 400. In some examples, image data 410 is graphically segmented to identify and indicate the location of anatomical features, such as anatomical targets, airways in the lungs, blood vessels, or the like.

[0026] The graphical user interface 400 displays information associated with planning medical procedures in one or more views that are visible to the user. Although in Figure 3 The illustration depicts an illustrative arrangement of views, but it should be understood that the graphical user interface 400 can display any suitable number of views in any suitable arrangement and / or on any suitable number of screens. In some examples, the number of views displayed simultaneously can be changed by opening and closing views, minimizing and maximizing views, moving views between the foreground and background of the graphical user interface 400, switching between screens, and / or otherwise completely or partially blurring views. Similarly, the arrangement of views, including their size, shape, orientation, order (in the case of overlapping views), and / or such arrangements, can be changed and / or can be user-configurable.

[0027] At step 220, the target location within the anatomical region is determined. For example, refer to... Figure 3 The target location 416 of the target 106 can be determined within the anatomical region 104. In some examples, user input for identifying the target location 416 in the three-dimensional anatomical region 104 is received via a user input device. In some embodiments, the user input may be provided by the user via a mouse, touchscreen, stylus, or the like. In some embodiments, the target location 416 can be determined using, for example, image analysis techniques without user input, to determine the target location based on shape, density, position, or other characteristics determined from computer analysis of image data. Figure 3 The depicted target location 416 can be displayed via a graphical user interface 400. In this example, the target location 416 may correspond to a biopsy site. Also in this example, the target location 416 may represent the location of a target nodule.

[0028] like Figure 3As further illustrated, in some examples, after determining the target location 416, a target edge region 418 can be generated around the target location 416. The target edge region 418 can be spherical, elliptical, rectangular, or any other shape. The target location 416 can represent the center C of the target edge region 418, such that the target edge region 418 extends outward from the target location 416. In the example where the anatomical target 106 represents a target nodule, the center of the target nodule can represent the center C of the target edge region 418. In some examples, the target edge region 418 is defined by a radius R. The radius R can be adjusted by the user, which will be described in more detail below.

[0029] At point 230 in the process, and as Figure 3 As shown, anatomical boundaries are defined. For example, an anatomical boundary may indicate a portion of a surface in a three-dimensional anatomical region 104, such as a vulnerable surface, or other surfaces of interest, and the anatomical boundary should be avoided by medical instruments during medical procedures (e.g., not touched, crossed, and / or punctured). In some examples, the surface of interest in anatomical region 104 may be the surface of the pleura (e.g., pleura 110), pulmonary fissures (e.g., fissure 112), large alveoli, and / or blood vessels. In some examples, a graphical representation of the anatomical boundary 420 may be displayed via a graphical user interface 400. According to some examples, the graphical representation of the anatomical boundary 420 may be overlaid on image data 410. Figure 3 The depicted three-dimensional representation of the anatomical boundary 420 can be displayed as a semi-transparent or mesh-line grid or similar on a three-dimensional anatomical model. The mesh may include multiple vertices. Additionally, in the thumbnail view 412, the cross-sectional representation of the anatomical boundary 420 can be displayed as a curve overlaid on a CT slice. The graphical user interface 400 may also include an adjustment menu 430 that allows the user to adjust factors used to determine the anatomical boundary 420 or other graphically illustrated risk areas.

[0030] Figure 2B The illustration shows several inputs 262-268 that can be used to determine the anatomical boundary 420 at process 230 of method 200. Any one of inputs 262-268 can be omitted when determining the anatomical boundary 420, and other inputs can be used. As previously mentioned, the target location input 262 can influence the determination of the anatomical boundary 420. For example, the position, shape, and size of the anatomical boundary 420 may be influenced by the position, shape, and size of the target location 416.

[0031] Adjusting the information input 264 can also, or alternatively, affect the determination of the anatomical boundary 420. For example, the target edge region 418 may intersect with the surface of interest in the anatomical region to define the anatomical boundary 420. The anatomical boundary 420 may represent the intersection region between the target edge region 418 and the surface of interest. The size of the intersection region may vary depending on the length of the radius R of the target edge region 418. For example, the intersection region may increase with the length of the radius R. Therefore, in some examples, the size of the anatomical boundary 420 shown in the image data 410 may increase with the length of the radius R of the target edge region 418. Figure 3 As further shown, the graphical user interface 400 may include an adjustment menu 430 that allows the user to adjust the size of the target edge region 418. For example, the adjustment menu 430 may include an adjustment mechanism such as a slider 432. The slider 432 illustrates a range of sizes for the target edge region 418, from a minimum size 434 to a maximum size 436. The size of the target edge region 418 may be based on the length of its radius R. Therefore, the minimum size 434 of the target edge region 418 corresponds to the shortest length of radius R. Similarly, the maximum size 436 of the target edge region 418 corresponds to the longest length of radius R. When the slider 432 is adjusted, the radius R of the target edge region 418 can also be adjusted. In some examples, the default radius R is 40 mm. In other examples, the radius R can be larger or smaller. For example, the radius R can range from 0 mm to 40 mm or from 40 mm to 120 mm. In other examples, the radius R can be greater than 120 mm. In some examples, the slider 432 can be adjusted based on user input.

[0032] Image data characteristics input 266 can also, or alternatively, influence the determination of anatomical boundaries 420. For example, anatomical boundaries 420 can represent regions of image data 410 with characteristics such as high-intensity gradients, as high-intensity gradients indicate the presence of surfaces of interest (e.g., the pleura of the lung, lung clefts, blood vessel walls, etc.). Computer vision techniques, including machine learning algorithms, can be applied to image data 410 to identify image data characteristics associated with candidate anatomical boundaries. Consistent with such examples, anatomical boundaries 420 can include, or be a portion of, candidate anatomical boundaries determined by such computer vision or machine learning techniques.

[0033] Patient movement input 268 can also, or alternatively, affect the determination of anatomical boundaries 420. During navigation, the patient's anatomy, and therefore the three-dimensional anatomical model, may move or deform due to forces, for example, from the medical device (e.g., medical device 100), lung exhalation and inspiration, and / or the beating of the heart. Deformation can be measured, for example, by a shape sensor in the medical device, or predicted by simulation, and can be applied to the three-dimensional anatomical model. Anatomical boundaries 420 can also be adjusted or deformed to correspond to actual or planned deformations of the three-dimensional anatomical model.

[0034] In some examples, after the anatomical boundary 420 is determined, the user (e.g., a surgeon) can enter a manual adjustment mode to further refine the anatomical boundary 420. In manual adjustment mode, the surgeon can manually edit and / or fine-tune the anatomical boundary 420. In this regard, the user can adjust any part of the anatomical boundary 420. For example, the user can smooth one or more parts of the anatomical boundary 420, connect one or more parts of the anatomical boundary 420 that may be disconnected, expand one or more parts of the anatomical boundary 420, reduce one or more parts of the anatomical boundary 420, etc. In some examples, icons (e.g., buttons, pop-ups, etc.) can be presented in the graphical user interface 400 to allow the user to enter manual adjustment mode. For example, icon 460 can be presented in an adjustment menu 430 that says "Modify Edge" or "Manual Edit". In such examples, the user can enter manual adjustment mode by clicking and / or pressing icon 460.

[0035] In manual adjustment mode, in some examples, the three-dimensional anatomical boundary can be manually adjusted, such as anatomical boundary 420 superimposed on image data 410. The user can rotate the anatomical boundary 420 in the graphical user interface 400 to view it from all angles. This allows the user to determine if all desired adjustments have been made to the anatomical boundary 420. For example, to manipulate / modify the anatomical boundary 420, the user can click and drag a portion of the anatomical boundary 420. Alternatively or additionally, the user can modify the anatomical boundary 420 to fill in any missing portions and / or connect any disconnected portions. These adjustments can be made via additional buttons, sliders, or other icons that may be presented in the adjustment menu 430. In some examples, the user can move at least a portion of the anatomical boundary 420 outward in a direction away from the target edge region 418. In some examples, the user can move at least a portion of the anatomical boundary 420 inward in a direction toward the target edge region 418. The user can select discrete portions of the anatomical boundary 420 to move toward or away from the target edge region 418. Alternatively or additionally, the entire anatomical boundary 420 can be moved toward or away from the target edge region 418. In some examples, the user can draw modified anatomical boundaries by hand, in the form of polylines, with a series of plotted points, or in a similar manner.

[0036] In an alternative example, in manual adjustment mode, the two-dimensional anatomical boundary (e.g., the anatomical boundary 420 illustrated in thumbnail view 412) can be manually adjusted. In such an example, the graphical user interface 400 can present view 412 in the main viewing window and image data 410 in the thumbnail view. The anatomical boundary 420 can be adjusted in two dimensions in one or more of the ways discussed above regarding three-dimensional adjustment. For example, a user can click and drag a portion of the anatomical boundary 420 to fill in any missing parts of the anatomical boundary 420, and / or connect any disconnected parts of the anatomical boundary 420. In some examples, the anatomical boundary 420 can be moved away from or toward the target edge region 418, as discussed above. Additionally, a user can draw modified anatomical boundaries by hand, in the form of polylines, as a series of plotted points, etc.

[0037] Refer again Figure 2A At process 240, a trajectory region can be determined around the path or tool trajectory between the medical device and the target location. For example, a trajectory region can be determined around the path between the distal end 108 of the medical device 100 and the target 106. In another example, Figure 4AThe illustration shows a portion 500 of an anatomical region (e.g., anatomical region 104) near target 520. A surface of interest 510 (e.g., a surface of pleura 110) extends near target 520. As described in process 230, an anatomical boundary 512 is defined on the surface of interest 510. The medical procedure may be a biopsy procedure or any other type of medical procedure in which a medical device 530 (e.g., medical device 100) is inserted into a portion 500 of the anatomical region near target 520. During the biopsy procedure, a biopsy tool (such as a biopsy needle) may extend from the distal end 535 (also referred to as the exit point) of the medical device 530 toward target 520. Therefore, during the biopsy procedure, the anatomical boundary 512 may be behind target 520 relative to the distal end 535, and thus there may be a risk of puncture if the needle, instrument 530, or other instrument extending from the distal end 535 extends too far beyond target 520. A trajectory path 546 extends from the distal end 535 of the medical device 530 through the target 520 to an intersection 555 on the surface of interest 510. The trajectory path 546 may have a distance D. In some embodiments, the direction of the trajectory path 546 corresponds to the orientation of the distal end 535 of the medical device 530. In some embodiments, the trajectory path 546 extends through the center C of the target 520. A trajectory region 542 extends around the trajectory path 546. The trajectory region 542 may have a three-dimensional volume. Figure 4A As shown, the trajectory region 542 is conical, but in other embodiments, the trajectory region 542 may be cylindrical, pyramidal, or any other suitable shape. In some embodiments, the trajectory region 542 is symmetrical about the trajectory path 546, but in other embodiments it may be off-center from the trajectory path 546. In some embodiments, uncertainties associated with the medical procedure (e.g., uncertainty about the location of the exit point 535, uncertainty about the location of the target 520, or both) may be factors in determining the trajectory region 542.

[0038] In some examples, after generating the target edge region 418, the target edge region 418 is analyzed to determine whether at least a threshold percentage of the target edge region 418 lies outside the surface of interest (e.g., the pleura of the lung). In some examples, the default threshold percentage for the target edge region 418 is 15%. In other examples, the threshold percentage for the target edge region 418 can be larger or smaller. For example, the threshold percentage can be in the range of 15%–30%. In other examples, the threshold percentage can be less than 15% or greater than 30%. In examples where the threshold percentage is not outside the surface of interest, the anatomical boundary 420 remains unchanged. In examples where the threshold percentage is outside the surface of interest, it is determined whether the target edge region 418 is close to the part of interest (e.g., blood vessels, heart, etc.), which may indicate a sensitive anatomical area that would raise concerns about potential accidental damage during the procedure.

[0039] If the target edge region 418 is not near the region of interest, the anatomical boundary 420 can be adjusted to include the portion of the target edge region 418 determined to be outside the surface of interest. For example, a portion of the anatomical boundary 420 can be extended such that all portions of the target edge region 418 are included within the anatomical boundary 420. To extend the anatomical boundary, a first margin is generated. At least a portion of the first margin can be illustrated as an extended portion of the anatomical boundary 420 and thus can be displayed in the graphical user interface 400. The center C of the target edge region 418 can also be the center of the first margin. The first margin can extend radially outward from its center. Additionally, the first margin can be larger than the target edge region 418. In other examples, the first margin can be the same size as the target edge region 418. In some examples, the size of the first margin is set based on the radius R of the target edge region 418. For example, the radius of the first margin can be 5 mm larger than the radius R. In other examples, the radius of the first margin can be larger or smaller. For example, the radius of the first margin can be 3 mm larger than the radius R. In other examples, the radius of the first edge can be 6 mm larger than the radius R. The lengths discussed above regarding the radius of the first edge are for illustrative purposes only—the radius of the first edge can be any other suitable length. After generating the first edge, a second edge can be generated. The second edge can be larger than the first edge. Like the first edge, the center C of the target edge region 418 can be the center of the second edge. The second edge can extend radially outward from its center. Additionally, the size of the second edge can be set based on the radius R of the target edge region 418. For example, the radius of the second edge can be 25 mm larger than the radius R. In other examples, the radius of the second edge can be larger or smaller. For example, the radius of the second edge can be 20 mm larger than the radius R. In other examples, the radius of the second edge can be 30 mm larger than the radius R. The lengths discussed above regarding the radius of the second edge are for illustrative purposes only—the radius of the second edge can be any other suitable length. After determining the second edge, the control system can use the second edge to smooth the first edge. This can be done to ensure that the anatomical boundary 420 smoothly transitions from the initially defined anatomical boundary 420 to the extended portion (i.e., the first edge) of the anatomical boundary 420 and back to the initially defined anatomical boundary 420. By not depicting corners or bends that may not exist in the surface of interest, a smooth anatomical boundary 420 including the first edge can more closely represent the surface of interest.

[0040] In other examples, if the target edge region 418 is close to the region of interest, a trajectory path (e.g., trajectory path 546) can be determined. In some examples, the trajectory path may point away from or substantially away from the region of interest. In such examples, the anatomical boundary 420 can be adjusted to include the portion of the target edge region 418 determined to be outside the surface of interest, as discussed above. In other examples, the trajectory path may point to or substantially point to the region of interest. In such examples, the control system may deactivate (e.g., delete, hide, etc.) the determined anatomical boundary 420 and instead prompt the user to manually generate the anatomical boundary. Various systems and methods for manually generating anatomical boundaries are described in U.S. Provisional Patent Application No. 62 / 741,157 (filed October 4, 2018) entitled “Graphical User Interface for Defining an Anatomical Boundary”, the entire contents of which are incorporated herein by reference.

[0041] At step 250, the region boundary can be determined based on one or more inputs, such as... Figure 5 As shown in the diagram. Figure 5 The diagram illustrates several inputs 272-278 that can be used to determine region boundaries. Anatomical boundary input 272 can influence the determination of anatomical boundaries (e.g., anatomical boundary 512). For example, region boundaries can be determined based on the intersection of trajectory region 542 and anatomical boundary 512. Figure 4A As shown in the example, the zone boundary 540 is determined based on the intersection of the anatomical boundary 512 and the trajectory zone 542. The zone boundary 540 can be a two-dimensional or three-dimensional region of surface 510, which may be at risk of being penetrated by biopsy instruments. Figure 4A In one example, the zone boundary 540 may extend entirely within the anatomical boundary 512, thus having an area smaller than that of the anatomical boundary 512. In an alternative example, the zone boundary 540 has the same dimensions and shape as the anatomical boundary 512. In other alternative examples, the zone boundary 540 may be a combination of the intersection of the anatomical boundary 512 and the trajectory zone 542 with the surface of interest 510. In some examples, the zone boundary 540 may include an additional edge 544 extending beyond the area directly within the trajectory zone 542. The additional edge 544 may be included to account for any uncertainties associated with the medical procedure (e.g., uncertainty about the location of the exit point 535, uncertainty about the location of the target 520, or both). In some examples, the zone boundary 540 may be determined in a binary manner (e.g., a given portion is considered risky or not risky) or in a progressive or continuous manner to reflect different risk levels at different locations.

[0042] Adjusting the input information 274 can affect the determination of the area boundary 540. In some examples, such as... Figure 3As shown, the adjustment menu 430 may also include a slider 438. The slider 438 illustrates the angular range of a trajectory area 542 surrounding the trajectory path 546 between the medical device 530 and the target location 520. In some examples, the slider 438 can be used to adjust the angle A of the trajectory area 542. In some examples, the default angle A of the trajectory area 542 is 120°. In other examples, the angle A of the trajectory area 542 can be larger or smaller. For example, the range of angle A can be from 60° to 120° or from 120° to 180°. In other examples, angle A can be less than 60° or greater than 180°. In some examples, user input can control the movement of the slider 438. In some examples, when the angle A of the trajectory area 542 decreases, the size of the area boundary 540 also decreases. In other examples, when the angle A of the trajectory area 542 increases, the size of the area boundary 540 also increases.

[0043] The multi-track path input 276 can also, or alternatively, affect the determination of the region boundary 540. (See reference) Figure 6 In some cases, there may be more than one trajectory path available for the medical device 530 to reach the anatomical target 520. In this case, depending on the selected trajectory path, the distal end 535 of the medical device 530 may be located in different orientations and / or locations within the patient's anatomy. Each configuration of the distal end 535 may correspond to its own trajectory path, trajectory region, and region boundary. For example, when the distal end 535a is in the configuration shown, the trajectory path 546a extends through the target 520, and the trajectory region 542a surrounding the trajectory path 546a generates a region boundary 540a at the intersection with the anatomical surface 510. Similarly, when the distal end 535b is in the configuration shown, the trajectory path 546b extends through the target 520, and the trajectory region 542b surrounding the trajectory path 546b generates a region boundary 540b at the intersection with the anatomical surface 510. Similarly, with distal end 535c in the configuration shown, trajectory path 546c extends through target 520, and trajectory area 542c surrounding trajectory path 546c generates zone boundary 540c at the intersection with anatomical surface 510. In this case, a final or composite zone boundary 540d is determined based on combined zone boundaries 540a, 540b, 540c, some of which may overlap each other. Thus, zone boundary 540d can indicate hazardous portions of surface 510 based on multiple trajectory paths of the instrument. Zone boundary 540d can be displayed as a single translucent or grid-line mesh on a three-dimensional anatomical model. In other examples, zone boundaries 540a, 540b, 540c can be displayed as multiple separate translucent or grid-line meshes on a three-dimensional anatomical model.

[0044] The trajectory path analysis input 278 may also, or alternatively, influence the determination of the region boundary 540. In some embodiments, the feasibility of one or more trajectory paths (e.g., trajectory path 546) may be analyzed to determine whether the distal end of the medical device is in a region of the patient anatomy separated from the target by a surface of interest. For example, it may be determined whether the distal end of the medical device is in a different lobe of the patient's lung, separated from the anatomical target by a pulmonary fissure. Since the fissure of the patient's anatomy separates the lobes of the patient's anatomy, in the case where the distal end of the device and the anatomical target are located in different lobes, the biopsy needle will pierce the fissure as it travels along the trajectory path between the distal end of the device and the anatomical target. While this discussion refers to fissures in the patient's anatomy, it should be understood that this discussion may also apply to any other part of interest (e.g., large alveoli, blood vessels, etc.). In some examples, analysis may be performed to determine whether a portion of trajectory path 546 intersects with a part of interest such as a fissure. The fissure may be modeled as a segmented mesh or voxel mask based on image data. If trajectory path 546 intersects with a fracture model, it can be determined that the trajectory path is unsafe and can be discarded, suppressed, or otherwise excluded from determining the region boundary 540. If the trajectory path does not intersect with a fracture, it can be determined that the trajectory path is acceptable and can be presented to the user as a candidate trajectory path.

[0045] Figure 4B The illustration shows a portion 500 of the anatomical region displayed on a graphical user interface 400 that provides guidance information during planning mode. Figure 4BIn this example, the guidance information may be depicted as a two-dimensional image, but in other embodiments it may be depicted as a three-dimensional image. In this example, image data 410a is combined with or overlaid by guidance information, which may include a graphical representation of the distal end of the instrument 535, trajectory path 546, trajectory area 542, intersection 555, anatomical boundary 512, and area boundary 540. Based on this guidance information, the user may be able to modify or fine-tune the trajectory path 546 as needed. For example, the operator may adjust the orientation of the distal end 535 of the medical device toward the anatomical target 520. As another example, providing more space between the distal end 535 and the intersection 555 may reduce the risk of needle puncture of surface 510 when the needle is extended. In some examples, pixels in the area boundary 540 may be displayed with different shades, colors, or translucent colors. Any guidance information graphics may be turned on or off automatically, by user selection, or by combination. Additionally or alternatively, an indication of whether the anatomical boundary 512 completely encompasses the area boundary 540 may be displayed or otherwise conveyed to the operator. During the planning process, a safety score can be calculated and provided to the operator, indicating the likelihood that the instrument and / or tool will breach the anatomical boundary 512. Based on the score, the trajectory path 546 can be adjusted or modified to achieve a safer route. Multiple trajectory paths with different safety scores can be offered to the operator for selection.

[0046] Additionally or alternatively, image data 410a may include a single plane or "slice" of image data, as depicted in a thumbnail view 412a of the graphical user interface 400. In some examples, thumbnail view 412a may also include guidance information, such as... Figure 4B As shown. The operator can modify or fine-tune the trajectory path 546 as needed using the guidance information in image data 410a and thumbnail 412a. In some cases, the guidance information is shown in image data 410a instead of thumbnail view 412a. In other cases, the guidance information is shown in thumbnail view 412a but not in image data 410a.

[0047] In some examples, one or more user inputs can be received to manipulate anatomical boundaries in the graphical user interface. Figure 7A The diagram illustrates a graphical user interface 400 within the planning patterns discussed above, based on some examples. Figure 7AIn some embodiments, the graphical user interface 400 includes a "show / hide" icon 450, which may be displayed, for example, in an adjustment menu 430. In some examples, the operator may want to more closely evaluate the portion of the three-dimensional anatomical model where the target location 416 is located. To achieve this evaluation, it may be beneficial to hide the anatomical boundary 420 in the image data 410. In some examples, when the anatomical boundary 420 is hidden, the data associated with the anatomical boundary 420 remains accessible in the control system—the anatomical boundary 420 is no longer simply displayed in the image data 410. Figure 7A As shown, the adjustment menu 430 includes a title “Pleural Edge”, which indicates that an anatomical boundary 420 has been generated. Icon 450 allows the anatomical boundary 420 to be hidden and displayed in the image data 410. For example, if the anatomical boundary 420 is displayed in the image data 410, it can be hidden based on user input. In some cases, the operator can use a user input device to select icon 450. When icon 450 is selected, the anatomical boundary 420 is hidden in the image data 410 (i.e., temporarily not displayed), as... Figure 7A As shown. To display the anatomical boundary 420 again in the image data 410, user input can be received to select icon 450. In several examples, icon 450 can be selected to toggle the display of the anatomical boundary 420 on and off as needed.

[0048] In some examples, the distance D between the exit point 535 and the intersection point 555 can still be displayed when the anatomical boundary 420 is hidden. In other embodiments, the distance D can also be hidden when the anatomical boundary 420 is hidden. Additionally or alternatively, any other features corresponding to the anatomical boundary 420 shown in the image data 410 (e.g., target location 416, target edge region 418, center C, radius R, etc.) can also be hidden when the anatomical boundary 420 is hidden. In some examples, sliders 432, 438 can be removed from the adjustment menu 430 when the anatomical boundary 420 is hidden, such as... Figure 7A As shown. When the anatomical boundary 420 is shown again in the image data 410, the sliders 432 and 438 can also be shown again in the adjustment menu 430. In other examples, the sliders 432 and 438 can still be displayed in the adjustment menu 430 when the anatomical boundary 420 is hidden.

[0049] like Figure 7AAs shown in the embodiments, the graphical user interface 400 may also include a "delete" icon 452, which may be displayed, for example, in the adjustment menu 430. In some examples, the operator may want to have newly generated anatomical boundaries, such as when a more accurate anatomical boundary can be obtained. In such an example, the current anatomical boundary 420 shown in the image data 410 can be deleted. In some examples, when the anatomical boundary 420 is deleted, the data associated with the anatomical boundary 420 is removed from the control system. For example, as... Figure 7B As shown, the adjustment menu 430 includes the title "Add Pleural Edge," which indicates that the anatomical boundary has not yet been generated and / or has been deleted. Icon 452 allows the deletion of the anatomical boundary 420 from image data 410. For example, if the anatomical boundary 420 is displayed in image data 410, it can be deleted based on user input. In some cases, the operator can utilize a user input device to select icon 452. When icon 452 is selected, the anatomical boundary 420 is deleted from image data 410, as... Figure 7B As shown, and may no longer be accessible in the control system. To re-display the anatomical boundary 420 in the image data 410, a new anatomical boundary 420 can be generated, as described above regarding... Figure 2A The subject of discussion.

[0050] In some examples, when the anatomical boundary 420 is deleted, the distance D between the exit point 535 and the intersection point 555 may also be deleted. Furthermore, when the anatomical boundary 420 is deleted, any other features shown in the image data 410 corresponding to the anatomical boundary 420 (e.g., target location 416, target edge region 418, center C, radius R, etc.) may also be deleted.

[0051] Figure 7C The diagram illustrates a graphical user interface 400 in the above planning pattern, based on some examples. Figure 7C In this embodiment, multiple anatomical boundaries 420, 560 are displayed via a graphical user interface 400. In some examples, there may be more than one anatomical target within the patient's anatomy. In such examples, the operator may want to evaluate some or all of the anatomical targets. In such examples, generating anatomical boundaries for some or all of the anatomical targets may be beneficial. Furthermore, the graphical user interface 400 may include menus corresponding to some or all of the anatomical targets. For example, as... Figure 7C As shown, the graphical user interface 400 includes a menu 580 labeled "Target 1" corresponding to the target location 516. (As...) Figure 7CAs further shown, the graphical user interface 400 includes a menu 590 labeled "Target 2" corresponding to target location 562. Each of the menus 580 and 590 may include some or all of the features described above with respect to the graphical user interface 400 (e.g., adjustment menu, "show / hide" icon, "delete" icon, etc.).

[0052] In some examples, when one of menus 580 and 590 is selected, details corresponding to the selected menu can be displayed in image data 410. For example, as shown... Figure 7C As shown, menu 590 is selected. Because menu 590 is selected, the details corresponding to target location 562 are displayed in image data 410. For example, image data 410 may include anatomical boundary 560, target location 562, target edge region 564, center C2 of target edge region 564, radius R2 of target edge region 564, traversal path 565, and / or any other details corresponding to target location 562. In some examples, when menu 590 is selected, image data 410 only displays the details corresponding to target location 562. In other examples, regardless of which menu 580 or 590 is selected, some or all of the details corresponding to target locations 562 and 566 can be displayed in image data 410. For example, as... Figure 7C As shown, anatomical boundaries 420 and target edge regions 418 can also be displayed in image data 410. In some embodiments, two menus 580 and 590 can be selected simultaneously, which can result in some or all of the details corresponding to the target location being displayed in image data 410. Furthermore, as discussed above, one or more of the anatomical boundaries 420 and 560 can be displayed as semi-transparent or mesh lines on the three-dimensional anatomical model. Additionally or alternatively, to further distinguish the anatomical boundaries 420 and 560, they can be displayed in image data 410 with different colors, patterns, etc.

[0053] Anatomical boundary 560 can be generated in the same manner as discussed above regarding the generation of anatomical boundary 420. Similarly, target edge region 564 can be generated and / or adjusted in the same manner as discussed above regarding target edge region 418. In several examples, one or both of anatomical boundaries 420, 560 can be shown, hidden, or deleted, as discussed above. For example, if the operator wants to analyze a portion of the 3D anatomical model surrounding target location 562, the operator can choose to hide anatomical boundary 420. Similarly, if the operator wants to analyze a portion of the 3D anatomical model surrounding target location 416, the operator can choose to hide anatomical boundary 560. Although Figure 7COnly two anatomical boundaries, 420 and 560, are illustrated. However, it should be understood that any number of anatomical boundaries corresponding to any number of anatomical targets can be displayed. For example, three anatomical boundaries corresponding to three anatomical targets can be displayed, or four anatomical boundaries corresponding to four anatomical targets can be displayed, and so on.

[0054] like Figure 7C As further shown, menu 590 indicates that anatomical boundary 560 is labeled "pleural edge". In some examples, operator O may want to rename anatomical boundary 560 to more easily determine which anatomical boundary has been selected in an example displaying multiple anatomical boundaries. For example, the operator can enter different names for anatomical boundary 560, such as "pleural edge 2", "cleft edge", "vessel", or any other desired name. Furthermore, the operator can rename anatomical boundary 420 in menu 580 to any desired name, such as "pleural edge 1", "cleft edge", "vessel", or any other desired name.

[0055] During navigation, the control system 612 can display the trajectory area and / or anatomical boundaries using a three-dimensional anatomical model of the patient's anatomy or other anatomical views presented on the user's display. As described above, prior to the navigation phase of a medical procedure (e.g., a biopsy procedure), the plan for the medical procedure can be saved (e.g., as one or more digital files) and transferred to the robot-assisted medical system used to perform the procedure. The saved plan may include a 3D model, airway identification, target location, trajectory to the target location, route through the 3D model, and / or similar elements. Alternatively or additionally, all data acquired prior to the planning phase (which may include the saved plan) can be transferred to the robot-assisted medical system. This data can then be used to dynamically generate anatomical boundaries during the navigation phase of the medical procedure. The anatomical boundaries can be displayed on a graphical user interface. As the medical device navigates through the patient's anatomy, the shape, orientation, and / or orientation of the anatomical boundaries can change as the orientation and orientation of the medical device change.

[0056] For example, Figures 8A-8E The diagram illustrates a simplified representation of a graphical user interface 800, according to some examples, that presents guidance information associated with navigation of a medical device (e.g., device 100) through the patient's anatomy (which may be represented by anatomical region 104). The graphical user interface 800 is used in navigation mode during medical procedures. Figure 8A As shown, image data 810 corresponding to the patient's three-dimensional anatomical region 104 is displayed via a graphical user interface 800. The displayed image data 810 has an image reference frame (X). I Y I Z IImage data 810 may include data obtained using one or more imaging techniques, such as CT, MRI, fluorescence examination, temperature recording, ultrasound, OCT, thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or the like. Image data 810 may include multiple images of a three-dimensional anatomical region 104, wherein... Figure 8A A 3D anatomical model is illustrated. Additionally or alternatively, image data 810 may include individual planes or “slices” of image data, which may be depicted in a thumbnail view within the graphical user interface 800. In some examples, image data 810 is graphically segmented to identify and indicate the location of anatomical features, such as anatomical targets, airways in the lungs, blood vessels, or the like. It should be understood that the graphical user interface 800 may display any suitable number of views in any suitable arrangement and / or on any suitable number of screens, as discussed above regarding graphical user interface 400.

[0057] like Figure 8A The diagram illustrates a traversal path 820 of the patient anatomy between the medical device passing through the patient's mouth (or any other insertion location of the medical device into the patient) and the target location 840 (e.g., the location of target 106). Traversal path 820 illustrates the path taken by the medical device as it is inserted and navigates through the patient anatomy.

[0058] In several examples, a trajectory area 830 can be defined. As the medical device is inserted and navigates through the patient's anatomy, the trajectory area 830 can be displayed in the graphical user interface 800. In some examples, the trajectory area 830 can be partially transparent, making the airway of the three-dimensional anatomical model behind the trajectory area 830 visible. The orientation and orientation of the trajectory area 830 can be based on the orientation and orientation of the distal portion of the medical device. For example, the trajectory area 830 is displayed in the graphical user interface 800 so that it appears to protrude from the distal portion of the medical device. In this way, the orientation and orientation of the trajectory area 830 can substantially match the orientation and orientation of the distal portion of the medical device. Therefore, as the orientation / orientation of the medical device changes during navigation, the orientation / orientation of the trajectory area 830 also changes accordingly.

[0059] The depth Dt of the trajectory region 830 can be based on the extension length of the tool that can extend from the medical device. For example, the depth Dt of the trajectory region 830 can be set to the maximum extension length of the tool. The maximum extension length can be measured from the distal end of the medical device to the distal end of the tool when the tool is fully extended from the medical device. In some examples, the maximum extension length is 30 mm. In other examples, the maximum extension length can be larger or smaller. For example, the maximum extension length can be 25 mm. In other examples, the maximum extension length can be 35 mm. The lengths discussed above regarding the maximum extension length of the tool are discussed for illustrative purposes only—the maximum extension length of the tool can be any other suitable length. As discussed above, in some examples, the tool is a needle.

[0060] like Figure 8A As shown, the adjustment menu 850 may include a slider 852. Slider 852 illustrates the range of angles of the trajectory area 830. In some examples, slider 852 can be used to adjust angle B of the trajectory area 830. In some examples, the default angle B of the trajectory area 830 is 120°. In other examples, angle B of the trajectory area 830 can be larger or smaller. For example, the range of angle B can be from 60° to 120° or from 120° to 180°. In other examples, angle B can be less than 60° or greater than 180°. In some examples, user input can control the movement of slider 852.

[0061] like Figure 8A As further shown, the graphical user interface 800 includes a "show / hide" icon 854, which may be displayed, for example, in an adjustment menu 850. In some examples, the operator may want to more closely evaluate a portion of the three-dimensional anatomical model where the distal portion of a medical device is located. To achieve this evaluation, it may be beneficial to hide the trajectory area 830 in the image data 810. In some examples, when the trajectory area 830 is hidden, the data associated with the trajectory area 830 remains accessible in the control system—the trajectory area 830 is simply no longer displayed in the image data 810. Icon 854 allows the trajectory area 830 to be hidden and displayed in the image data 810. For example, if the trajectory area 830 is displayed in the image data 810, the trajectory area 830 can be hidden based on user input. In some cases, the operator can use a user input device to select icon 854. When icon 854 is selected, the trajectory area 830 is hidden in the image data 810 (i.e., temporarily not displayed). To display the trajectory area 830 again in the image data 810, user input selecting icon 854 can be received. In several examples, icon 854 can be selected to toggle the display of track area 830 on and off as needed.

[0062] In several examples, during navigation, the patient's anatomy, and therefore the three-dimensional anatomical model, may be moved or deformed by forces, for example, from the medical device, lung exhalation and inspiration, and / or heartbeat. Deformation can be measured, for example, by a shape sensor in the medical device, or predicted by simulation, and can be applied to the three-dimensional anatomical model. The traversal path 820, trajectory area 830, target location 840, and anatomical boundary 860 are shown. Figure 8C It can also be adjusted to correspond to the actual or planned deformation of the three-dimensional anatomical model of the patient's anatomy.

[0063] As discussed above, the orientation / orientation of the traversal path 820 and trajectory region 830 are updated as the medical device traverses the patient's anatomy. Figure 8A In the example shown, the medical device is located near the main carina (MC) of the patient's anatomy. Therefore, the trajectory region 830 is also located near the main carina. Figure 8B In this case, the medical device has been further inserted into the patient's anatomy. Because the trajectory area 830 is shown as if it emanates from the distal portion of the medical device, the trajectory area 830 is displayed at the further insertion position of the medical device.

[0064] exist Figure 8C In this case, medical devices have been further inserted into the patient's anatomy. Also, as... Figure 8C The diagram shows anatomical boundary 860. During the navigation phase of a medical procedure, anatomical boundary 860 can be displayed in the graphical user interface 800 when the distal portion of the medical device is within a certain threshold distance of the target location 840. The default threshold distance between the distal portion of the medical device and the target location 840 is 60 mm. In other examples, the threshold distance can be larger or smaller. For example, the threshold distance can range from 35 mm to 70 mm. In other examples, the threshold distance can be less than 35 mm or greater than 70 mm. When the distal portion of the medical device reaches the threshold distance (e.g., 60 mm from the target location 840), then anatomical boundary 860 can be displayed on the graphical user interface 800 as an overlay on image data 810. Waiting for the display of anatomical boundary 860 until the medical device reaches the threshold distance provides the user with an indication that the medical device is approaching the target location 840. In an alternative example, the threshold distance is measured between the distal portion of the medical device and the exit point of the medical device.

[0065] As medical devices are further inserted into the patient's anatomy, such as Figure 8DAs shown, design 862 (e.g., pattern, color, edges, crosshairs, shading, etc.) can be displayed on anatomical boundary 860. In some examples, design 862 can be displayed when the distal portion of the medical device is within a certain distance (e.g., a first distance, which may be less than a threshold distance) of target location 840. The first distance can be set to any desired distance within the threshold distance. Waiting for design 862 to be displayed until the medical device reaches the first distance provides the user with an indication that the medical device is getting closer to reaching target location 840.

[0066] Also Figure 8D As shown, the size and / or shape of the anatomical boundary 860 can vary depending on the orientation and orientation of the medical device, and therefore, the orientation and orientation of the trajectory region 830. Similar to the discussion above regarding the determination of the region boundary 540, portions of the anatomical boundary 860 displayed in the image data 810 can be shown based on the extension of the trajectory region 830 (which, if the depth of the trajectory region 830 extends to the surface of interest, could be these portions of the trajectory region 830 that intersect the surface of interest) and the intersection between the surface of interest (e.g., the pleura of the lung). Because the orientation and orientation of the trajectory region 830 change as the medical device navigates through the patient's anatomy, the extension of the trajectory region 830 also changes its orientation and orientation. Therefore, in some examples, the size and shape of the anatomical boundary 860 can change with the changing orientation / orientation of the trajectory region 830.

[0067] As the medical device is further inserted into the patient's anatomy (e.g., to or substantially near the exit point), such as Figure 8E As shown, design 864 (e.g., patterns, colors, edges, crosshairs, shading, etc.) can be displayed on anatomical boundary 860. In some examples, design 864 can be displayed when the distal portion of the medical device is within a certain distance (e.g., a second distance, which may be less than a threshold distance and may be less than the first distance discussed above) of target location 840. The second distance can be set to any desired distance within the threshold distance. Waiting for design 864 to be displayed until the medical device reaches the second distance provides the user with an indication that the medical device is getting closer to reaching target location 840. In some examples, display design 864 indicates to the user that the medical device has reached the exit point.

[0068] In some examples, designs 862 and 864 can be displayed as the medical device reaches discrete insertion distances, as discussed above. In other examples, the anatomical boundary 860 changes from a display without any design to a display of a first design 862 and a second design 864, corresponding to the dynamic insertion of the medical device into the patient's anatomy. This dynamic change in the display of the anatomical boundary 860 helps the user determine when the medical device is approaching the target location 840 and how close it may be. In examples where designs 862 and 864 include color, the color of the anatomical boundary 860 can gradually change (e.g., with a gradient) as the medical device is inserted closer to the target location 840. For example, the anatomical boundary 860 can gradually change from yellow when it is first displayed (e.g., when the distal portion of the medical device reaches a threshold distance), to orange when the distal portion of the medical device reaches a first distance, and then to red when the distal portion of the medical device reaches the exit point. The gradual change in the display of the anatomical boundary 860 can be illustrated in any other suitable manner. For example, as... Figure 8D and Figure 8E As shown, in the examples of designs 862 and 864 that include crosshairs, the crosshairs can gradually thicken as the medical device is inserted closer to the target location 840.

[0069] In alternative examples, the gradual change in the display of the anatomical boundary 860 can be illustrated by changes in the brightness of the anatomical boundary 860, the shadow of the anatomical boundary 860, the pattern of the anatomical boundary 860, the edge of the anatomical boundary 860, or any other suitable manner. In further examples, as the medical device reaches each of the threshold distance, the first distance, and the second distance, different audible indicators / indications can be presented to the user to indicate that the medical device is getting closer to the target location 840. Alternatively, as the medical device gets closer to the target location 840, gradually changing audible indicators can be presented to the user. For example, as the medical device gets closer to the target location 840, the volume of the tone may change (e.g., louder or softer), the pitch of the tone may change (e.g., from low to high or from high to low), etc. Any other audible indicators can be used to indicate that the medical device is approaching the target location 840. In further examples, as the medical device reaches each of the threshold distance, the first distance, and the second distance, different and / or gradually changing text indicators / indications can be presented to the user to indicate that the medical device is getting closer to the target location 840. In another alternative example, different and / or gradually changing tactile indicators / indications (e.g., tactile feedback) may change as the medical device gets closer to the target location 840.

[0070] In some embodiments, the planning and / or navigation techniques of this disclosure can be used in image-guided medical procedures performed using a remotely operated medical system, as described in further detail below. Figure 9 As shown, a remotely operated medical system 600 typically includes a manipulator assembly 602 for operating a medical device 604 while performing various procedures on a patient P positioned on a table T in a surgical setting 601. The medical device 604 may correspond to device 100. The manipulator assembly 602 may be a remotely operated, non-remotely operated, or hybrid remotely operated and non-remotely operated assembly, having selectable degrees of freedom of motion that may be motorized and / or remotely operated, and selectable degrees of freedom of motion that may be non-motorized and / or non-remotely operated. A main assembly 606, which may be inside or outside the surgical setting 601, typically includes one or more control devices for controlling the manipulator assembly 602. The manipulator assembly 602 supports the medical device 604 and may optionally include a plurality of actuators or motors that drive inputs on the medical device 604 in response to commands from a control system 612. The actuators may optionally include a drive system that, when coupled to the medical device 604, can advance the medical device 604 into a natural or surgically generated anatomical opening. Other actuation systems may move the distal end of the medical device 604 with multiple degrees of freedom, including three linear degrees of freedom (e.g., linear motion along the X, Y, and Z Cartesian axes) and three rotational degrees of freedom (e.g., rotation about the X, Y, and Z Cartesian axes). Additionally, actuators may be used to actuate the articulated end effector of the medical device 604 for grasping tissue in the jaws of a biopsy device and / or similar device.

[0071] The remote-operated medical system 600 further includes a display system 610 (which may include a graphical user interface 400) for displaying images or representations of the surgical site and medical device 604 generated by the sensor system 608 and / or the endoscopic imaging system 609. The display system 610 and the main component 606 may be oriented such that an operator O can control the medical device 604 and the main component 606 through remotely presented perception. Any of the previously described graphical user interfaces may be displayed on the display system 610 and / or the display system of a stand-alone planning workstation.

[0072] In some embodiments, medical device 604 may include components for surgery, biopsy, ablation, illumination, irrigation, or aspiration. Optionally, medical device 604, together with sensor system 608, may be used to collect (e.g., measure or survey) a set of data points corresponding to a position within an anatomical passage of a patient (e.g., patient P). In some embodiments, medical device 604 may include components of imaging system 609, which may include an imaging range component or imaging instrument that records concurrent or real-time images of the surgical site and provides the images to an operator or operator O via display system 610. Concurrent images may be, for example, two-dimensional or three-dimensional images captured by an imaging instrument positioned within the surgical site. In some embodiments, imaging system components may be integrally or removably coupled to medical device 604. However, in some embodiments, a separate endoscope attached to a separate manipulator component may be used with medical device 604 to image the surgical site. Imaging system 609 may be implemented as hardware, firmware, software, or a combination thereof, which interact with or are otherwise executed by one or more computer processors, which may include a processor of control system 612.

[0073] Sensor system 608 may include orientation / position sensor system (e.g., electromagnetic (EM) sensor system) and / or shape sensor system for determining the orientation, orientation, velocity, rate, attitude and / or shape of medical device 604.

[0074] The remote-operated medical system 600 may also include a control system 612. The control system 612 includes at least one memory 616 and at least one computer processor 614 for implementing control between the medical device 604, main component 606, sensor system 608, endoscopic imaging system 609, and display system 610. The control system 612 also includes programming instructions (e.g., a non-transitory machine-readable medium storing the instructions) to implement various operating modes of the remote operating system, including a navigation planning mode, a navigation mode, and / or a program mode. The control system 612 also includes programming instructions (e.g., a non-transitory machine-readable medium storing the instructions) to implement some or all of the methods described according to various aspects of the disclosure herein, including, for example, instructions for providing information to the display system 610, instructions for determining a target location, instructions for determining anatomical boundaries, instructions for determining a trajectory area, instructions for determining area boundaries, and instructions for receiving input from a user (e.g., operator O) into the planning mode.

[0075] Planning for medical procedures such as biopsy procedures can be stored and used by the control system 612 to provide automated navigation of the medical device or operator-assisted navigation for performing biopsy procedures. During navigation, the control system 612 can display anatomical boundaries and / or zone boundaries using a three-dimensional anatomical model of the anatomical region, an intraluminal view, or other anatomical views presented on the user's display. Anatomical boundaries and / or zone boundaries can also be displayed (e.g., overlaid) together with registered images from other imaging techniques, such as fluorescence examination images acquired during the medical procedure.

[0076] The control system 612 may optionally further include a virtual visualization system to provide navigational assistance to the operator O when controlling the medical device 604 during an image-guided surgical procedure. Virtual navigation using the virtual visualization system may be based on a reference to a preoperative or intraoperative dataset of acquired anatomical pathways. The virtual visualization system processes images of the surgical site imaged using imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), fluorescein scanning, thermography, ultrasound, optical coherence tomography (OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-ray imaging, and / or the like.

[0077] Figure 10 The illustration depicts a surgical environment 700 in which patient P is positioned on a surgical table T. Patient P can remain still within the surgical environment in the sense that the patient's overall movement is restricted by sedation, restraint, and / or other means. Circulatory anatomical movements, including the patient P's breathing and cardiac movements, can continue unless the patient is instructed to hold his or her breath to temporarily suspend respiratory movements. Within the surgical environment 700, there is an instrument reference system (X). S Y S Z SA medical device 704 (e.g., device 100, 604) is coupled to a device holder 706. In this embodiment, the medical device 704 includes an elongated device 710, such as a flexible catheter, coupled to a device body 712. The device holder 706 is mounted to an insertion stage 708 fixed within a surgical environment 700. Alternatively, the insertion stage 708 may be movable but has a known position within the surgical environment 700 (e.g., via a tracking sensor or other tracking device). In these alternatives, the medical device reference frame is fixed or otherwise known relative to a surgical reference frame. The device holder 706 may be a component of a remotely operated manipulator assembly (e.g., remotely operated manipulator assembly 602) coupled to the medical device 704 to control insertion movement (i.e., movement along axis A), and optionally, to control movement of the distal end 718 of the elongated device 710 in multiple directions including yaw, pitch, and roll. The instrument holder 706 or insertion stage 708 may include an actuator, such as a servo motor (not shown), for controlling the movement of the instrument holder 706 along the insertion stage 708.

[0078] In this embodiment, the sensor system (e.g., sensor system 608) includes a shape sensor 714. The shape sensor 714 may include an optical fiber extending within and aligned with the elongated device 710. In one embodiment, the optical fiber has a diameter of approximately 200 μm. In other embodiments, the size may be larger or smaller. The optical fiber of the shape sensor 714 forms an optical fiber bending sensor for determining the shape of the elongated device 710. Alternatively, an optical fiber including a fiber Bragg grating (FBG) is used to provide strain measurements in one-dimensional or multi-dimensional structures. Various systems and methods for three-dimensionally monitoring the shape and relative orientation of optical fibers are described in the following documents: U.S. Patent Application No. 11 / 180,389 (filed July 13, 2005) (disclosing "Fiber optic position and shape sensing device and method relating thereto"); U.S. Patent Application No. 12 / 047,056 (filed July 16, 2004) (disclosing "Fiber-optic shape and relative position sensing"); and U.S. Patent No. 6,389,187 (filed June 17, 1998) (disclosing "Optical Fibre Bend Sensor"), which are incorporated herein by reference in their entirety. In some embodiments, the sensor may employ other suitable strain sensing techniques, such as Rayleigh scattering, Raman scattering, Brillouin scattering, and fluorescence scattering. In some embodiments, the shape of the conduit may be determined using other techniques. For example, the history of the distal orientation of the elongated device 710 may be used to reconstruct the shape of the elongated device 710 over time intervals.

[0079] like Figure 10 As shown, the instrument body 712 is coupled and fixed relative to the instrument holder 706. In some embodiments, a fiber optic shape sensor 714 is fixed to a proximal point 716 on the instrument body 712. In some embodiments, the proximal point 716 of the fiber optic shape sensor 714 may move with the instrument body 712, but the position of the proximal point 716 may be known (e.g., via a tracking sensor or other tracking device). The shape sensor 714 measures the shape from the proximal point 716 to another point (such as the distal end 718 of the elongated device 710).

[0080] The elongated device 710 includes a channel (not shown) sized and shaped to receive a medical device 722. In some embodiments, the medical device 722 can be used for procedures such as surgery, biopsy, ablation, illumination, irrigation, or aspiration. The medical device 722 can be deployed through the elongated device 710 and used at a target location within the anatomy. The medical device 722 may include, for example, an image-capturing probe, a biopsy instrument, a laser ablation fiber, and / or other surgical, diagnostic, or therapeutic tools. The medical device 722 can be advanced from the distal end 718 of the elongated device 710 to perform a procedure and then retracted into the channel upon completion of the procedure. The medical device 722 can be removed from the proximal end of the elongated device 710 or from another optional instrument port (not shown) along the elongated device 710.

[0081] The elongated device 710 may also accommodate cables, linkages, or other steering controls (not shown) to controllably bend the distal end 718. In some examples, at least four cables are used to provide independent "up-down" steering to control the pitch of the distal end 718 and "left-right" steering to control the yaw of the distal end 718.

[0082] The orientation measuring device 720 can provide information about the orientation of the instrument body 712 as it moves along the insertion axis A on the insertion stage 708. The orientation measuring device 720 may include a resolver, encoder, potentiometer, and / or other sensors that determine the rotation and / or orientation of actuators that control the movement of the instrument carriage 706 and thus the movement of the instrument body 712. In some embodiments, the insertion stage 708 is linear. In other embodiments, the insertion stage 708 may be curved or have a combination of curved and linear segments.

[0083] In this description, specific details describing some embodiments have been set forth. Numerous specific details have been set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are illustrative and not restrictive. Those skilled in the art will recognize other elements within the scope and spirit of this disclosure, although not specifically described herein.

[0084] In other embodiments, implementations, or applications not specifically shown or described, elements described in detail with reference to one embodiment, implementation, or application may optionally be included, provided it is practicable. For example, if an element is described in detail with reference to one embodiment but not with reference to a second embodiment, that element may still be claimed to be included in the second embodiment. Therefore, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation, or application may be incorporated into other embodiments, implementations, or aspects unless otherwise specifically described, unless one or more elements would render one embodiment or implementation inoperable, or unless two or more elements provide conflicting functionality. Not all illustrated processes can be performed in all embodiments of the disclosed methods. Furthermore, one or more processes not explicitly illustrated may be included before, after, between, or as part of the illustrated processes. In some embodiments, one or more processes may be performed by a control system, or may be implemented at least in part in the form of executable code stored on a non-transitory, tangible, machine-readable medium that, when run by a control system, could cause one or more processors to execute one or more processes.

[0085] Any variations and further modifications to the described apparatus, instruments, and methods, as well as any further application of the principles of this disclosure, are fully contemplated, as would normally be expected by one of ordinary skill in the art to which this disclosure pertains. Furthermore, the dimensions provided herein are for specific examples, and different sizes, dimensions, and / or ratios may be contemplated for implementing the concepts of this disclosure. To avoid unnecessary descriptive repetition, one or more components or actions described according to one illustrative embodiment may be used or omitted where applicable to other illustrative embodiments. For the sake of brevity, numerous iterations of these combinations will not be described separately. For simplicity, in some cases, the same reference numerals are used throughout the drawings to refer to the same or similar parts.

[0086] The systems and methods described herein can be applied to the navigation and treatment of anatomical tissues via natural or surgically created connecting channels in any and many anatomical systems, including the lungs, colon, intestines, kidneys and renal calyces, brain, heart, circulatory system including the vascular system, and / or the like. While some examples of medical procedures are provided herein, any references to medical or surgical instruments and methods are non-limiting. For example, the instruments, systems, and methods described herein can be used for non-medical purposes, including industrial use, general robotic use, and sensing or manipulating non-tissue artifacts. Other example applications include cosmetic enhancements, imaging of human or animal anatomy, collecting data from human or animal anatomy, and training medical or non-medical personnel. Additional example applications include surgical procedures on tissue removed from human or animal anatomy (without returning the anatomy) and procedures performed on human or animal cadavers. Furthermore, these techniques can also be used in surgical and non-surgical medical treatments or diagnostic procedures.

[0087] One or more elements in the embodiments of this disclosure can be implemented in software to execute on a processor of a computer system, such as a control processing system. When implemented in software, elements in the embodiments of this disclosure may be code segments that perform various tasks. Programs or code segments can be stored in a processor-readable storage medium or device that can be downloaded via a transmission medium or communication link in the form of computer data signals embodied in a carrier wave. Processor-readable storage devices can include any medium capable of storing information including optical, semiconductor, and magnetic media. Examples of processor-readable storage devices include electronic circuits; semiconductor devices, semiconductor storage devices, read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM); floppy disks, CD-ROMs, optical disks, hard disks, or other storage devices. Code segments can be downloaded via computer networks such as the Internet, intranets, etc. Any of a variety of centralized or distributed data processing architectures can be employed. Programming instructions can be implemented as a number of individual programs or subroutines, or can be integrated into many other aspects of the system described herein. In one embodiment, the control system supports wireless communication protocols such as Bluetooth, IrDA (Infrared Data Communication), HomeRF (Home Radio Frequency), IEEE 802.11, Digital Enhanced Wireless Communication (DECT), Ultra Wideband (UWB), ZigBee, and wireless telemetry.

[0088] Note that the presented processes and displays may be inherently unrelated to any particular computer or other device. Various general-purpose systems can be used in accordance with the teachings and programming herein, or it may be demonstrated that a more specialized device can be constructed to perform the described operations. The necessary structures for various such systems will be shown as elements in the claims. Furthermore, embodiments of the invention are described without reference to any particular programming language. It will be understood that various programming languages ​​can be used to implement the teachings of the invention as described herein.

[0089] This disclosure describes various instruments, parts of instruments, and anatomical structures in three-dimensional space. As used herein, the term "orientation" refers to the position of an object or part of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian x, y, and z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or part of an object (three rotational degrees of freedom—e.g., roll, pitch, and yaw). As used herein, the term "attitude" refers to the orientation of an object or part of an object in at least one translational degree of freedom and the orientation of an object or part of an object in at least one rotational degree of freedom (up to six total degrees of freedom). As used herein, the term "shape" refers to a set of attitudes, orientations, or orientations measured along the object.

[0090] While certain exemplary embodiments of the invention have been described and illustrated in the accompanying drawings, it should be understood that these embodiments are illustrative only and not limiting to the broader invention, and that embodiments of the invention are not limited to the specific structures and arrangements shown and described, as various other modifications can be made by those skilled in the art.

Claims

1. A medical system comprising: Display system; Medical devices; as well as The control system, which is communicatively coupled to the display system, is configured to: The display system displays image data of the patient's anatomy. When the medical device navigates the patient anatomy, it determines that the distal portion of the medical device is within a threshold distance of a target location in the patient anatomy. Based on the determination that the distal portion of the medical device is within the threshold distance of the target location, an anatomical boundary is displayed via the display system, the anatomical boundary indicating the surface of the anatomical structure of the patient's anatomy.

2. The medical system of claim 1, wherein the control system is further configured to instruct the distal portion of the medical device at a first distance from the target location, the first distance being within the threshold distance.

3. The medical system of claim 2, wherein the control system is further configured to indicate that the distal portion of the medical device is at a second distance from the target location, the second distance being within the threshold distance and less than the first distance.

4. The medical system of claim 2, wherein instructing the distal portion of the medical device at the first distance includes generating a first pattern for display on the anatomical boundary.

5. The medical system of claim 2, wherein instructing the distal portion of the medical device at the first distance includes generating a first text instruction.

6. The medical system of claim 2, wherein instructing the distal portion of the medical device at the first distance includes generating a first auditory indication.

7. The medical system of claim 3, wherein instructing the distal portion of the medical device at the second distance includes generating a second pattern for display on the anatomical boundary.

8. The medical system of claim 3, wherein indicating the distal portion of the medical device at the second distance includes generating a second text indication.

9. The medical system of claim 3, wherein instructing the distal portion of the medical device at the second distance includes generating a second auditory indication.

10. The medical system of claim 1, wherein the control system is further configured to determine an extension distance between the distal end of the medical device and the distal end of a tool capable of extending from the medical device.

11. The medical system of claim 10, wherein the tool comprises a needle.

12. The medical system of claim 10, wherein the control system is further configured to: Determine the trajectory region surrounding the extended distance; and The trajectory area is displayed via the display system, wherein the trajectory area is displayed as if it protrudes from the distal portion of the medical device.

13. The medical system of claim 12, wherein the trajectory region is conical.

14. The medical system of claim 13, wherein the angle of the conical trajectory region is adjustable.

15. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions, which, when executed by one or more processors, are adapted to cause the one or more processors to perform a method comprising: The system displays image data of the patient's anatomy. When the medical device navigates the patient anatomy, it determines that the distal portion of the medical device is within a threshold distance of a target location in the patient anatomy. as well as Based on the determination that the distal portion of the medical device is within the threshold distance of the target location, an anatomical boundary is displayed via the display system, the anatomical boundary indicating the surface of the anatomical structure of the patient's anatomy.

16. The non-transitory machine-readable medium of claim 15, wherein the plurality of machine-readable instructions, when executed by the one or more processors, are further adapted to cause the one or more processors to instruct the distal portion of the medical device at a first distance from the target location, the first distance being within the threshold distance.

17. The non-transitory machine-readable medium of claim 16, wherein the plurality of machine-readable instructions, when executed by the one or more processors, are further adapted to cause the one or more processors to indicate the distal portion of the medical device at a second distance from the target location, the second distance being within the threshold distance and less than the first distance.

18. The non-transitory machine-readable medium of claim 16, wherein indicating the distal portion of the medical device at the first distance includes generating a first pattern for display on the anatomical boundary.

19. The non-transitory machine-readable medium of claim 16, wherein indicating the distal portion of the medical device at the first distance includes generating a first text indication.

20. The non-transitory machine-readable medium of claim 16, wherein indicating the distal portion of the medical device at the first distance includes generating a first auditory indication.

21. The non-transitory machine-readable medium of claim 17, wherein indicating the distal portion of the medical device at the second distance includes generating a second pattern for display on the anatomical boundary.

22. The non-transitory machine-readable medium of claim 17, wherein indicating the distal portion of the medical device at the second distance includes generating a second text indication.

23. The non-transitory machine-readable medium of claim 17, wherein indicating the distal portion of the medical device at the second distance includes generating a second auditory indication.

24. The non-transitory machine-readable medium of claim 15, wherein the plurality of machine-readable instructions, when executed by the one or more processors, are further adapted to cause the one or more processors to determine an extension distance between the distal end of the medical device and the distal end of a tool capable of extending from the medical device.

25. The non-transitory machine-readable medium of claim 24, wherein the tool comprises a needle.

26. The non-transitory machine-readable medium of claim 24, wherein the plurality of machine-readable instructions, when executed by the one or more processors, are further adapted to cause the one or more processors to: Determine the trajectory region surrounding the extended distance; and The trajectory area is displayed via the display system, wherein the trajectory area is displayed as if it protrudes from the distal portion of the medical device.

27. The non-transitory machine-readable medium of claim 26, wherein the trajectory region is tapered.

28. The non-transitory machine-readable medium of claim 27, wherein the angle of the conical trajectory region is adjustable.

Images