Temporary path planning in real-time endoscopic surgery

Abstract

A method for deriving a guided path based on a pre-loaded initial surgical regime in a real-time surgical endoscopic procedure, where the guided path leads to any selected site, including an unplanned ROI site observed on a radiographic image view or an endoscopic video view of an auxiliary endoscopy system. A new part s at an unplanned ROI is identified as a new examination target on one of the graphic visualization tools of the image-guided endoscope system, and a new navigation airway path leading to the new examination target is automatically calculated, so that the initial operation scheme is updated.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 482,636, filed on February 1, 2023, the entire contents of which are incorporated herein by reference. Government sponsorship

[0002] This invention was made with government support under Grant #CA151433 awarded by the National Cancer Institute of the National Institutes of Health. The government has certain rights in this invention. Technical Field

[0003] The present invention relates generally to surgical planning and guidance, and more particularly to real-time planning of a guidance path to a newly selected surgical site suitable for endoscopy. Background Art

[0004] Many endoscopic surgical procedures often require "dynamic" inspection or management of previously unplanned anatomical areas that arise during real-time surgery. Exemplary surgical areas (and related equipment) where such situations exist include:

[0005] 1. Chest and bronchoscopy;

[0006] 2. Abdomen and laparoscopy;

[0007] 3. Colon and colonoscopy;

[0008] 4. Bladder and cystoscopy.

[0009] For our research, we focus on lung / thoracic issues and lung cancer management (diagnosis, staging, surveillance, follow-up, treatment).

[0010] Image-guided bronchoscopy systems and newer robotic-assisted bronchoscopy systems are revolutionizing bronchoscopy practice, particularly for examining distal peripheral sites and performing more complex thoracic procedures [10, 30]. To use these systems, physicians first develop a surgical plan offline before performing real-time bronchoscopy [12, 28]. To develop the plan, the physician selects a diagnostic region of interest (ROI), such as a suspicious pulmonary nodule or an anatomical site identified for treatment, on the patient's three-dimensional (3D) chest computed tomography (CT) scan or, if available, a registered positron emission tomography (PET) scan. Next, for each ROI, an airway path from the trachea to the ROI is automatically derived by computer analysis or manually derived by visual inspection of the CT scan. Subsequently, during real-time surgery, the physician then uses the assisted bronchoscopy system, driven by the surgical plan, to navigate the bronchoscope to each ROI along its preplanned airway path.

[0011] However, many situations arise during live surgery that lead the physician to wish to examine some new, unplanned adjunct site. Given the complexity of anatomy, the complications involved in assessing the patient's condition—especially during live surgery where the patient's condition may be changing—and the ever-present difficulty of thoroughly evaluating a patient's imaging scans offline, such "dynamic" ad hoc findings are a natural occurrence. In fact, such unexpected findings may necessitate unforeseen adjustments to the preplanned procedure that are necessary and crucial to providing appropriate patient care. The following are examples of such intraoperative situations that arise in clinical practice:

[0012] 1. The pre-planned ROI airway path is found to be blocked or collapsed during real-time surgery, forcing the doctor to deduce a new path close to the ROI.

[0013] 2. The doctor finds an area that is particularly suitable or convenient for local treatment such as cryotherapy

[27] . To carry out the treatment, the doctor needs to call on different equipment, which in turn requires a path to the area.

[0014] 3. Because the airway path planning for a specific peripheral ROI is incomplete, physicians are often forced to use radiation-intensive fluoroscopy to navigate the bronchoscope to the final desired position.

[15] Although fluoroscopy is not very accurate in verifying 3D position, it is still necessary to do so. A more accurate approach is to select the desired location on CT and automatically calculate the new path while avoiding the accompanying radiation exposure.

[0015] 4. Physicians use time-consuming and error-prone manual analysis—for example, CT-based interactive virtual bronchoscopy (VB) and CT oblique views—to deduce airway pathways to deep peripheral sites.

[21] Automated methods will be more accurate and less time-consuming.

[0016] In addition to the four scenarios highlighted above, many other situations may arise in clinical practice that require real-time unplanned intraoperative adjustments during real-time bronchoscopy. These include:

[0017] 1. The physician determines that it is necessary to examine a small secondary tumor that may be located near the primary tumor identified earlier during offline planning.

[0018] 2. The doctor wants to examine the area within the diffuse ground-glass opacity shown on the preoperative CT scan.

[0019] 3. The doctor finds a location that shows suspected narrowing (stenosis) or obstruction of the airway and wishes to mark that location.

[0020] 4. Lymph nodes that were not identified preemptively during down-line surgical planning (ie, lymph node staging preparation) were considered for examination.

[0021] 5. The doctor wants to generate and save an airway path to the vicinity of the bronchoscope's current registration location because that location shows suspicious CT or PET findings that were not observed offline earlier. Based on the path, the doctor can initiate a treatment plan at that location.

[0022] 6. The physician wishes to preserve a location adjacent to the airway for future reference; the location will provide a robust RP-EBUS (radial probe endobronchial ultrasound) image of previously unidentified extraluminal structures; that is, the physician wishes to identify the "RP-EBUS sign."

[0023] 7. The physician wishes to examine the extraluminal sites identified on the imaging scans during surgery and further examine them using a second auxiliary bronchoscopic device such as optical coherence tomography (OCT) or RP-EBUS.

[0024] 8. Physicians use manually determined “bronchial signs” or “vascular signs” (i.e., visible airways or vessels near the intended diagnostic site on a 2D CT slice) to help navigate the bronchoscope to the vicinity of the site of interest observed on CT [1, 19]. Automatically discovering an appropriate airway path to the intended site will be more accurate and less prone to error.

[0025] 9. During bronchoscopic airway examination, whether using standard white light bronchoscopy, autofluorescence bronchoscopy, narrow-band imaging bronchoscopy, or optical coherence tomography bronchoscopy [4, 11, 20], physicians have observed signs of early lung cancer along the airway wall in the bronchoscopic video. Physicians wish to mark this area so they can return to it for further examination and / or treatment.

[0026] 10. Doctors realized that they needed to use ultra-thin bronchoscopes to reach deep, small airways—this required new guidance pathways

[18] .

[0027] To address these situations, assisted bronchoscopy systems need to be able to calculate airway paths to new assisted sites in real time during the procedure. Unfortunately, although assisted bronchoscopy systems have long supported automatic path planning based on preoperative offline planning [12, 28], there is currently no path planning method that can manage new diagnostic sites selected by the physician in real time during bronchoscopy. In addition, offline planning usually requires clearly defined 2D or 3D ROIs on the imaging scan, while some regions of interest do not correspond to ROIs themselves, but to locations that can only be determined during real-time surgery; for example, the site selected on CT to examine the diffuse ground-glass opacity area, or the current internal airway location observed in the bronchoscopic video view.

[0028] In summary, many surgical endoscopic procedures that require traversing complex anatomical regions (e.g., the lung airways or the abdomen) often require "dynamic" temporary inspection of previously unplanned anatomical regions that arise during real-time surgery. Unfortunately, there is a lack of methods or systems that can effectively and efficiently plan paths to newly selected anatomical regions in real time, whether for image-guided or robot-assisted endoscopy systems. Summary of the Invention

[0029] Embodiments of the present invention provide a method for deriving a guidance path during real-time surgical endoscopic procedures, wherein the guidance path can lead to any selected site, including an unplanned ROI site in a radiographic view or an endoscopic video view observed on a guidance display of an auxiliary endoscopy system.

[0030] Embodiments of the present invention include a computational method for deriving a desired guidance path and an interactive graphical tool called "Airway Locator" that is applicable within the framework of an assisted bronchoscopy system and facilitates the selection and management of regions of interest (ROIs).

[0031] Before the real-time endoscopic procedure was performed, information such as the airway tree, airway lumen surface, and airway centerline were calculated offline using the patient's chest CT scan and PET scan.

[0032] Based on this previously calculated information, an initial surgical plan is calculated offline, in which an initial ROI that is different from the unplanned ROI is clearly defined, and an appropriate airway path leading to the initial ROI is derived.

[0033] When preparing to begin a real-time endoscopic procedure, the physician may accidentally notice a location on the CT scan and wish to examine it. The physician will then invoke the airway locator to calculate a new navigation path to the unplanned location. This method updates the initial surgical plan by identifying the new location s in the unplanned ROI as a new examination target on one of the graphical visualization tools of the image-guided endoscopy system and then automatically calculating a new navigation airway path to the new examination target.

[0034] In another scenario, a surgical plan is similarly developed preoperatively based on available radiological data. During surgery, when a suspicious lesion, such as a mass, is observed on a CT scan, the surgeon follows a pre-planned path. The surgeon decides to examine the previously unplanned area. To do so, the surgeon selects a new ROI (Region of Interest) for the mass and then invokes the airway localizer to calculate a new guidance path to the unplanned area.

[0035] In another scenario, when a doctor is traveling along a pre-planned path during a routine airway examination or a pre-planned operation, a suspicious lesion is observed from the endoscopic video. At this time, the airway locator can be called to derive the airway guidance path using the method disclosed in the present invention. The suspicious lesion may be distributed along the airway wall and cannot be observed on the CT view. The video may be derived from white light bronchoscopy, narrow band imaging (NBI) bronchoscopy, or autofluorescence bronchoscopy.

[0036] Once a new navigation airway path is derived, the physician can preview the new navigation airway path using the image-guided endoscopy system's graphical visualization tools before guiding the endoscope along the new navigation airway path. The endoscope can be guided along the new navigation airway path through the cavitary organ system to reach the new examination target. The necessary distance from one observation site to the new site can also be calculated.

[0037] Any newly created ROI and associated airway pathway can be saved as part of the overall case study for future reference.

[0038] According to the disclosed method, the airway tree centerline comprises a set of directional paths, each path comprising a set of observation locations that are visited only once along the path, the path originating from the trachea and terminating at a distal airway branch, each observation location comprising a vector and a camera imaging orientation for visualization along the path. The six-parameter vector specifies a 3D (x, y, z) position and three orientation angles (α, β, γ).

[0039] The steps of automatically calculating a new navigation airway path include: 1. finding the observation part v closest to the new part s. c ; 2. Select the path containing the observation part v in the set of directional paths c The path p i ; 3. After deriving the compensation amount of ∈ observation part, calculate the new observation part v s =v c-∈ , as the initial airway path p to the new site s s The initial and final observation sites, the new observation site v s In the selected path p i The center is located at v c Before ∈ observation site, and the initial airway path p s Satisfy the constraints; 4. Calculate the new position s starting from v s And the vector z=sv pointing to the part s s , thereby obtaining the preliminary airway path p s The final observation position v f ; and use the initial airway path p s The final observation position v in f Replace vs , thereby providing a new navigation airway path to the new site s.

[0040] The constraints include the airway size and the endoscope tip size. The constraints include that the endoscope tip must be able to pass through the airway.

[0041] The new location s may be identified in a 2D CT slice, a 2D CT / PET slice, a sliding thin-section view, a stenosis location observed in a VB renderer, a location along the airway wall in a 2D MRI slice, or a video stream view.

[0042] A second auxiliary device capable of traveling through the airway to the new location can be used to navigate the new navigation airway path to further diagnose or treat the new ROI location. The second auxiliary device is selected from an optical coherence tomography probe, a cryotherapy probe, or a NaG laser diagnostic or therapeutic probe.

[0043] This method integrates smoothly with real-time clinical workflow and is particularly useful when unexpected situations arise during real-time assisted bronchoscopy.

[0044] This method provides a unique guided path planning approach for endoscopic navigation within hollow organ systems to new, previously unplanned anatomical sites of interest.

[0045] The hollow organ system may be the airways of the lungs or the colon, stomach, bladder, or hollow areas within the pelvic / abdominal cavity.

[0046] The endoscope may be a bronchoscope, colonoscope, laparoscope or cystoscope.

[0047] The ROI may be a suspected tumor, a cancerous nodule, a suspected airway wall site, a treatment delivery site, a lesion along an organ wall, or a location of routine visual inspection.

[0048] The radiological imaging data is a chest CT scan, a PET scan or a magnetic resonance imaging scan.

[0049] ROI inspection may involve performing a visual assessment of the ROI, delivering a treatment to the ROI, or harvesting dissected tissue from the ROI by the endoscopist.

[0050] In one embodiment of the present invention, an automated method is capable of deriving a guided path in real time to a region of interest (ROI) identified during surgery. In a second embodiment, a software system that interacts with an existing image-guided bronchoscopy system is used to facilitate user interaction in identifying a new ROI and creating an associated path. The software runs on a Windows-based PC. The software and method are also adaptable to many other surgical applications involving endoscopic traversal of luminal organ systems for ROI inspection, biopsy, diagnosis, monitoring, or treatment.

[0051] During the procedure, the physician interacts with the airway locator and the display of the assisted bronchoscopy system in real time. When a site is selected on either display view, the airway locator calculates the expected airway path to that site in real time. By having the airway locator automatically derive the airway path to that site "on the fly" (temporarily) during the live procedure, the physician can immediately:

[0052] 1. Guide the bronchoscope to the site for a closer examination.

[0053] 2. Make real-time surgical decisions regarding patient diagnosis, investigation, or treatment based on previously unplanned surgical adjustments.

[0054] 3. Preserve observable findings for future reference and deduce a route to re-access the site during subsequent bronchoscopy. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 The ROI location in the right lung near the diaphragm selected by the doctor on the PET / CT coronal slicer (left view) and the PET / CT cross-sectional slicer (right view) is shown.

[0056] Figure 2 Shown is the Pathway Derivation tab of the Airway Locator tool.

[0057] Figure 3 Derivation of the airway locator leading to Figure 1 The expected guidance path of the selected ROI site in .

[0058] Figure 4 Shown is the viewing of the selected ROI and its newly calculated guidance path in the Virtual Navigator's 3D surface tool and endoluminal renderer.

[0059] Figures 5A to 5D An example of temporary airway path calculation and subsequent guidance is shown. Left-top-left: Fusion of 2D PET / CT coronal slices (color bar shows PET SUV scale). Left-top-right: 3D airway tree rendering. Left-bottom: Endoluminal renderer (VB Viewer). Right: Airway locator interface. Figure 5A System view at the beginning of a preplanned bronchoscopy. Figure 5B This is a system view after deriving the path of the unplanned ROI area. Figure 5C This is a system view for guiding navigation along the radial direction of the new gas path to the unplanned ROI area. Figure 5D This is the system view when the navigation reaches the unplanned ROI area.

[0060] Figure 6AShown are the ROI locations (suspicious tumors) selected on the coronal PET / CT slicer view and viewed on all three PET / CT viewers (transverse, sagittal, coronal).

[0061] Figure 6B The figure shows the physician indicating to the airway locator that it is a new region of interest and creating a path, as shown in FIG. Figure 5B As shown;

[0062] Figures 7A to 7D Another example of “dynamic” guidance by an image-guided bronchoscopy system after ROI selection is shown. Figure 7A 2D CT sagittal view is shown. Figure 7B A 2D CT coronal block view is shown. Figure 7C A 3D airway tree rendering is shown. Figure 7D The airway locator interface is shown.

[0063] Figures 8A-8D Shows the way to Figures 7A-7D Real-time bronchoscopic guidance with a new ROI defined in [Figure 2]. The left view is a 2D CT coronal block; the middle view is a 3D airway tree rendering; the right view is a combination of a real-time bronchoscopic video view (left) and a registered 3D endobronchial VB view (right). Figure 8A This is a systematic view of the proximal trachea. Figure 8B This is a systemic view of the distal trachea; Figure 8C This is a systemic view of the right mainstem bronchus. Figure 8C This is a systemic view of the right secondary bronchus.

[0064] Figure 9A Shown is the use of a virtual navigator to select a suspicious mucosal lesion site on a bronchoscopy video frame.

[0065] Figure 9B Shown Figure 9A Mapping of the selected lesion to the corresponding CT-based VB view surface.

[0066] Figures 10A-10D Shown is the display of the virtual navigator guidance system after derivation of a guidance path to a selected mucosal lesion. Figure 10A 2D CT coronal section. Figure 10B 2D CT sagittal section. Figure 10C Rendering of a 3D airway tree. Figure 10D This is the airway locator interface.

[0067] Figures 11A-11CShows guided navigation along a calculated airway path. The left-upper-left view is a 2D CT coronal section. The left-upper-right view is a 3D airway tree rendering. The lower left view is the endoluminal renderer (VB view). The right view is the airway locator interface. Figure 11A This is a systematic view of the proximal trachea. Figure 11B This is a systemic view of the distal trachea. Figure 11C This is a systemic view of the final observation point of the airway pathway.

[0068] Figure 12A Automated “dynamic” mucosal lesion detection in narrow-band imaging bronchoscopy videos is shown.

[0069] Figure 12B Shown is a 3D airway tree surface rendering showing the automatically calculated airway paths leading to the lesion.

[0070] Figure 12C Shown are the registered real-time video view and the CT-based reference VB view during re-entry of the lesion by image-guided bronchoscopy. DETAILED DESCRIPTION

[0071] According to an embodiment of the present invention, a planning method and an associated interactive tool, an airway locator, are provided for dynamically (temporarily) deriving a guidance path to any newly selected region of interest in real time during a live endoscopic surgical procedure. An endoscope can then be guided through a luminal organ system along the derived guidance path to the region of interest.

[0072] The present invention provides a unique method for real-time generation of guidance paths to newly selected unplanned regions of interest.

[0073] 1. Top-level overview

[0074] The airway locator method can be integrated into the existing multimodal image-guided bronchoscopy system developed in our laboratory for planning and guiding multimodal bronchoscopic procedures [2, 12, 16, 17, 33].

[0075] First, the physician follows the standard two-stage process required by the image-guided bronchoscopy system: 1) offline surgical planning; followed by 2) real-time guided bronchoscopy. These two stages are detailed below.

[0076] Phase 1 - Offline Surgical Planning - This phase involves the following standard procedures.

[0077] (a) Using the patient's chest CT scan and PET scan (if available), the airway tree, airway lumen surface, and airway centerline are automatically calculated.

[0078] (b) Develop a routine surgical plan specific to the assisted bronchoscopy system. This involves identifying clinically relevant ROIs and deriving appropriate airway pathways to each ROI.

[0079] (c) Save all computational data (e.g., surgical plan, airway tree) and other data to the case study data structure

[17] .

[0080] Stage 2 - Live Boot - The standard procedure for this stage is as follows.

[0081] (a) In the operating room, the surgical plan is loaded and the guidance display of the assisted bronchoscopy system is initialized. The display typically includes multiple visualization tools that display 2D CT / PET slices, CT-based virtual bronchoscopy (VB) renderings, global 3D airway tree renderings, real-time video streams from video bronchoscopy, or accompanying text-based guidance instructions in the case of endobronchial ultrasound.

[0082] (b) The physician performs a pre-planned bronchoscopic procedure.

[0083] Regarding the above process, we used previously validated CT / PET lung image processing methods to create surgical plans [6, 9, 12-14, 23, 24, 32]. In addition, we used previously developed 2D and 3D graphics tools for radiographic image viewing and video analysis [2, 8, 17, 29].

[0084] When performing a pre-planned bronchoscopic procedure, doctors often encounter unexpected events or observations. Specifically, doctors may encounter findings on the display that interest them or alter the course of the procedure. In either case, doctors may wish to guide the bronchoscope closer to the newly discovered, unplanned area for a closer examination or may deem it crucial.

[0085] The above situation requires the use of the method disclosed herein, which requires a "dynamic" temporary update of the existing surgical plan previously developed offline. The top-level steps of this method are as follows.

[0086] Real-time "dynamic" surgical plan updates

[0087] 1. To initiate a modification / addition to an existing pre-planned surgical plan, the physician selects a location on one of the graphical visualization tools displayed on the assisted bronchoscopy system monitor. The location may appear on a 2D CT slice, a fused 2D CT / PET slice, a sliding thin-section view, the location of the stenosis observed in the VB renderer, or a location in the video stream view.

[0088] 2. The physician activates the airway locator by signaling to it that he wishes to consider bronchoscopy of the newly selected ROI.

[0089] 3. The airway locator automatically calculates the airway path to the site using the method detailed in Section 2 below.

[0090] It should be noted that the selected part can be located at any selected position, such as the position indicated by the viewer used to specify the part.

[0091] Once the new path is derived, the physician can use the assisted bronchoscopy system to guide the bronchoscope along the newly calculated airway path to the new location. Furthermore, before navigating, the physician can preview the path derived by the airway locator using the assisted bronchoscopy system's display. Finally, the physician can save any newly created ROIs and associated airway paths as part of the overall bronchoscopy case study for future reference.

[0092] 2. Path Calculation

[0093] According to step 3 above, the automated method integrated into the airway locator performs a series of calculations to determine the new path. First, the method uses the airway trees, surfaces, and centerlines previously calculated during the offline planning phase and saved as part of the case study.

[0094] It should be noted that the pre-calculated airway centerline includes N directional paths p i The set P of P={p i ,i=1,2…,N}. (1).

[0095] Each path p i ∈P starts from the trachea and proceeds to adjacent airway branches of progressively deeper (higher order) levels until it terminates at one of the N specific distal airway branches that constitute the segmented airway tree. i Also includes a unique continuous observation area v j The set of observation sites that start from the trachea, are visited once and only once along the path, and end at the observation site v within the i-th distal airway branch Di ;Right now

[0096] Among them, each observation part v j ∈V, V represents the complete set of observation sites required to construct all centerlines covering the airway tree, and v1 located in the trachea is always the first observation site to start the path.

[0097] The observation location consists of: 1) a 6-parameter vector specifying the 3D (x, y, z) position and three orientation angles (α, β, γ); and 2) an up vector specifying the camera imaging orientation when visualizing along the path. Therefore, overall, the data structure {P, V} representing the airway centerline precomputed in the earlier offline planning phase is used as an additional common input. It should be noted that all of the above definitions follow the standards adopted for airway centerlines, branches, and observation locations, as described in [12, 22].

[0098] The goal here is to calculate an airway path that leads to the site s selected by the user during real-time surgery using one of the guidance system display tools. The specific steps are as follows:

[0099] 1. Find the observation part v closest to part s c ∈V.

[0100] 2. Explicitly include v c The path p i ,i=1,2…,N. There may be multiple feasible paths. Any one of them can be selected as p i .

[0101] 3. As mentioned in

[12] , the tip of the bronchoscope must be able to fit into the v c To meet this requirement, the method uses the tip and airway dimensions and uses constraints to find a feasible path as described in

[12] . In addition, we derive the compensation for each observation location, thus obtaining v s =v c-∈ As a new airway path to site s s The initial and final observation sites.

[0102] More specifically, observing the location v s In the selected path p i The center is located at v c Before∈ observation position; that is

[0103] This results in the inclusion of p i Middle until v s Initial airway path for all observation sites p s : p s ={v1,…,v s-1 ,v s}. (4)

[0104] 4. Calculation starts with v s And the vector z=sv pointing to the part s s .

[0105] 5. By solving the problem from z and v s The current heading angle (α vs ,β vs ,γ vs ) to calculate the Euler angles (ψ,θ,φ).

[0106] 6. Create a new observation area v f =(x s ,y s ,z s ,ψ,θ,φ), where (x s ,y s ,z s ) is v s 3D position coordinates.

[0107] 7. Set the initial path p s The new observation site v in f Replace v s , from which the final expected airway path to the new position s is derived p s ={v1,…,v s-1 ,v f}. (5)

[0108] Then along the new path p s During navigation, the physician may wish to know the remaining distance to reach a new ROI. Therefore, during real-time guided navigation, the airway locator provides information about the location of the ROIs (x s ,y s ,z s ), the position of the current observation part v (x v ,y v ,z v ) and feedback of the distance d between v and r.

[0109] 3. Airway Locator Implementation

[0110] The Airway Locator tool is an interactive mechanism for creating new guided pathways to newly observed, unplanned anatomical sites during real-time bronchoscopy. The tool is integrated into an existing image-guided bronchoscopy system called Virtual Navigator [2, 12, 17, 33].

[0111] The airway locator is always used in conjunction with other virtual navigator visualization tools. In addition, as mentioned above, it requires offline calculation of the initial surgical plan before the real-time guided bronchoscopy is performed.

[0112] To use the airway locator in conjunction with the virtual navigator during real-time guided bronchoscopy, follow these steps:

[0113] 1. Load the offline surgical plan into the virtual navigator.

[0114] 2. Invoke any desired virtual navigator tools on the guidance system display. Examples of such tools include [2, 8, 9, 12, 16, 17, 29, 33]: multimodal PET / CT 2D slice viewer, sliding slice viewer, tube viewer, 3D airway tree surface renderer, CT-video matching tool, intraluminal VB renderer, 2D CT projection tool, and video analysis tool. For PET / CT, slice block, and projection tools, consider transverse, coronal, and sagittal orientations. Simultaneously invoke the airway localizer at this time.

[0115] This completes the display setup of the guidance computer for real-time surgery.

[0116] 3. The doctor now performs the guided bronchoscopy according to the offline plan.

[0117] 4. During the procedure, if the doctor discovers a new ROI or region of interest, the following actions are performed:

[0118] (a) The doctor selects a location within the desired new ROI in any invoked visualization tool; thereby setting the virtual navigator system's view state to the location of the newly selected ROI. This ROI will serve as the target of the new guided path.

[0119] Figure 1 An example of selecting a right lung region near the diaphragm is shown. A physician selects a right lung region of interest (ROI) near the diaphragm on the PET / CT coronal slicer. The viewer indicates the region with a red crosshair and xyz coordinates from the CT scan. The selected region is simultaneously displayed on the PET / CT cross-sectional slicer.

[0120] (b) The physician now interacts with the Airway Locator tab to derive the intended path - see Figure 2 Before any action is taken, all fields in the tab are empty (marked with "NaN"). There are two ways to do this:

[0121] i. Step-by-step operation: Press the "Select ROI", "Show Path" and "Show View" buttons successively to select the ROI location and derive all necessary information required for the new airway path.

[0122] ii. Fast construction: All computational steps are performed continuously at one time to derive the airway path.

[0123] The end result is a new guidance path and a 3D cube graphical marker that can be used to represent the ROI location on the virtual navigator system display.

[0124] Figure 3 shows a sample output after path calculation. Figure 1 The user performs all necessary steps to derive the desired new airway path to the ROI. The dialog box is now populated with all the parameters that specify the new path. The parameters indicate the xyz coordinates of the ROI, the original path p used to construct the new path, and the i ∈P, the offset of the best observation part, and the viewing angle of the best observation part.

[0125] 5. If necessary, the clinician can now preview the new path by first invoking "Show View" on the Airway Localizer's Path Derivation tab and playing the path on the system monitor using the standard movie controls on the Virtual Navigator. This also puts the Virtual Navigator into a ready state, ready to guide the bronchoscope to the new ROI.

[0126] Figure 4 An example preview is shown at a location within 21.5 mm of the ROI site, where the selected ROI and its newly calculated guidance path can be viewed in the Virtual Navigator's 3D Surface Tool and Endoluminal Renderer. The blue line indicates the new path. The orange cube in the 3D Surface Tool and the green cube in the Endoluminal Renderer both indicate the ROI site. The orange cylinder icon and green pointer illustrate the current system viewing position of both visualization tools—the system is positioned near the final target on the path 21.5 mm from the ROI site, as shown in the Endoluminal Renderer. It is important to note that the other blue ROIs displayed in the 3D Surface Tool correspond to the pre-planned ROIs that were clearly defined before the live procedure.

[0127] Finally, the physician can save the new ROI location and route as part of a case study for future reference.

[0128] 6. The physician can now perform a guided bronchoscopy using the new guidance path. During guided navigation, the physician can use the "Distance Calculator" tab in the Airway Locator to view the distance required to reach the new ROI. This functionality is further illustrated in the application example in the next section.

[0129] 4. System Implementation

[0130] The present disclosure includes a path planning method and an associated airway locator software module that can be integrated into an existing multimodal bronchoscopy planning and guidance system called Virtual Navigator [2, 3, 5, 7-9, 12, 16, 17, 22, 25, 33, 34]. The Virtual Navigator environment includes a toolkit that facilitates a standard two-stage workflow for completing real-time image-guided bronchoscopic procedures. The Virtual Navigator can receive the following data sources: 1) chest CT scan; 2) PET / CT study, including whole-body PET scan and chest CT scan; 3) standard white light bronchoscopy video; 4) convex probe endobronchial ultrasound video; 5) radial probe endobronchial ultrasound video; 6) autofluorescence bronchoscopy video; and 7) narrow band imaging bronchoscopy video. Tools exist for CT / PET analysis, offline surgical planning, deformable registration of CT and PET scans, offline CT / PET visualization, image-guided bronchoscopy (with and without endobronchial ultrasound), and multimodal video analysis. The boot system software has a large number of tools for image visualization and user interaction, as described in reference.

[0131] The software runs on a Windows PC and interacts with the bronchoscopy hardware during real-time surgery. It is written in C++ using Microsoft Visual Studio. In our latest study, we used a Dell Precision 7920 Tower workstation (64-bit Windows 10, 64GB of memory, an Intel Xeon Gold 6138 20-core 2.0GHz processor) for software development and system testing, including an NVIDIA RTX 2080Ti graphics card and a Matrox ClarityUHD frame grabber. The virtual navigator software uses multiple libraries, including VTK, Qt, and OpenCV. Many computationally intensive functions utilize CUDA and a GPU.

[0132] All software of the present invention are developed and tested in the described environment.

[0133] 5. Application Examples

[0134] Application examples are provided herein that demonstrate various ways to practice the present invention.

[0135] Example 1 :Tumor temporary observation

[0136] Figures 5A-5D A complete example is provided for a patient with lung cancer (Case 21405-108). Figures 5A-5DFigure 1. A fused 2D PET / CT coronal section (left-upper-left) (the color bar shows the PET SUV scale); a 3D airway tree rendering (left-upper-right); an endoluminal renderer (VB viewer) (left-bottom); and the airway locator interface (right). All views are synchronized to the same position during guided bronchoscopy, indicated by: 1) crosshairs on the PET / CT view; 2) an orange bronchoscope tip icon on the blue airway guidance path; 3) a blue line on the VB view; and 4) quantitative data in the airway locator tool.

[0137] The patient underwent a combined 3D CT / PET study and gave informed consent to be enrolled in a research project at our affiliated hospital. The surgical plan was first derived offline using available radiographic data before surgery. Figure 5A 、 Figure 5B 、 Figure 5C and Figure 5D Different time points during simulated bronchoscopy using our image-guided bronchoscopy system are shown.

[0138] Figure 5A The figure shows the status of the guidance system display at the beginning of a pre-planned bronchoscopy. The doctor is now navigating along the pre-planned path p and has reached the observation site v at the base of the tracheal apex on the path. 15 The location is marked with a red crosshair on the PET / CT slice, while the green annotation indicates the 3D location (x 15 ,y 15 ,z 15 ), CT HU (Hounsfield Unit) values, and PET SUV (Standard Uptake Value). In the 3D airway tree rendering, this location is indicated by an orange cylinder icon on the blue line representing path p. Finally, the VB view shows the v 15 Finally, the airway locator is currently in the initialization state. It is worth noting that at this location, the PET / CT view clearly shows a cancerous mass in the left upper lobe of the lung - the doctor decides to examine this previously unplanned area. To do this, the doctor selects a new ROI location s for the tumor as indicated by the red crosshairs, and then calls the airway locator to calculate a new guidance path to this unplanned area, as shown in Figure 1. Figure 6A and Figure 6B As shown. Figure 6A As shown in FIG, a ROI (suspicious tumor) is selected on the coronal PET / CT slice view and observed on all three PET / CT viewers (transverse, sagittal, and coronal). The physician signals the airway locator that the selected ROI is a new region of interest and creates a path, as shown in FIG. Figure 6BWhen the doctor clicks the "Quick Create" button on the airway locator, the tool immediately calculates a new airway path p to the selected site s s .

[0139] After the doctor clicks "Show View" on the airway locator, the new path p s At this time, it is displayed on the 3D airway tree and VB view and initialized at the path starting point, such as Figure 5B The airway locator also provides information about the path p s The final observation position v f To ensure that the bronchoscope can capture the path during subsequent guided navigation, s For the complete view of s at the end point, we use an offset compensation ∈ = 20 voxels. This ensures that the terminal observation site v f Located behind the airway wall and maintained a distance of approximately 10 mm from it (CT resolution in this case was approximately 0.5 mm in the x, y, and z axes).

[0140] Next, Figure 5C The guidance navigation along the new air path radially to the new position s is shown, and the arrival at the main bulge (observation position v 291 ) when booting the system in p s The Airway Localizer provides information about the current 3D position of the bronchoscope and the remaining distance to the ROI (as shown in the Airway Localizer's Distance Calculation tab). Figure 5D Shows the arrival position s (observation position v f =v 834 ) system display. The PET / CT viewer indicates how close the bronchoscope has navigated to s. The 3D airway tree shows the position and orientation of the bronchoscope tip toward s. The VB view displays a green cube ROI icon representing site s. Finally, the airway locator indicates that the bronchoscope has navigated to within 17.9 mm of the suspected tumor site s.

[0141] Example 2 "Dynamic" guided bronchoscopy for tumors

[0142] Figures 7A-7D and Figures 8A-8D Figure 2 shows a guided bronchoscopy of a previously unplanned ROI. For this example, we used a 3D printed airway model in combination with 3D CT images of a lung cancer patient (case 21405-116) who was enrolled in a research project at our affiliated hospital with informed consent. The surgical plan was first derived offline using the available radiographic data before surgery. During surgery (with the airway model replacing the patient), new ROIs were selected on the 2D sagittal CT slices, and the corresponding new airway guidance path p was derived. s .

[0143] Figures 7A-7D The image-guided bronchoscopy system after selecting a new ROI and the derived airway path to the ROI is shown. s The ROI is a suspicious nodule in the right upper lobe. The location of the ROI is indicated by red crosshairs in the 2D CT sagittal and coronal views and by a red cube in the 3D airway tree rendering. Furthermore, quantitative data related to the ROI is displayed on the Airway Locator tool. The airway path of the ROI is displayed on the Airway Locator interface, and the blue line in the 3D airway tree rendering represents the path. Figure 7A A 2D CT sagittal section is shown. Figure 7B A 2D CT coronal block view is shown. Figure 7C A 3D airway tree rendering is shown. Figure 7D The airway locator interface is shown.

[0144] Figures 8A-8D Shows that along p s At four consecutive positions Figures 7A-7D The physician followed the basic procedures discussed in the literature [12, 22, 25] and navigated along the airway path p within the airway tree model. s Navigate the bronchoscope and guide the system to synchronize all monitor views to each position.

[0145] Figure 8A 、 Figure 8B 、 Figure 8C and Figure 8D The states of the display of the image-guided bronchoscopy system at different observation positions along the airway guidance path are shown respectively. Figure 8A 、 Figure 8B 、 Figure 8C and Figure 8D The left view shows a 2D CT coronal block; the middle view shows a 3D airway tree rendering; and the right view shows a combination of a live bronchoscopic video on the left and a registered 3D endoluminal VB view on the right. During guided bronchoscopy, all views are automatically synchronized to the same position, indicated by: 1) crosshairs on the CT block; 2) an orange bronchoscope tip icon on the blue airway guidance path; and 3) a blue line on the VB view.

[0146] Figure 8A A view of the guidance system is shown of the initial observation site v1 within the trachea. Figure 8B Shows the view when the doctor reaches the distal end of the trachea; it corresponds to the airway path p s Observation site v on 232The physician now continues to navigate the bronchoscope toward the right lung, following the blue line path on the VB view shown. Figure 8C shows the system status when the doctor reaches the end of the right main bronchus; which corresponds to p s Observation site v on 339 .at last, Figure 8D The figure shows the bronchoscope navigated to maximum access to the right secondary bronchus. At this airway tree bifurcation, the surgeon navigated to the right superior branch using the VB view, as indicated by the blue line. The procedure was stopped here because the airway path terminated at a relatively deep branch that was too narrow to navigate during the live procedure. The Olympus EVIS Exera II BF TYPE P180 4.9 mm bronchoscope used in this study was too large to pass through the p in the airway tree model. s of the remaining airway.

[0147] Example 3 "Dynamic" guidance of bronchoscopy for suspicious mucosal lesions

[0148] Figures 9A-9B 、 Figures 10A-10D and Figures 11A-11C This example shows the real-time temporary ROI selection and subsequent bronchoscopic guidance for a suspicious mucosal lesion observed on the airway wall. For this example, we used 3D CT images and bronchoscopic videos of a lung cancer patient (case 20349-3-90) who had given informed consent to participate in a research project at our affiliated hospital.

[0149] First, mucosal lesions were observed in bronchoscopic videos during routine airway examination. Using the video analysis tools integrated into the virtual navigator, a ROI was identified at the suspected lesion site in the selected video frame [2]. The location of the ROI on the video frame was then correlated with the patient's 3D CT scan using an intraluminal VB renderer, providing the known 3D coordinates of the new site s. Figure 9A Shown is the use of a virtual navigator to select a suspicious mucosal lesion site (green box) on a bronchoscopy video frame. Figure 9B The mapping of the selected lesion to the corresponding simultaneous CT-based VB view surface is shown.

[0150] Given the location of the lesion s identified in the real-time bronchoscopic video, the airway locator is invoked to derive the airway guidance path p s . Figures 10A-10D The figure shows the display of the virtual navigator guidance system after the guidance path to the selected mucosal lesion is derived. Unlike the previous example where the ROI is not distributed along the airway wall and is identified using CT data, the ROI in this example is identified using video and is distributed along the airway mucosa. Figure 10A and Figure 10BIn the CT-based view, the red crosshairs pinpoint the ROI location at the edge between the airway (dark area) and tissue (gray area). Figure 10A A 2D CT coronal section is shown. Figure 10B A 2D CT sagittal section is shown. Figure 10C A 3D airway tree rendering is shown. Figure 10D The Airway Locator interface is shown. The ROI location in each view is indicated by: 1) red crosshairs on the sagittal and coronal views; 2) red areas on the airway walls in the 3D airway tree rendering; and 3) quantitative data from the Airway Locator tool.

[0151] Figures 11A-11C The following is a simulation guidance procedure, where the doctor follows the derived airway path p s Navigate the bronchoscope until the lesion is located on the airway wall.

[0152] Figure 11A 、 Figure 11B and Figure 11C The displays of the image-guided bronchoscopy system are shown in specific positions. Figures 11A-11C The left-upper-left view shows a 2D CT coronal section, the left-upper-right view shows a 3D airway tree rendering, the left-lower view shows the endoluminal renderer (VB view), and the right view shows the airway locator interface. Except for real-time virtual bronchoscopy, all views automatically synchronize to the same position during guided bronchoscopy, indicated by: 1) crosshairs on the CT view; 2) an orange bronchoscope tip icon on the blue airway guidance path; 3) a blue line on the endoluminal rendering view; and 4) quantitative data from the airway locator tool.

[0153] Figure 11A The bronchoscope is shown at the start of the endotracheal path. The bronchoscope is located at the observation site v1 of the original pre-planned path p36. Figure 11B Shows the bronchoscope reaching the distal end of the trachea (observation site v 236 ) navigation status. As shown in the 3D airway tree and VB views, the bronchoscope is now clearly close to the lesion; the distance calculator of the airway locator shows that the remaining distance from the current observation site to the ROI is 38.8 mm. Finally, Figure 11C Shows the final observation site of the bronchoscope path v 320 Because the airway locator automatically adjusts the viewing angle to the final observation site, the ROI is centered on the VB view. As indicated by the airway locator, the bronchoscope has been navigated to within 17.5 mm of the lesion.

[0154] Example 4: "Dynamic" localization of suspicious mucosal lesions during narrow-band imaging bronchoscopy

[0155] Figures 12A-12C An example of automated “dynamic” mucosal lesion detection and subsequent airway path calculation in narrow band imaging (NBI) bronchoscopy is provided. NBI bronchoscopy uses an airway illumination source that highlights the vascular structures of the airway mucosa

[26] . The resulting video images are known to show enhanced vascular structures. The locations of the airway wall that exhibit such structures often correspond to suspected early-stage lung cancer sites

[31] .

[0156] For our example, we use patient case 21405-197, who also gave informed consent to participate in the research project of our affiliated hospital. The doctor first performed an airway examination of the main airways of both lungs using an Olympus NBI bronchoscope. The obtained video stream was automatically processed by our NBI lesion detection software built into the virtual navigator system

[11] . Based on this, the lesion was found in the 158th frame of the airway examination video, and the 121st frame showed that the lesion was located in the center of the video, as shown in Figure 11. Figure 12A Using the obtained lesion bounding box as the ROI site, we then automatically calculated the airway path from the trachea to the site. Figure 12B 3D airway tree surface rendering, which shows the automatically calculated airway path to the lesion (blue line). When the bronchoscope subsequently returns to the site according to the new guidance path, it is navigated to the detected lesion at frame 121. Figure 12C Figure 2 shows the registered real-time video view and the CT-based reference VB view during image-guided bronchoscopy retracing of the lesion; the green area indicates the lesion. For this test, we simulated the lesion by using the recorded airway NBI video as input to the virtual navigator system. Figure 12C The bronchoscope retraction procedure is shown.

[0157] 6. Discussion

[0158] Newer assisted bronchoscopy systems, whether image-guided or robotic, now enable physicians to perform complex bronchoscopic procedures that were previously impossible. Unfortunately, these systems generally lack the ability to make necessary intraoperative adjustments to the surgical plan. Because these critical situations arise so frequently, providing the ability to make these adjustments is crucial.

[0159] The present invention provides, for the first time, a method for deriving an airway path during real-time bronchoscopy that can lead to any selected site visible on the guidance system's display. Given this airway path, the physician can use the assisted bronchoscopy system to navigate the bronchoscope to a new site. This allows the physician to respond to unexpected findings observed during live surgery and alter the surgical plan in real time.

[0160] The airway locator method disclosed in the present invention has been shown to provide such a means for creating a path to a newly observed clinically interesting site in real time. Given this path, the assisted bronchoscopy system can guide the physician to the new site.

[0161] On the other hand, in other areas of endoscopic surgery (laparoscopy, colonoscopy, cystoscopy), when doctors need to navigate equipment through hollow organ systems, these surgeries also encounter unexpected situations and require doctors to be able to adjust their surgical plans "on the fly." Our method is also applicable to these other fields.

[0162] As is known to those skilled in the art, the embodiments of the present invention described and discussed herein may be modified in various ways without departing from the scope or teachings of the present invention. In addition, elements and aspects of one embodiment may be combined with elements and aspects of another embodiment. The scope of the present invention is defined by the following claims (including all equivalent claims). References [1] Fumihiro Asano, Naofumi Shinagawa, Takashi Ishida, et al. Virtual bronchoscopic navigation improves the diagnostic accuracy of radial endobronchial ultrasound for peripheral lung lesions with bronchial involvement shown on CT. Journal of Internal Medicine, 54(9): 1021–1025, 2015. [2] Patrick D Byrnes and William Evan Higgins. Efficient bronchoscopic video summarization technology. IEEE Transactions on Biomedical Engineering, 66(3):848–863, March 2019. [3] P. Byrnes and WE Higgins. Endobronchial video analysis system. In RJ Webster and B. Fei (eds.), SPIE Medical Imaging 2017: Image-guided Surgery, Robotic Interventions and Modeling, vol. 10135, pp. 101351Q–1–101351Q–9, 2017. [4] Q. Chang, D. Ahmad, J. Toth, R. Bascom, and W.E. Higgins. ESFPNet: An efficient deep learning architecture for real-time lesion segmentation in autofluorescence bronchoscopy videos. In B.S. Gimi and A. Krol (eds.), SPIE Medical Imaging 2023: Biomedical Applications of Molecular, Structural, and Functional Imaging, vol. 12468, 2023. [5] Q. Chang, P. Byrnes, D. Ahmad, J. Toth, R. Bascom, and W.E. Higgins. Application of bronchoscopic video synchronization technology in multimodal interactive examination of bronchial lesions. In CA Linte and J.H. Siewerdsen (eds.), SPIE Medical Imaging 2021: Image-Guided Surgery, Robotic Intervention and Modeling, vol. 11598, pp. 115981V–1–115981V–11, 2021. [6] R. Cheirsilp, R. Bascom, TW Allen, RPM Mahraj, and WE Higgins. Deformable image registration for multimodal lung cancer staging. In M. Styner and E. Angelini (eds.), SPIE Medical Imaging 2016: Image Processing, vol. 9784, pp. 97843Z–1–97843Z–10, 2016. [7] R. Cheirsilp, R. Bascom, TW Allen, RPM Mahraj, and WE Higgins. Deformable image registration for multimodal lung cancer staging. In M. Styner and E. Angelini (eds.), SPIE Medical Imaging 2016: Image Processing, volume 9784, pages 9784Z–1–9784Z–10, 2016. [8] R. Cheirsilp, Bascom, T. W. Allen, and W. E. Higgins. Thoracic cavity definition methods for 3D PET / CT analysis and visualization. Computational Biology and Medicine, 62: 222–238, July 2015. [9] R. Cheirsilp and WE Higgins. A multimodal 3D PET / CT system for bronchoscopic surgery planning. In CL Novak and S. Aylward (eds.), SPIE Medical Imaging 2013: Computer-Assisted Diagnosis, vol. 8670, pp. 86702X–1–86702X–14, February 2013.

[10] Gerard J Criner, Ralf Eberhardt, Sebastian Fernandez-Bussy, Momen MWahidi, et al. A review of the current status of interventional bronchoscopy. American Journal of Respiratory and Critical Care Medicine, (1): 29–50, July 2020.

[11] V. Daneshpajooh, D. Ahmad, J. Toth, R. Bascom, and W.E. Higgins. Video analysis framework for narrow-band bronchoscopic lesion detection. In B.S. Gimi and A. Krol (eds.), SPIE Medical Imaging 2022: Biomedical Applications of Molecular, Structural, and Functional Imaging, vol. 12036, pp. 120360M–1–120360M–9, 2022.

[12] JD Gibbs, MW Graham, R. Bascom, DC Cornish, R. Khare, and WE Higgins. Optimal surgical planning and navigation system for peripheral bronchoscopy. IEEE Transactions on Biomedical Engineering, 61(3):638–657, March 2014.

[13] JD Gibbs, MW Graham, and WE Higgins. Peripheral bronchoscopy planning system based on 3D MDCT. Computational Biology and Medicine, 39(3):266–279, March 2009.

[14] MW Graham, JD Gibbs, DC Cornish, and WE Higgins. Robust 3D airway tree segmentation for image-guided peripheral bronchoscopy. IEEE Transactions on Medical Imaging, 29(4):982–997, April 2010.

[15] W. R. Grier, M. Saleh, E. Pickering, A. Sachdeva, V. K. Holden, and F. Nasim. Evaluation of 3D fluoroscopy-assisted robotic bronchoscopy for biopsy of peripheral pulmonary nodules. American Journal of Respiratory and Critical Care Medicine, 205: A3671, May 2022.

[16] WE Higgins, R. Cheirsilp, X. Zang, and P. Byrnes. A multimodal system for bronchoscopic planning and navigation. In Z. Yaniv and R. Webster (eds.), SPIE Medical Imaging 2015: Image-Guided Surgery, Robotic Interventions, and Modeling, vol. 9415, pp. 941508–1–941508–9, 2015.

[17] WE Higgins, JP Helferty, K. Lu, S. A. Merritt, L. Rai, and K. C. Yu. Three-dimensional CT-video fusion technology for image-guided bronchoscopy. Computer Medical Imaging and Graphics, 32(3):159–173, April 2008.

[18] D. Himeji, M. Oki, H. Saka, T. Imabayashi, Y. Nishii, and M. Ando. Multicenter prospective study of adding ultrathin bronchoscope to standard thin bronchoscope for the evaluation of peripheral lung lesions. American Journal of Respiratory and Critical Care Medicine, 205:A3673, May 2022.

[19] E. Ho, R. J. Cho, J. Keenan, and S. Murgu. Feasibility of using the “vascular sign” for preoperative planning in navigational bronchoscopy for sampling of peripheral pulmonary lesions: a two-center retrospective study. American Journal of Respiratory and Critical Care Medicine, 205: A3670, May 2022.

[20] T. Inage, T. Nakajima, I. Yoshino, and K. Yasufuku. Early detection of lung cancer. Clinical Chest Medicine, 39:45–55, 2018.

[21] T. Inoue, T. Kawamura, K. Kunimasa, M. Tamiya, H. Kuhara, K. Nishino, S. Odani, F. Imamura, T. Kumagai, and K. Miyake. Manual path correction using the oblique method after automated virtual bronchoscopy navigation. Medicine, 101(17): 1–7, 2022.

[22] R. Khare, R. Bascom, and WE Higgins. Hands-free operation system for bronchoscopic planning and navigation. IEEE Transactions on Biomedical Engineering, 62(12):2794–2811, 2015.

[23] T. Kuhlengel, R. Bascom, and W. Higgins. Efficient surgical planning for comprehensive lymph node staging bronchoscopy. Journal of Medical Imaging, 9(5): 055001–1—055001–22, September 2022.

[24] K. Lu and WE Higgins. Interactive segmentation method of 3D CT chest images based on dynamic contours. International Journal of Computer-Assisted Radiosurgery, 2(3-4):151–167, December 2007.

[25] S. A. Merritt, R. Khare, R. Bascom, and W. E. Higgins. Interactive CT-video registration for image-guided bronchoscopy. IEEE Transactions on Medical Imaging, 32(8):1376–1396, August 2013.

[26] T. Nakajima and K. Yasufuku. Early detection of lung cancer. In Jose Pablo Diaz-Jimenez and Alicia Rodriguez (eds.), Interventional Techniques in Pulmonary Medicine, pp. 245–256. Springer, 2018.

[27] M. Oki, H. Saka, Y. Kogure, H. Niwa, A. Yamada, A. Torii, and C. Kitagawa. Application of ultrathin bronchoscopic cryobiopsy in peripheral lung lesions. American Journal of Respiratory and Critical Care Medicine, 205: A3674, May 2022.

[28] PJ Reynisson, H.O. Leira, T.N. Hernes, E.F. Hofstad, et al. A review of navigational bronchoscopy. Journal of Bronchology and Interventional Pulmonology, 21(3):242–264, July 2014.

[29] JZ Turlington and WE Higgins. A new and efficient technique for sliding thin-slice volume visualization. IEEE Transactions on Medical Imaging, 20(8):823–835, August 2001.

[30] MM Wahidi, FJF Herth, A. Chen, G. Cheng, and L. Yarmus. Current status of interventional pulmonology techniques. Journal of Thoracic Surgery, 157(3):724–736, March 2020.

[31] Juan P Wisnivesky, Rex Chin-Wei Yung, Praveen N Mathur, and Javier J Zulueta. Diagnosis and treatment of central airway bronchial intraepithelial neoplasia and early lung cancer: Diagnosis and management of lung cancer—American College of Chest Physicians evidence-based clinical practice guideline. Chest Journal, 143(5_suppl):e263S–e277S, May 2013.

[32] KC Yu, E L Ritman, and WE Higgins. Vascular system analysis method based on three-dimensional model and differential geometry. IEEE International Symposium on Biomedical Imaging, pp. 177–180, April 15–18, 2004.

[33] X. Zang, R. Cheirsilp, P.D. Byrnes, W. Higgins, et al. Image-guided EBUS bronchoscopy system for lung cancer staging. Medical Informatics (Open Access), 25(100665): 1–13, 2021.

[34] X. Zang, JD Gibbs, R. Cheirsilp, P. D. Byrnes, J. Toth, R. Bascom, and W. E. Higgins. Optimal path planning method for image-guided EBUS bronchoscopy. Computational Biology and Medicine, 112(7):103361, September 2019.

Claims

1. A method for automatically deriving a new navigational airway path temporarily through a patient's cavitary organ system, wherein: The new navigation airway path leads to a previously unplanned region of interest (ROI) observed by a physician performing a real-time endoscopic procedure, and the method comprises the following steps: loading information about the hollow organ system previously calculated based on radiographic data of the patient, including an airway tree, an airway lumen surface, and an airway centerline; Based on the previously calculated information, an initial surgical plan calculated offline before the real-time endoscopic surgery is performed is provided, wherein the initial surgical plan clearly defines an initial ROI different from the unplanned ROI and derives an appropriate airway path leading to the initial ROI; Update the initial surgical plan by: marking a new portion s at the unplanned ROI as a new inspection target on one of the graphic visualization tools of the image-guided endoscopy system; automatically calculating a new navigation airway path leading to the new examination target; and The endoscope is guided along the new navigation airway path through the hollow organ system to the new examination target.

2. The method according to claim 1, wherein The airway centerline comprises a set of directional paths, each path comprising a set of observation locations that are visited only once along the path, the path originating from the trachea and terminating at a distal airway branch, each observation location comprising a vector and a camera imaging orientation when visualized along the path, the vector specifying coordinates and an orientation angle; The steps of automatically calculating the new navigation airway path include: Find the observed part v closest to the new part s c ; Select the directional path set containing the observation part v c The path p i ; After deriving the compensation amount of ∈ observation parts, calculate the new observation part v s =v c-∈ , as the initial airway path p to the new site s s The initial and final observation sites, the new observation site v s In the selected path p i The center is located at v c Before ∈ observation site, and the preliminary airway path p s Satisfy the constraints; Calculate the new position s starting from v s And the vector z=sv pointing to the part s s , thereby obtaining the preliminary airway path p s The final observation position v f ;as well as The initial airway path p s The final observation position v in f Replace v s , thereby providing the new navigation airway path to the new site s.

3. The method according to claim 1 or 2, wherein: The radiological imaging data is a chest CT scan, a PET scan or a magnetic resonance imaging scan.

4. The method according to any one of claims 1 to 3, wherein The new location s is identified on a 2D CT slice, a fused 2D CT / PET slice, a sliding thin-section view, a stenosis location annotated in a VB renderer, a location along the airway wall in a 2D MRI slice, or a video stream view.

5. The method according to any one of claims 1 to 4, wherein The new site s is identified before the start of or during the real-time endoscopic procedure.

6. The method of any one of claims 1 to 5, further comprising guiding progression along the new navigated airway path using a second auxiliary device.

7. The method according to claim 6, wherein: The second auxiliary device is selected from an optical coherence tomography probe, a cryotherapy probe, or a NaG laser diagnostic or therapeutic probe.

8. The method of any one of claims 1 to 7, further comprising previewing the new navigational airway path using the graphical visualization tool of the image-guided endoscopy system before guiding the endoscope along the new navigational airway path.

9. The method according to any one of claims 1 to 8, further comprising saving any newly created ROI and associated airway pathway as part of the overall case study for future reference.

10. The method according to any one of claims 1 to 9, further comprising calculating the necessary distance to travel from one observation location to the new location.

11. The method according to claim 2, wherein: The constraints required include airway size and endoscope tip size.

12. The method according to any one of claims 1 to 10, wherein The endoscope is a bronchoscope.

13. A system for generating a real-time guidance path through a patient's cavitary organ system to a previously unplanned diagnostic site during real-time endoscopic surgery, comprising: Image-guided endoscopy system, including: an endoscope operable to navigate within said hollow organ system; a display device operative to display a plurality of graphic visualization tools, the plurality of graphic visualization tools displaying 2D CT / PET slices, CT-based virtual bronchoscopy (VB) renderings, global 3D airway tree renderings, or a real-time video stream derived from an endoscope; a memory for storing surgical plans and updates thereof; A processor is configured to communicate with the memory and the display device, the processor being operative to perform the following steps: Loading information pre-calculated before the real-time endoscopic surgery and an airway path determined offline, wherein the pre-calculated information includes an airway tree, an airway lumen surface, and an airway centerline based on patient radiographic data; and Automatically calculate navigational airway paths to newly observed diagnostic sites during real-time endoscopic procedures.

14. The system according to claim 13, wherein: The airway centerline comprises a set of directional paths, each path comprising a set of observation locations that are visited only once along the path, the path originating from the trachea and terminating at a distal airway branch, each observation location comprising a vector and a camera imaging orientation when visualized along the path, the vector specifying coordinates and an orientation angle; The processor is further operable to perform the following steps: Find the observed part v that is closest to the new part s c ; Select the path containing the observation part v in the set of directional paths c The path p i ; After deriving the compensation amount of ∈ observation parts, calculate the new observation part v s =v c-∈ , as the initial airway path p to the new site s s The initial and final observation sites, the new observation site v s In the selected path p i The center is located at v c Before ∈ observation site, and the initial airway path p s Satisfy the constraints; Calculate the new position s starting from v s And the vector z=sv pointing to the part s s , thereby obtaining the preliminary airway path p s The final observation position v f ; as well as The initial airway path p s The final observation position v in f Replace v s , thereby providing a new navigation airway path to the new site s.

15. The system according to claim 13 or 14, wherein: The new location s is identified on a 2D CT slice, a fused 2D CT / PET slice, a sliding thin-section view, a stenosis location observed in a VB renderer, a location along the airway wall in a 2D MRI slice, or a video stream view.

16. The system according to any one of claims 13 to 15, wherein: The new site s is identified before the start of or during the real-time endoscopic procedure.

17. The system according to any one of claims 13 to 16, wherein: The processor is further operable to perform the step of calculating the necessary distance to travel from one observation location to a new location.

18. The system of claim 14, wherein: The constraints required include airway size and endoscope tip size.

Images