Multi-destination surgical planning and guidance for endoscopic surgery

Endoscopy route generated by 3D imaging scanning and ant colony optimization algorithm solves the planning and execution of complex multi-destination surgical procedures, and achieves efficient and accurate examination and treatment of multiple diagnostic sites, reducing the death of lung cancer Rate.

CN118984668APending Publication Date: 2024-11-19THE PENN STATE RES FOUND INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380030535.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-07-07
Filing Date
2023-01-25
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The existing technology is difficult to effectively plan and perform complex multi-destination surgery, especially in surgical procedures that require traversing complex hollow tubular anatomical areas, resulting in insufficient diagnosis and treatment, especially the difficulty of early detection and staging of lung cancer, resulting in high mortality rate.

Method used

By using 3D radiation imaging scans to generate virtual space, calculate the shortest route and the best patrol path, combine the ant colony optimization algorithm to generate detailed endoscopic routes, provide image guidance and visual prompts, and realize coordinated patrols of multiple diagnostic sites and guide.

Benefits of technology

Improve the planning and execution efficiency of surgical procedures, ensure complete examination and treatment of multiple diagnostic sites, reduce doctors' skills dependence, improve diagnosis accuracy and comprehensive treatment, especially in stages and staged surgery for lung cancer significantly improved the survival rate of patients.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984668A_ABST
    Figure CN118984668A_ABST
Patent Text Reader

Abstract

A touring planning method for accessing a set of target diagnostic sites through a hollow organ system by endoscopy includes: calculating a 3D virtual space of the hollow organ system based on a 3D imaging scan depicting the diagnostic sites and the hollow organ system; calculating a series of preliminary endoscopic routes to each diagnostic site, including a set of consecutive device gestures through the hollow organ system, the gestures connecting each target diagnostic site to all other target diagnostic sites; and calculating an optimal tour with a minimum cost, the optimal tour corresponding to a minimum set of contiguous gestures accessing all target diagnostic sites once and only once, where the cost between each pair of diagnostic sites is defined as the distance accumulated by the minimum set of contiguous gestures connecting pairs of diagnostic sites together within the hollow organ system.
Need to check novelty before this filing date? Find Prior Art

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 303,712 filed on January 27, 2022 and U.S. Provisional Patent Application Serial No. 63 / 358,935 filed on July 7, 2022, both of which are incorporated herein by reference in their entirety. Government Sponsorship

[0002] This invention was made with government support under Grant No. CA151433 awarded by the National Institutes of Health. The government has certain rights in this invention. Technical Field

[0003] The present invention relates generally to surgical planning and guidance, and more particularly to a multi-destination route planning method and guidance method suitable for guiding endoscopy and other fields. Background Art

[0004] Many surgical procedures, particularly those requiring the use of an endoscope to traverse hollow tubular anatomical structures (e.g., the airway) or hollow regions (e.g., the stomach), require comprehensive inspection, biopsy, diagnosis, or treatment of multiple locations within the anatomical region. Examples of such applications are discussed in the literature [17, 20, 23, 27, 37, 55, 56, 61, 63] and include:

[0005] 1. Staging of intrathoracic lymph nodes.

[0006] 2. Check the airways across the lungs to treat asthma, cancer and other diseases.

[0007] 3. Repeat bilateral pulmonary airway examinations to monitor disease and follow up on examinations performed at different times (e.g., to monitor for lung tumor development or transplant rejection).

[0008] 4. Guide multiple mucosal (airway wall) lesions in the chest cavity. Such lesions may be early indicators (biomarkers) of cancer formation or origin.

[0009] 5. Examine the vascular system within an organ to determine the location of narrowed blood vessels. Organs such as the heart, brain, liver, and kidneys have extensive networks of blood vessels. For example, diseased (narrowed) blood vessels may indicate a high risk of heart attack or stroke.

[0010] 6. Examine and biopsy multiple suspected cancerous polyps in the colon.

[0011] 7. Use laparoscopy to examine and treat multiple diagnostic sites in the abdomen, stomach, or pelvis.

[0012] Such comprehensive surgeries—that is, surgeries that require a large number of surgical sites (“destinations”) to be completed satisfactorily—are currently never performed because they are too complex or impractical. Two problems contribute to this situation. First, physicians’ skills vary, resulting in a wide range of performance for various surgeries. Second, the tools required to effectively plan and complete such surgeries do not exist. Furthermore, while it may always be quite difficult for humans to perform such surgeries even in the best of circumstances, suitable automated robots may be able to perform such surgeries more efficiently without relying on the physician’s skills. Unfortunately, even for robotic applications, methods currently do not exist to plan and guide such complex surgeries.

[0013] In one specific area, lung cancer is responsible for more than 20% of all cancer-related deaths in the United States.

[62] For patients at risk for lung cancer, doctors diagnose and stage the disease in two steps.[56, 63] First, doctors plan surgery offline by identifying suspected metastatic lung tumors and any suspicious chest lymph nodes based on the patient's chest computed tomography (CT) imaging scans and, if available, positron emission tomography (PET) scans. Next, during surgery, doctors use the imaging observations to guide a bronchoscope through the lung airways to biopsy the tumor and selected lymph nodes.

[20] Tumor biopsy helps confirm the diagnosis, while lymph node biopsy can determine the stage or spread of the disease.

[0014] Of note, accurate disease staging is critical because it determines subsequent treatment options and disease prognosis [56, 63]. For example, the five-year survival rate for patients with stage 1A1 disease is 92%, while the five-year survival rate for patients with stage IIIB disease is only 26% [8]. Therefore, incomplete or uncertain lymph node staging surgery can lead to inadequate treatment planning and have a devastating impact on patient survival. Unfortunately, physicians are often unable to plan and perform lymph node staging surgeries that are complete enough to ensure accurate disease staging [49, 70].

[0015] Unfortunately, high mortality rates from lung cancer persist because a large proportion of patients are often diagnosed with advanced stage 3–4 disease

[19] . To address this problem, efforts are currently underway to expand CT-based lung cancer screening programs worldwide, which are driving a major shift toward detecting early, treatable disease [52, 65]. However, this exciting development has greatly increased the need for accurate staging of thoracic lymph nodes [56, 63]. Unfortunately, Miller et al. found that many physicians do not perform appropriate lymph node staging

[49] .

[0016] As a standard aid in determining the stage of disease, the International Association for the Study of Lung Cancer (IASLC) has developed the TNM lung cancer staging system [8, 26, 60]. Physicians use the TNM system to derive “TxNyMz” disease categories to determine disease stage, where the T, N, and M descriptors refer to the primary tumor, lymph node involvement, and distant metastases outside the chest cavity, respectively. The “x,” “y,” and “z” subdescriptors help stratify the spread of disease within the three anatomical regions. The TNM system divides the chest cavity into a set of distinct lymph node sites, as shown in the IASLC lymph node map [8, 15]. Lymph node sites are sequentially assigned to staging zones N1 to N3, with the detection of positive metastatic lymph nodes in higher-numbered zones implying more advanced disease stages [56, 60].

[0017] Current clinical guidelines recommend complete sampling of at least 6 visible lymph nodes (considering ≥6 sites) to accurately determine the disease stage [9, 20, 34, 63]. This recommended staging strategy includes first examining the lymph nodes in the highest level N3 lymph node site, then the N2 level lymph nodes, and finally the N1 level lymph nodes, with positive lymph nodes in the highest stage area being assigned the descriptor “Ny” [26, 34].

[0018] Unfortunately, studies have shown that most staging surgeries are inadequate, with physicians often employing a so-called “selective sampling” approach, sampling only a few prominent lymph nodes or even just one enlarged lymph node [9, 20, 34, 38, 49]. (Zhong et al. speculate that ≥11–16 lymph nodes must be examined to yield a definitive staging assessment

[70] ). To date, physicians participating in the large multicenter staging study by Ost et al. have biopsied an average of only 2.18 lymph nodes per patient, with 66% of patients sampling only 1–2 lymph node sites

[54] . As a result, lung cancer mortality remains high, as a large proportion of patients are often diagnosed at stage 3–4

[19] .

[0019] Limitations in physician skills are a major reason for this. Given these limitations, physicians must first review the patient’s three-dimensional (3D) CT imaging scan to plan the lung airway routes to each identified diagnostic site and decide on the order of lymph node sampling [10, 51]. Next, in the operating room, the physician must mentally translate the developed plan into the live airway observations seen during bronchoscopy. Unfortunately, physicians vary widely in their ability to construct airway routes based on CT observations and their skill in performing bronchoscopy [10, 47, 50]. Although physicians do receive bronchoscopy training, these plans are dictated by how to perform the minimum number of procedures rather than acquiring the necessary technical skills [16, 20, 21, 34].

[0020] In addition, researchers have found that if physicians must perform complete nodal staging, they face two additional burdens [9]. First, they must navigate the bronchoscope to many distant lymph node sites within the chest. Second, they must have the skills to mentally translate the IASLC lymph node map from 3D CT observations to live bronchoscopy. Ideal surgical planning includes an efficient “tour” of the airways to all designated lymph nodes, with the tour order consistent with clinical staging guidelines.

[0021] The task of deriving such an efficient tour is an example of the Traveling Salesman Problem

[31] . The Traveling Salesman Problem is considered NP-hard; that is, the best algorithm that guarantees an optimal solution using the graph search above has exponential time worst-case complexity [22, 31]. Since this is not feasible in practice, we need a method that can quickly find a near-optimal solution. Methods such as nearest neighbor search

[59] , local graph search heuristics [7, 40], greedy search [3], and genetic and evolutionary algorithms [39, 66] have been applied to this problem. Their success varies, and some methods are susceptible to local minima or poor quality results.

[0022] Although common in industrial applications, the traveling salesman problem has limited applications in biomedicine, such as studying organisms

[58] and selecting surgical tools

[53] . For surgical procedures, Goyette and Riviere applied linear optimization to minimize the travel time of a surgical robot to reach various points on the pericardial surface of the heart

[27] . Madeira et al. optimized a finite element model to correct scoliosis

[45] . Falcone et al. simulated a task related to laparoscopic surgery

[17] . Burrows et al. simulated needle steering to allow biopsies to be taken at multiple locations through a single incision [4]. Relatedly, Gayle et al. rigorously formulated a path planning task to guide a flexible robot through the complex vasculature of the liver, but did not consider the need to visit multiple destinations

[23] . Notably, these studies generally did not use real data and did not have direct applications to image-guided surgery or bronchoscopy.

[0023] On the other hand, for the relevant lung cancer management task of peripheral tumor biopsy, recently introduced image-guided bronchoscopic systems have greatly alleviated some of the above-mentioned bronchoscopic skill issues [5, 6]. Specifically, they use computer-based image analysis to derive the optimal airway route to the target tumor and provide live image guidance along the derived route during bronchoscopy [2, 24, 57]. As a result, these systems have been shown to nearly eliminate the skill differences among physicians in planning and performing bronchoscopy for peripheral tumor biopsies, while also improving biopsy success rates [1, 24, 47].

[0024] Unfortunately, these systems only provide individual airway routing for each diagnostic site, without providing coordinated guidance for multiple sites. So far, they are unable to provide accurate and comprehensive lymph node staging as recommended by clinical guidelines, such as: 1) a coordinated tour of all required lymph nodes and lymphatic stations; 2) conversion of IASLC lymph node maps from CT to surgical planning, which enables the ordering of lymph nodes by staging zone.

[0025] In summary, comprehensive endoscopic / surgical procedures that require traversal of complex hollow tubular anatomical regions or hollow areas in general are not currently performed because they are too complex or impractical, and many important medical tasks require these procedures. Although surgical plans can be created for a single diagnostic site, there is no way to effectively plan comprehensive surgeries involving many distributed diagnostic sites ("destinations"). In addition, there is no way to use the created multi-destination surgical plan ("round trip") for subsequent on-site surgical guidance. Summary of the invention

[0026] Embodiments of the present invention provide a method for planning and guiding complex multi-destination surgical procedures that require the use of an endoscope operated by an operator (e.g., a physician) through a complex, branched, hollow, tubular organ system or other hollow organ (e.g., abdomen, stomach, etc.). Specifically, such procedures require the physician to examine, diagnose, biopsy, or treat multiple distributed locations of a target region of interest within an anatomical region.

[0027] First, a 3D radiographic scan of the patient's hollow organ system will be provided as the primary input. The region of interest will be identified and a 3D virtual space of the hollow organ system can be derived from the 3D preoperative imaging scan. The 3D imaging scan can be a CT scan or an MRI scan.

[0028] The hollow organ system may be the vasculature in the lung airways, heart, brain, liver, kidneys, or hollow regions in the colon, stomach, bladder, or pelvis / abdomen. The endoscope may be a bronchoscope, colonoscope, laparoscope, angioscope, or cystoscope. The target diagnostic site may be a lymph node, a suspicious lesion, or a therapeutic delivery site.

[0029] For example, the method utilizes a 3D imaging scan of the patient to derive a patient-specific multi-destination tour to efficiently navigate to all preselected lymph nodes. The route may include a set of endoscopic device postures that span at least a portion of a hollow organ system between a pair of lymph nodes or target diagnostic sites. Each posture consists of its coordinates in a 3D virtual space and a set of vectors indicating a direction of travel through the hollow organ system. In addition, each lymph node or target diagnostic site is connected to each other lymph node by a minimum continuous subset of postures, representing a minimum distance route between lymph node pairs. The posture connects all lymph nodes or target diagnostic sites together. A complete set of continuous postures that visit all lymph nodes or target diagnostic sites constitutes a tour. A tour constitutes a route that connects all preselected lymph nodes or diagnostic sites together, so that each site is visited only once. The optimal tour is defined as the shortest route corresponding to the minimum cost of visiting all lymph nodes or target diagnostic sites once and only once, where the cost represents the distance traveled by the endoscope to connect the continuous subset of postures between each pair of lymph nodes or diagnostic sites. The minimum cost is defined as the distance traveled by the smallest continuous subset of poses connecting each pair of lymph nodes or diagnostic sites. The optimal tour corresponds to the minimum cost tour, i.e., the smallest continuous set of poses connecting all preselected lymph nodes or diagnostic sites together.

[0030] If each diagnostic site is considered as a vertex and the distance connecting two vertices is considered as an edge, a graph structure can be constructed. The graph structure includes a cost matrix, where each cost entry represents the travel distance from one target diagnostic site to another target diagnostic site. The optimal tour is the minimum cost sequence of vertices that visits each vertex once and only once.

[0031] If the target diagnostic site is a lymph node, the lymph nodes belong to certain (IASLC) anatomical lymph node sites and adhere to IASLC guidelines. Given the known location of the cancer nodules, the labeled nodes are mapped to staging zones. The method further incorporates the constraints imposed by the staging zones when calculating the shortest route. The method determines the optimal or near-optimal tour that visits all target lymph nodes, consistent with the recommended guidelines for comprehensive lymph node staging.

[0032] To calculate the desired route, the method consists of three main steps:

[0033] 1. Imaging scan preprocessing: Define 3D models of target diagnostic sites and hollow organ systems (e.g., lymph nodes, tumors, airway tree structures, preliminary single-node airway alignments, and IASLC lymph node sites).

[0034] 2. Graph and staging area construction: Setting up the data structures required for route calculation.

[0035] 3. Route calculation, calculate the shortest route for subsequent image-guided endoscopic examination.

[0036] Surgical planning can be developed based on the shortest route to guide subsequent on-site surgery.

[0037] Next, image-based approaches use surgical planning to guide comprehensive, multi-destination medical procedures.

[0038] A range of graphical tools are provided, including visual cues and guided instructions, such as linking planning data obtained in 3D virtual space to the real space of live medical surgery by linking a virtual endoscopic intracavitary view obtained in 3D virtual space with a live video endoscopic view of a hollow organ system in the real space of live medical surgery.

[0039] The guidance control strategy can effectively guide the doctor or surgical robot to each surgical site. For example, it can provide recommended device operations to reach each diagnostic site.

[0040] During a guided procedure, the procedure history keeps track of all guidance and imaging information for sites visited. For each site, the pathway history records the time the site was visited, the status of the guidance system, and the imaging characteristics of the site. The procedure history records which sites have been visited and which have not been visited during a live medical procedure. The procedure history can be called up during a subsequent procedure to evaluate treatment or disease progression at a previously examined site.

[0041] Both planning and guidance methods have been implemented as interactive software systems. The planning system accepts appropriate radiographic data and outputs a surgical plan consisting of a tour and other information to optimally reach each surgical site. The guidance system accepts the surgical plan as input and interacts with the surgical endoscope hardware to accept live endoscopic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of airway distance calculation is shown; (the routes r1 and r2 of lymph nodes n1 and n2 share positions p1 to p2 in the trachea, respectively) LCA Two routes in pose p a and p b to different paths);

[0043] Figure 2A A schematic 6-node example showing the graph structure and optimal tour; (CT scan of human case 21405-3a was used. Figure 2A A graph structure is shown, with nodes represented as numbered circles and edges as red lines;

[0044] Figure 2BA CT-based rendering of the airway tree is shown, depicting the physical location of the nodes and the optimal patrol, shown as a series of directed black arrows; (the patrol is also depicted on the diagram as a series of black arrows);

[0045] Figure 3A A complete example of human case 20349-33 is shown; ( Figure 3A A CT-based airway tree is shown, depicting the derived tour);

[0046] Figure 3B A 3D rendering of a selected IASLC node site is shown;

[0047] Figure 3C An example of a 2D coronal CT slice is shown, showing color-coded staging zones; (for clarity, not all tissues making up the zones are colored);

[0048] Figure 3D The graph showing the best patrol (black arrow);

[0049] Figure 4 It shows the top-level layout and workflow of the software system;

[0050] Figure 5 A data flow diagram of a two-stage multi-destination lung cancer staging process is shown; (the upper row represents the virtual space derived from the patient's preoperative PET / CT data, while the lower row represents the real space, including bronchoscopic video and possible endobronchial ultrasound (EBUS) generated during surgery. During surgical planning, the system creates a tour of designated lymph nodes and desired diagnostic sites. During live bronchoscopy, the guidance system provides image-based guidance for comprehensive lymph node staging);

[0051] Figure 6 A graphical user interface (GUI) for the surgical planning subsystem is shown; (the left image shows a loaded patient study with a CT-based chest model component loaded. It also provides a method for specifying the primary tumor (top right) and lists the operations required to perform the 3-step planning process (bottom). The right image depicts how a tour through a specified lymph node is calculated);

[0052] Fig. 7A The LNS Planner tool is shown for visualizing the generated lymph node tour; ( Fig. 7A shows a dynamic visualization through the airway tree, which can be played as an interactive movie);

[0053] Figure 7B It shows that doctors can play intracavitary movies through the airway tree to view the access to nodes along the way;

[0054] Figure 8 A GUI of an image-guided system for guiding a physician through a multi-destination anatomical ROI tour through an airway tree is shown;

[0055] Fig. 9 For along Figure 8 Some guidance tools are displayed when the tour shown reaches the destination of node 1;

[0056] FIG10 shows a view of the image-guided bronchoscopy system after the physician reaches the 12L lymph node site of the left lower lobe bronchus of patient 20349-3a; (the 3D airway tree view and the site progress map indicate the physician's position in the chest cavity with red crosshairs and open disks, respectively. The CT video tool displays the live video bronchoscopy frame [left image] and the corresponding VB view [right image]; the green area of ​​the VB view indicates the 12L node);

[0057] Fig.11 A sample guided system view of patient 20349-13 is shown, who has 38 lymph nodes covering the range of IASLC lymph node stations;

[0058] Figures 12A-12I An example of an automatic node site view showing an automatic view site is shown; ( Figures 12A-12C Use VB view to display site M10L. Figures 12D-12F Use VB view to display site M10R. Figure 12G-12I Use VB view to display site M7. Fig. 12A , 12D , 12G shows the endoluminal rendering of the airway lumen using the VB camera at the canonical site view position. Fig. 12B , 12E , 12H shows an external view of the airway 3D surface, showing the soft tissue surface of the lymph node site within the site cuboid (red) and the airway guidance route to that location (blue). Fig. 12C , 12F 12I shows a semi-transparent intraluminal rendering that uses the same camera pose as the left column to show through the airway wall, revealing the surface of the soft tissue volume from inside the airway. (Data from case 21405-67));

[0059] Fig.13 A guidance system tool for generating sample canonical site and canonical posture views of the airway tree is shown; and

[0060] Fig.14 The surgical history tool in the guidance system is shown. DETAILED DESCRIPTION

[0061] The present invention provides a method and system for planning and guiding complex multi-destination surgical procedures that require navigating a hollow organ system using an endoscope operated by an operator (e.g., a doctor or a surgical robot). Specifically, such procedures require a doctor to inspect, diagnose, biopsy, or treat multiple destination sites in a target area of ​​interest.

[0062] A 3D imaging scan of the patient's hollow organ system will be provided and used as the primary input. Regions of interest will be identified and a 3D virtual space of the hollow organ system can be derived from the 3D preoperative imaging scan.

[0063] The method utilizes a 3D imaging scan of a patient to derive a patient-specific multi-destination route for efficient navigation to all predetermined lymph nodes. The route may include a set of endoscopic device poses that span at least a portion of a hollow organ system between a pair of lymph nodes or target diagnostic sites. Each pose consists of its coordinates in 3D virtual space and a set of vectors indicating a direction of travel through the hollow organ system.

[0064] The complete set of consecutive poses that visit all lymph nodes or target diagnostic sites constitutes a tour. The optimal tour is defined as the shortest route corresponding to the minimum cost that visits all lymph nodes or target diagnostic sites once and only once, where the cost represents the distance traveled by the endoscope to connect each pair of lymph nodes or diagnostic sites with a subset of consecutive poses. The minimum cost is defined as the distance traveled by the minimum set of consecutive poses that connect each pair of lymph nodes or diagnostic sites. The optimal tour corresponds to the minimum cost tour, that is, the minimum set of consecutive poses that connects all pre-selected lymph nodes or diagnostic sites together.

[0065] Surgical planning can be developed based on the shortest rounds to guide subsequent on-site surgery.

[0066] Next, image-based approaches use surgical planning to guide comprehensive, multi-destination medical procedures.

[0067] Returning to the lung cancer problem, the ideal comprehensive lymph node staging plan or tour is a continuous path through the airways, passing through a specified set of lymph nodes, and visiting them in an order that conforms to clinical staging guidelines. For colon cancer diagnosis, the plan would be a continuous path through the colon, visiting various suspected cancerous polyps. To find an appropriate tour, the problem is transformed into a form of the Traveling Salesman Problem, which can be stated as follows

[31] . A salesman wishes to visit N interconnected cities in his sales territory once and only once. Because travel is expensive, the salesman wishes to choose the shortest tour connecting these cities.

[0068] For lymph node staging, regions correspond to lung airways and cities correspond to lymph nodes: n i , i = 1, 2, ..., N. (1)

[0069] In general, for N ≥ 3, we have Therefore, for N = 3 lymph nodes, there are only 3 possible patrols: n1→n2→n3, n1→n3→n2 and n3→n1→n2

[0070] (Note that the tour n3→n2→n1 is equivalent to n1→n2→n3, and so on.) Unfortunately, as N increases, a brute force search of all possible tours becomes impractical. For example, for a patient with N=23 identified lymph nodes, the astronomical 1.293×10 22 A tour is possible.

[0071] To find the shortest tour, the problem is formulated as an Optimal Graph Search, where the search space is defined by a directed graph consisting of:

[0072] A set of vertices V = {1, 2, ..., N} (2)

[0073] Each vertex i = 1, 2, ..., N represents a lymph node n in the set (1). i .

[0074] A set of edges E = {(i, j), i, j∈V} (3)

[0075] Connect each pair of vertices (i, j), or equivalently each pair of lymph nodes (n i , n j ), where the airway tree defines a physical space depicting all possible connections.

[0076] N×N cost matrix C(i, j) = c ij ,i,j=1,2,...,N, (4)

[0077] where c ij ≥0 is the cost associated with each edge (i, j). For our problem, c ij will be equal to lymph node n i and n j The airway distance between them.

[0078] The desired tour is a minimum-cost sequence of vertices (lymph nodes) that visits each vertex once and only once.

[0079] The present method for planning comprehensive nodal staging surgery combines these features. Specifically, the method uses the patient's chest CT scan to derive a multi-destination tour for a specific patient in order to efficiently navigate to all preselected lymph nodes. The planning task is formulated as a traveling salesman problem, a graph search problem defined as (1-4). To solve this problem, the concept of ant colony optimization (ACO) is applied to derive airway tours connecting the target lymph nodes [11-14]. ACO is an iterative algorithm with a random element in which a colony of "worker ants" creates individual tours through a graph. After each iteration, the most frequently visited tour edges acquire "pheromone" deposits that increase their attractiveness (likelihood) to the ants in future iterations. As described later, ACO was chosen because it consistently derives optimal or near-optimal tours in polynomial time complexity [64, 71].

[0080] In addition, our approach further constrains optimization by incorporating CT-derived nodal site labels, primary tumor location, and staging zone. Thus, this method identifies the optimal or near-optimal shortest tour that visits all target lymph nodes, consistent with recommended guidelines for comprehensive nodal staging.

[0081] Consider a complete nodal staging task for a patient with N pre-determined lymph nodes (1) and a primary tumor involved. The goal is to calculate the shortest distance tour through the airway tree in order to visit all lymph nodes once while adhering to current lung cancer staging guidelines.

[0082] The patient’s 3D preoperative chest CT scan was the primary input, delineating the discrete 3D chest space used for all calculations. As with modern image-guided bronchoscopy systems, we used high-resolution scans [33, 42, 57]. Specifically, we utilized scans with an axial plane resolution of Δx, Δy < 1.0 mm and a cross-sectional spacing of Δz = 0.5 mm. Such scans provide submillimeter accuracy for airway distance calculations and enable detailed definition of airways (whose luminal diameters range from 3.5 mm to 20 mm

[24] ) and clinically relevant lymph nodes (whose short-axis diameters are ≥ 10 mm

[56] ).

[0083] To calculate the required tours, the method consists of three main steps:

[0084] 1. CT pre-processing to identify lymph nodes, tumors, airway tree structure, preliminary single-node airway routes, and IASLC lymph node sites.

[0085] 2. Build the graph and phased areas, and set up the data structures required for patrol calculations.

[0086] 3. Patrol calculations to derive lymph node staging plans for subsequent image-guided bronchoscopic lymph node staging.

[0087] This round then provides a subsequent field-guided surgical plan.

[0088] Next, a comprehensive image-based multi-destination lymph node staging surgical guidance method was applied for on-site bronchoscopy guidance. According to the surgical plan, the method includes: 1) a series of graphic tools that provide visual cues and guidance instructions; 2) a guidance control strategy that guides the physician or surgical robot to effectively reach each surgical site; and 3) a surgical history mechanism that records all guidance and imaging information related to the examination site.

[0089] Both planning and guidance methods have been implemented as interactive software systems. The planning system accepts appropriate radiographic data and outputs a surgical plan consisting of this tour and other information to optimally reach each surgical site. The guidance system accepts the surgical plan as input and interacts with the surgical endoscope hardware to accept live endoscopic information.

[0090] Sections 1–4 provide details and validation results of the 3-step planning approach, and Section 5 presents the bootstrapping strategy and software system.

[0091] 1. CT pretreatment

[0092] Based on the patient's chest CT scan, CT preprocessing can derive various anatomical structures required for subsequent surgery. These anatomical structures include lymph nodes, primary tumor, airway tree structure, initial route to each lymph node, and IASLC lymph node station.

[0093] First, define the 3D region R t ,i=1,2,...,,N, (5)

[0094] These regions constitute each lymph node n in the set (1) i For our testing, we used a semi-automated live wire to define these areas on a chest CT scan.

[0095] Continuing, previously designed automated CT analysis methods segment the airway tree and its associated airway luminal surfaces [25, 28]. Based on these data structures, the airway tree centerline spanning the segmented airways is defined, where the centerline is modeled as a set of bronchoscope device postures P or view positions, given by the following formula: P = {p1, p2, ..., p L}, (6)

[0096] The p that constitutes the posture kThe airway tree is spaced equally, approximately 0.5 mm [24, 25, 68, 69]. Each pose consists of its (x, y, z) position in 3D chest space and a set of vectors indicating the direction of travel through the airway

[24] . We only need to work with the positions here. Note that the centerline consists of a set of hierarchically connected airway branches that span both lungs. The tree starts with the trachea, then branches into two sub-branches (the left main bronchus and the right main bronchus), which then continue to branch into other constituent sub-branches. Each branch in turn consists of poses from the set (6). Similarly for (6), a path p can be formed connecting two airway tree poses a and p b , consisting of an ordered sequence of poses in P: p a , p a+1 , p a+2 , ..., p b (7)

[0097] where p a With p a+1 Adjacent, p a+1 With p a+2 Adjacent, and so on.

[0098] Next, a series of individual preliminary bronchoscopic airway routes to each lymph node are calculated. These routes are used as the distances c required to later calculate the cost matrix C(i, j) in (4) ij As the bronchoscope navigates within the airway tree, the standard approach to representing the trajectory of the bronchoscope toward the diagnostic site is by traversing a set of discrete connected device postures along the airway centerline, as shown on a patient chest CT scan [1, 24, 33, 42, 69]. Using this approach, for each lymph node n i , i = 1, 2, ..., N, we compute a path r consisting of a series of consecutive device postures in P i :

[0099] In (8), route r i Starting from the proximal (apical) trachea position p1, proceed to the subsequent positions p2, p3, etc., and end in position Indicates when the bronchoscope reaches the target node n i Note that all routes start from posture p1, representing the starting position of the bronchoscope in the trachea. Furthermore, posture p1 corresponds to the root position from which all airway centerlines traversing the airway tree are generated.

[0100] As a final preprocessing operation, the existing automated method defines 15 disjoint IASLC node sites S c, represents a 3D tissue region in chest space, where c is in the node site index set I [30, 35, 43]:

[0101] This calculation follows the IASLC guidelines, where some sites are split into disjoint left and right chest sub-regions, while other sites are merged [8, 15]. Thus, in (9), the descriptors “L” and “R” refer to the left or right side of the chest, respectively, and more peripheral (higher indexed) IASLC sites are merged, e.g., IASLC sites 10 and 11 are merged into a single site “10-11”. Thus, S 4R The site includes, for example, tissue corresponding to the right side of site 4 shown in the chest CT, while site S 12-14L Includes tissue to the left of sites 12 to 14 that were merged.

[0102] 2. Construction of graphics and staging areas

[0103] This section constructs the graph structure {V, E, C(·, ·)} and staging regions derived from (1-4), which can add important inspection constraints related to staging.

[0104] The graph vertices V are obtained according to (2). Relative to the 3D chest space, each vertex i is actually physically connected to the lymph node n. i The tracheal route r is pre-calculated according to (8) i The final pose correspond.

[0105] Before specifying the edge set E, it should be noted that the CT-based segmented airway tree demarcates a closed "air" area that can be used as the bronchoscope navigation area in the 3D chest space. Therefore, if the bronchoscope is currently resting on the lymph node n i (Physically in posture ), then it is conceivable that it will be able to travel to any other N-1 lymph nodes n through a series of connected postures in the set P. j , j≠i, which is close to the result of (7). To represent these options on a graph, each vertex i has N-1 edges (i, j), i≠j, denoted by E.

[0106] To construct the cost matrix C, each entry c must be calculated ij , that is, from lymph nodes i to n j These distances can be calculated using standard binary tree concepts. Figure 1 As shown, from the position Lymph nodes i To the posture Lymph nodesj The distance is calculated as follows:

[0107] Where: p LCA Represents the lymph nodes n i and n j The route i and r j The last common ancestor pose of , and length(r) represents the length of the route: r={p a , p a+1 ,…,p b-1 , p b} (11)

[0108] It connects two postures p a , p b ∈P. In (10), length(r) is equal to the pose p a and p b The distance d(p a , p b ), calculated as follows:

[0109] Where: ||·|| is the Euclidean length. Therefore, filling (10-12) into the cost matrix C of (4) yields:

[0110] For i, j = 1, 2, ..., N, we assign valid costs only to edges (i, j) ∈ E. Since there is no valid edge cycle from vertex i to itself, we assign C(i, i) = ∞. Note also that c ij =c ji for alli≠j.

[0111] Figure 2 shows a simple diagram of the graph structure {V, E, C(·, ·)} drawn from a real human chest CT scan. The figure assumes that there is a right primary tumor and 6 lymph nodes distributed in the chest, as shown in the figure. The entire graph has 6 vertices and 30 edges. The values ​​in the 6×6 cost matrix C range from 1.7 to 348.1 mm. The optimal patrol starts from the left N3 lymph node with a length of 426 mm, expressed as. 2→4→6→1→3→5

[0112] Next, build the staging area N k , where each staging zone represents a CT-based subspace of the chest that contains lymph nodes with a specific lymph node descriptor. For example, staging zone N3 contains the portion of the chest space where N3 lymph nodes are located. Each N kAll include IASLC lymph node sites S c The site zone is a specified union of N and depends on the location of the primary tumor. Table 1 provides the site zone definitions based on the IASLC guidelines [26, 60]. Therefore, each N k The calculation of is as follows:

[0113] In (14), the site index set I of (9) is divided into three disjoint subsets:

[0114] Among them: I1, I2 and I3 correspond to the sites containing N1, N2 and N3 lymph nodes, respectively, as shown in Table 1.

[0115] Table 1: Mapping of IASLC lymph node sites to staging zones [26, 60]. Each row represents the lymph node stage descriptors N1 to N3 of the TNM staging class. The third and fourth columns of each row are used to construct each staging zone N according to the location of the patient's primary tumor as obtained by (9). k The index c∈I lists the site volume S c .

[0116] In the final operation, the site label l i and lymph node descriptor z i Distributed to each lymph node i ,,i=1,2,...,N,as shown below. First,

[0117] Among them: |·| represents the cardinality of the set, R i is n obtained according to (5) i The corresponding 3D area. Therefore, by convention, n i Assign to the site with the most overlap [35, 43]. i Given a descriptor z i According to Table 1. For example, if a patient with a left lung tumor has i =10-11R site has lymph nodes n i , then l i ∈I3, indicating that this is an N3 lymph node, and z i =N3.

[0118] 3. Patrol calculation

[0119] To derive the desired lymph node tour, we transform the search into a variant of the traveling salesman problem on the search space defined by the graph {V, E, C}. The goal is to derive the optimal tour F = {f1, f2, ..., f N} (16)

[0120] There are N ordered vertices f p ∈V, p = 1, 2, ..., N, and obey the following constraints:

[0121] (i.e., along this tour, only each vertex f is passed p Once), so the length of the tour

[0122] The time between all possible patrols F can be shortened as much as possible. In other words, we need to solve the optimization problem under the constraints of (17):

[0123] Note that according to (16-17), each candidate tour F visits the following edges in order (f p , f p+1 )∈E: (f1, f2), (f2, f3),..., (f N-1 , f N ).

[0124] To solve the problems raised in (16-19), we adapted a variant of the Ant Colony Optimization (ACO) algorithm [11-14]. In summary, the approach we adopted is as follows.

[0125] During M iterations, K worker ants independently construct complete candidate tours that traverse the vertex set V. During the iteration, each worker ant selects vertices driven by the state transition rule to create a candidate tour. In general, ants are attracted to shorter edges by a large amount of pheromones. The number of pheromones at an edge (r, s) shows its overall attraction to the ants when constructing a candidate tour. After all ants have completed the tour, they record a length of L (F best ) of the lowest cost patrol F best , and update the existing best patrol F accordingly opt . To end the iteration, the amount of pheromone at the edges is modified by two update rules. First, the global update rule adds pheromones inversely proportional to the length of the tour at each edge of the tour as a reward for the tour with the lowest cost. Next, a local update is used to add fixed pheromones to all edges visited by any K ants. During the two update processes, some pre-existing pheromones on all visited edges evaporate. Then, the same process is repeated in the next iteration and the final F is returned after M iterations. opt .

[0126] The complete detailed method is shown below.

[0127] To initialize the search, we set up two N × N matrices that quantify the relative attractiveness of each graph edge. The first of these is a fixed distance heuristic matrix η, which consists simply of the inverse of the edge cost C derived from (4) and (10-13): η(r,s)=1 / c rs , r, s=1, 2,...N. (20)

[0128] Next, the pheromone matrix τ (also called the edge desirability matrix) is initialized as: τ(r,s)=τ0,r,s=1,2,...,N, (21)

[0129] Among them: τ0>0 is a parameter, and the element τ(r,s) represents the vertex r (lymph node n r ) and vertex s (lymph node n s ) is the number of pheromones currently deposited on the edge connected to it. For τ0, we adopt a widely used suggestion

[14]

[0130] The higher the amount of pheromone at an edge, the more willing the ants are to walk along it. Therefore, all edges are equally attractive to ants at the beginning. The pheromone matrix is ​​a learned component that is shared among all ants and updated after each iteration. In the final initialization step, the ordered set of vertices and the length of the desired optimal tour are initialized:

[0131] Next, for each iteration m=1, 2, ..., M, the airway tree graph is randomly traversed under probability constraints to create candidate tours for each of the K ants. For the Kth ant, its tour vertex set and length need to be initialized first:

[0132] Next, randomly select a vertex r∈V to start the tour, thus obtaining: F k = {r}, (24)

[0133] Among them: All vertices r have equal probability of being selected at the beginning. Then, the next successor vertex s is selected by the state transition rule:

[0134] Where: Operator "A / B" represents the elements contained in set A but not in set B, and set V / F k = {u1, u2, ..., u p , ...} (26)

[0135] is an ordered set of vertices that have not yet appeared in the patrol of the Kth ant, and x represents a uniform random variable distributed on [0, 1], and

[0136] is the cumulative distribution function (cdf) defined according to the probability density function

[0137] (β>0 is a parameter). Please note that when patrolling F k When entering the state shown in (24) VF k ={1,2,...,r-1,r+1,...,N}.

[0138] In (28), p k (r,u) determines the candidate vertex u∈V / F k desirability, and represents the probability that ant k chooses to move from one vertex r to another vertex u. As (28) clearly shows, p k (r, u) will weigh the length and pheromone quantity of the candidate next edge (r, u), and adjust the relative weight with the help of β. The cdf c defined according to (27) and the random variable x in (25) k is a fair way to cycle through the various candidate successor vertices during tour construction. After applying the state transition rule (25), F k ={r,s}

[0139] And L(F k ) is updated according to (18). Subsequent vertices are added in a similar manner until This means that the patrol F k Completed. Note that we perform this process in parallel for all ants. After deriving all K tours, we can notice the tour with the shortest length:

[0140] If L(F best )<L opt , then F opt and L opt Update successful.

[0141] The iterative process can be summarized as follows: the pheromone matrix τ undergoes two updates, both of which aim to dissipate (evaporate) and increase the amount of pheromone on certain edges according to the travel experience gained by the K ants. First, the global update rule provides rewards to the best ants by depositing pheromones at all edges visited by the ants, and the total amount of pheromones accumulated is inversely proportional to the length of the ant's patrol; τ(r,s)=(1-α)·τ(r,s)+α·Δτ global (r, s), (30)

[0142] in Δτ global (r, s) = 1 / L(P best ). (31)

[0143] Next, in the current iteration, according to the local update rule, a fixed amount of pheromone Δτ is added to all edges visited by K ants. local (r, s) = τ0, and take an ant visiting a specific edge as an example, adding individual contributions one by one; that is, for the components of F k Visit all edges (r, s) of one of them, k = 1, 2, ..., K, τ(r,s)=(1-ρ)·τ(r,s)+ρ·Δτ local (r, s). (32)

[0144] In (30) and (32), 0≤α, ρ≤1 are parameters, and τ0 is obtained according to (22) and is used to give an initial value to the one obtained in (21). After these updates, the choice of edges visited by the ants in future patrols will be more diverse.

[0145] As defined in (24-32), the above tour construction process does not include any constraints related to specific surgical tasks. For some tasks, such tours are sufficient. For other surgical tasks that face professional constraints, it is not difficult to incorporate these professional constraints into the overall plan.

[0146] For example, returning to our lymph node staging scheme, we note that the method provided by (24–32) does not include staging region constraints. Therefore, all remaining vertices (lymph nodes), regardless of their assigned lymph node descriptor z i In any case, they can be selected as the next candidate vertex to visit according to the state transition rules (25-28). In order to perform appropriate lymph node staging surgery, all N3 lymph nodes should be visited first, followed by N2 and N1 lymph nodes. The staging area constraint can be easily incorporated by reconverting the vertex set V into an ordered set of subsets. V S ={V3,V2,V1}, (33)

[0147] where V3, V2, and V1 include the vertices corresponding to N3, N2, and N1, respectively. Therefore, the retransformed set V obtained by using (25-28) is S , that is, (33), the state transition rule first considers only the candidate N3 vertex in set V3 until it moves to V2 and V1 in sequence , select the next vertex s.

[0148] After completing M iterations, the optimal multi-destination patrol F opt The order in which the N lymph nodes are visited will be clear. We also construct a continuous route r through the airway tree for the practical bronchoscopic navigation problem. opt , similar to (7-8), this airway tree is constructed by bronchoscope device postures. This route starts at the proximal trachea at device posture p1 and then proceeds continuously to the terminal posture of each lymph node This route is easily constructed using the centerline structure given by (6); i.e.

[0149] Among them: posture p1, p2, ... p a , ...∈P and f1,f2,...f N ∈F opt . Please note that p a The poses that connect the terminal poses together can be used multiple times. In addition, the typical route r opt Physicians are required to first navigate the bronchoscope to the “most distant” lymph node f1 opposite the primary tumor, then track through the airway tree to the next lymph node, and so on.

[0150] Figures 3A-3D A complete example of this process is given for a lung cancer patient. Initially, there are 15 lymph nodes and a left-sided tumor. After automatic calculation, the airway tree, IASLC lymph node sites, and staging areas are generated, as shown in Figure 2. Figures 3A-3C Then, the patrol F is generated by automatic patrol calculation opt , as shown in parts a and d. The tour is 595 mm long and takes 248 milliseconds to calculate, as follows: F opt ={15,14,13,7,6,2,4,1,5,3,9,11,12,8,10}.

[0151] This tour is the true optimal choice in this case. Furthermore, the brute force search method took 28.4 seconds to generate this optimal value. It is worth noting that this graph shows how the lymph nodes are clustered in the graph based on the stage region. The bidirectional red arrows in the graph represent edges that connect all the lymph nodes in a given stage region, while the green arrows point in one direction, from the lymph nodes in the lower stage region to the lymph nodes in the higher stage region.

[0152] Algorithms 1 and 2 give the implementation methods of patrol calculation. In Algorithm 1, lines 2-4 are the initialization of the Ant Colony Optimization (ACO) algorithm, lines 5-12 can derive the optimal patrol, and lines 13-14 are the updates of the pheromone matrix. In each iteration, the algorithm uses the Create patrol function (Algorithm 2) to calculate the patrols of all K ants in parallel. The Create patrol function will perform loop calculations according to (24-28) and merge the phased area information given by (33).

[0153] Regarding the implementation of patrol calculation, we use the Eigen linear algebra library for matrix operations and OpenMP

[29] for parallel operations in Algorithm 1. Each ant is given thread-specific memory within an OpenMP thread. In line 11 of Algorithm 2, we use the 64-bit Mersenne twister algorithm to provide the random number in (28)

[46] . Data synchronization and pheromone update are performed at the end of each iteration.

[0154] A few points need to be clarified. Each preliminary route r in (8) i In fact, they all end in one posture This position can be used to treat lymph nodes i Perform the most comprehensive bronchoscopic biopsy. i The calculation method draws on the anatomical constraints displayed in the 3D chest CT scan and the specifications of the bronchoscope used [24, 69]. In addition, the last round of (19) ropt Adjacent postures were kept approximately 0.5 mm apart, consistent with the spatial resolution of CT scanning and sufficient for subsequent image-guided bronchoscopy.

[0155] 4. Planning method verification

[0156] The current method was tested using a database of 31 patients with lung cancer

[44] . These patients were admitted with informed consent according to a protocol approved by the ethics committee (patient age range: 28–85 years; 12 males, 18 females, 1 gender unknown). Scans were generated using Siemens Sensation 16, Emotion 16, or Sensation 40 multi-slice spiral CT scanners (13 with contrast medium and 18 without contrast medium). Each scan consisted of 512 × 512 axial plane sections with a section thickness of 0.75 mm and a section spacing of 0.5 mm. The number of sections per scan ranged from 570 to 720, while the axial plane resolution Δx = Δy ranged from 0.52 to 0.92 mm. A total of 528 lymph nodes were identified from the scans (mean per scan, 17; range, 3–38). These lymph nodes are distributed in staging areas as follows: N1 - 40 lymph nodes; N2 - 312 lymph nodes; N3 - 176 lymph nodes.

[0157] The ant colony optimization-based tour computation method was compared with four well-known algorithms: 1) brute force search (BFS)

[31] ; 2) nearest neighbor search (NN1), which searches for a path from each possible starting lymph node and selects only the nearest unvisited neighboring lymph nodes of each added lymph node until the tour is completed

[59] ; 3) K-nearest neighbor search, K = 2 (NN2), which explores the two nearest neighbors of each lymph node and selects the shortest overall tour that visits all lymph nodes

[32] ; and greedy “largest node first” search, which sorts the lymph nodes according to physical size within each staging area - although this is not an optimization algorithm in itself, it is a common clinical strategy for performing lymph node staging.

[0158] These tests used 25 cases from our database (9 males and 16 females; age range: 39-82) with a total of 403 lymph nodes found (mean, 16; range: [3, 31]). The goal of all methods is to find a tour that solves the optimization problem (19).

[0159] For the current method, the following default parameters were used in the patrol calculation: M = 1000 iterations; ant colony size: K = 30; weighting parameter: β = 4.0 in (28); best patrol reward: α = 0.1 in (30); pheromone evaporation rate: ρ = 0.08 in (32). Through extensive training, it was found that the parameter selection attempts to balance the goals of deriving optimal / near-optimal patrols while requiring compliance with tolerable computational time constraints

[36] . Finally, all four alternative algorithms use graphs and staging areas generated by CT preprocessing and graph / staging area construction.

[0160] We set a threshold of 2 billion exploration visits during optimization for each case, which corresponds to a computational time of about 5 minutes. Under this constraint, brute force search (BFS), an NP-hard solution to the traveling salesman problem, failed to reach optimality in 9 cases with the staging area constraint and in 16 cases without the staging area constraint. Therefore, for these cases, we first calculated the number of patrol brute force searches required to complete the search and then used linear regression to extrapolate the known times of other cases to complete the time estimation for these cases. The shortest estimated time for these cases with the constraint was 5.28E+06 milliseconds = 88 minutes (e.g., 17-lymph node case 20349_3_13), while the shortest estimated time without the constraint was 205 minutes (e.g., 14-lymph node case 20349_3_7).

[0161] Table 2 summarizes the average computation times for the test database, and Table 3 gives examples of computation times for six cases. For all results, we tested the methods with and without staging constraints (33). (Table 2 does not include times for NN1 and greedy search because these methods took less than 1 millisecond in all cases. However, they often produced unacceptable tours.)

[0162] Table 2: Mean round computation time in milliseconds per case. In each column, “mean ± sd” = mean ± sd, and “range” indicates the time range for all cases. “staging zone applied” indicates that staging zone constraints (33) were applied, while “no staging zone applied” indicates that no staging zone constraints were applied. ACO = ant colony optimization, NN2 = K-nearest neighbor search with K = 2, BFS = brute force search. Italic values ​​are case estimates that could not be optimally reached within the allocated time.

[0163] First, it can be observed that the ACO runs extremely fast, as all cases took less than 0.6 seconds. However, the computation times of the other methods vary widely. In this regard, NN2 computed in less than 10 milliseconds in eight cases with 12 or fewer lymph nodes, and brute force search computed in less than 10 milliseconds in four cases with eight or fewer lymph nodes (all with staging region constraints). Unfortunately, as shown in Table 3, the computation times of both methods increase exponentially with the number of lymph nodes. In an anecdotal test, we applied a staging region constraint to case 23-lymph node case 20349-15 and did not find a solution for one week (estimated runtime of approximately 1.64E+10 milliseconds = 190 days) - see Table 3 for details. On the other hand, the ACO and NN2 computed in just 249 milliseconds and 904 milliseconds, respectively, for the same case.

[0164] Table 3: Computation time of the rounds in milliseconds for the six database cases sorted by the number of lymph nodes N. “Stage zones applied” indicates the optimization when staging zone constraints are applied, while “no staging zones applied” indicates no constraints are used. The values ​​in italics are the same as those in Table 2 and are estimates.

[0165] In addition, if Figure 3D As shown in the figure in , the use of the stage zone constraint is consistent with expectations and basically decomposes the complex optimization problem into three independent smaller optimization problems. This often helps to significantly reduce the computation time, as shown in Table 2-3. Nevertheless, for more complex patient cases, it still takes a lot of effort to perform an optimal brute force search. Along these lines, if any stage zone contains a large number of lymph nodes (more than 16), the computational effort will be prohibitive. (Incidentally, the number of lymph nodes in each stage zone in our database is unevenly distributed. However, the reason for the uneven distribution is that we agreed to use a specific case random selection method. In fact, the lymph nodes can be distributed in any area according to the same assumption.)

[0166] Table 4: Planning / calculation times in seconds for the complete 3-step approach to comprehensive nodal staging surgical planning. Region delineation requires user interaction and region delineation times are calculated assuming 23 lymph nodes of interest. All other times are averages from 4 patient cases.

[0167] To place the round calculation in the context of the current approach to planning comprehensive nodal staging bronchoscopy, we measured the total planning time for 4 patient cases (mean, 23 lymph nodes per case; range, [13, 31]). Table 4 gives the detailed calculation times for our 3-step planning approach.

[0168] With the exception of the calculation of lymph node sites, all CT preprocessing operations are standard offline operations for image-guided bronchoscopic planning [2, 24, 57, 69]. Therefore, the computation time of these operations can be used as a benchmark for the total planning time. These operations are computationally intensive because they involve a lot of user interaction and the analysis of large 3D CT volumes (typical size: 200 MB). Interactive region segmentation via live wires is particularly time-consuming and depends on the number of lymph nodes considered. Table 5 assumes that each case contains 23 lymph nodes, and the time required to segment each lymph node is 1 minute (60 seconds). To reduce these interaction times, recently proposed automatic lymph node identification methods can be applied

[41] .

[0169] Fortunately, the additional steps required for planning a comprehensive nodal staging procedure (i.e., IASLC nodal sites, graphics, staging zones, nodal labeling, and ACO round calculations) are now fully automated and take less than 6 seconds. In contrast, standard CT preprocessing steps take more than 1 / 2 hour per case, accounting for more than 99% of the total planning time.

[0170] Next, we compared the tour lengths generated by various tour calculation methods. Note that the tour time varies greatly from patient to patient due to the different sizes of the lungs and the number of lymph nodes considered. Therefore, for the 25 cases considered in the calculation time test, we calculated two tour lengths: the actual tour length L and the relative tour length, which is calculated as follows: L rel =((L / L opt )-1)×100 (35)

[0171] Where: L is the patrol length found by the optimization method, L opt is the optimal (minimum) patrol length

[32] . opt can only be found by brute force search—as mentioned earlier, multiple cases would require an extremely complex search—so we ran an ant colony optimization algorithm (the best non-brute force search algorithm) on 50 separate trials to estimate L for each case. opt and use the minimum length obtained from these experiments as L opt As for (35), L rel = 0 means that the derived tour is 0% longer than the optimal tour, which indicates that the tour is almost certainly the optimal solution, while L rel=50 means the derived patrol ratio L opt 50% longer.

[0172] Table 5 summarizes the tour length results. We first observe that the absolute tour lengths vary widely across the 25 test cases, as expected, ranging from as high as 105 mm to as low as 562 mm. Therefore, these results provide limited insight. However, the mean line lengths provided by the ACO algorithm are closest to the optimal brute force search, and the resulting L rel The results are more insightful. The near-optimal L obtained by the ACO algorithm with and without the staging area constraint rel are 1.4% and 1.9% respectively, and the maximum (worst) patrol length L generated is rel Only than the corresponding L opt In addition, the ACO algorithm provided optimal rounds in 13 of the 16 cases where the true optimal solution was known, while the other methods provided optimal rounds in 2 cases for NN1, 5 cases for NN2, and 0 cases for greedy search. Obviously, both nearest neighbor algorithms performed poorly under the condition of staging constraints and therefore cannot be used as a reliable method in staging surgery applications. Interestingly, the greedy search method is a lymph node staging method frequently used by doctors, and the super-long rounds it provided were on average 1.5-2.5 times longer than the optimal rounds. If the largest lymph node-first biopsy method is used, doctors usually have to backtrack between the two lungs multiple times to complete the surgery.

[0173] Table 5: Summary of derived tour lengths over 25 test cases. The “Length” column summarizes tour length measurements in millimeters, while the “Relative Length” column uses metric units (35). “Staging Area Applied” means staging area constraints were applied, while “No Staging Area Applied” means no staging area constraints were applied. Since brute force search can provide optimal results, all L rel The values ​​are all equal to 0.

[0174] As expected, the derived tours were generally shorter when no staging zone constraint was applied. However, for the case with 3 lymph nodes N = 12, the optimal tour was the same with or without this constraint. This is particularly likely to occur in cases with a small number of lymph nodes (e.g., N < 4), but such cases do not require a truly comprehensive staging bronchoscopy that requires access to multiple lymph nodes and lymph node stations across the chest. Finally, staging zone constraints are of course a necessary condition for comprehensive staging. However, as emphasized in Section 4, the present method has important practical applications when no constraints are used.

[0175] 5. Guide strategy and software implementation

[0176] The planning system and image-guided bronchoscopy system are suitable for comprehensive lymph node staging using the present planning method. All software was coded in Microsoft Visual C++ using Visual Studio 2015 and utilized multiple external software libraries such as VTK, Qt, and OpenCV. We ran all tests on a custom Windows 10 64-bit computer (a 16-core 3.2GHz AMD Threadripper 1950x CPU, 64GB RAM) equipped with an NVIDIA GTX 1080Ti GPU. First, the entire system follows a workflow consisting of two parts, such as Figure 4 As shown:

[0177] 1. Offline planning system: Before staging bronchoscopy, the surgical plan was developed according to the previously described planning method;

[0178] 2. Image-guided bronchoscopy (IGB) system: used during live lymph node staging procedures to guide the physician along a pre-planned tour of the patient's relevant lymph nodes.

[0179] This two-component system is built around 2D space and is used to link planning data acquired in the 3D virtual chest space with the real space where the live surgery takes place ( Figure 5 ). The patient’s CT scan, PET scan (if available), chest model, and surgical plan are placed in a virtual chest space based on 3D CT. The live video of the bronchoscopy hardware, the bronchoscopy flow, and the ultrasound bronchoscope (if available) exist in the real space. The primary mechanism that connects the two spaces and enables image guidance is the CT-based virtual bronchoscopic (VB) view (also known as the endoluminal view) and the live video view of the interior of the airway tree captured by the bronchoscope. It is well known that the virtual bronchoscopic view is very similar in appearance to the live video bronchoscopic view

[48] .

[0180] First, the physician can use a planning subsystem called "LNS Planner" (LNS = Lymph Node Staging) to plan the i Quickly divide the required inspections, such as Figure 6 For the example in the figure, the ACO algorithm was used to generate a 301.2 mm long tour that accesses six lymph nodes. The automatically assigned lymph node site labels and staging areas are also shown.

[0181] After the tour calculation is completed, the doctor can use LNS Planner's visualization tools to interactively review the generated tours, such as Fig. 7A and Figure 7BThe user can dynamically view the patrol along the 3D airway tree or on the luminal surface of the airway tree.

[0182] Next, we provide a live guidance system that utilizes previously calculated multi-destination tours during bronchoscopy. The system takes acquired live endoscopic video as input and provides virtual guidance along pre-planned tours. This online guidance system also provides interactive plan review, live surgical guidance for multiple bronchoscopic targets, and supports comprehensive multi-destination lymph node staging procedures in line with clinical best practice guidelines.

[0183] During live surgery, the system display provides a range of tools to assist in guidance, such as Figure 8 These tools include:

[0184] 1. Description box ( Figure 8 - Center): Written instructions displayed during the procedure. The top line of text presents nodal endpoint information, i.e., the current number of nodes, location of the region of interest, IASLC site, and N stage descriptor. The second line of text indicates the next recommended bronchoscopic maneuver - and is updated after each guided action is completed.

[0185] 2.CT video supporting tools ( Figure 8 -Middle left): Provides the main visual tools to consider when guiding. The left view shows the live endoscope video source, while the right view shows the virtual bronchoscope view generated by the system for the next guided position. The tool combines the real video and virtual space in a single interface so that both real and virtual bronchoscope postures can be viewed at any time. In addition, as shown in the figure, the tool can also host virtual space in the real bronchoscope video

[48] .

[0186] 3. Lymph Node Site Mapping Tool ( Figure 8 - Bottom left): Static image showing IASLC lymph node sites and live updating tracking screen - This screen will show all lymph node sites resolved during staging surgery. The checklist on the right is an easy-to-follow surgical progress tracker to help track resolved lymph nodes and lymph node sites. Yellow check marks indicate partially resolved multiple lymph node sites, while colored boxes with green check marks indicate resolved lymph node sites. The lymph node sites are shown on the static image, and the lobe of the primary tumor is indicated using a small black circle with a T.

[0187] 4.3D Airway Tree Renderer ( Figure 8 -Top center right): This view is a global view of the current guidance system position. The 3D airway tree, tracheal centerline, guidance system position are displayed in the form of crosshairs, and the current guidance path is displayed as a blue line.

[0188] 5. Surgery Sequencing Tool ( Figure 8 -Upper left): This tool serves as a centralized control interface for the guidance system, providing a detailed list of each region of interest site, staging area, and color, while updating guidance instructions for the physician. The tool can start and stop multi-destination guidance mode and move the system to site-specific intracavitary views.

[0189] 6. Virtual Panorama ( Figure 8 -Top right): A wide-angle view of the endoluminal renderer, which expands on the virtual bronchoscopic view displayed by the CT-Video companion tool, allowing visualization of regions of interest that may not be visible when the scope is retracted.

[0190] 7. Coronal Projection Tool ( Figure 8 -Far right): A chest X-ray view with the tracheal centerline overlaid, which allows real-time tracking of the system's progress along the tour path. This visualization tool is similar to the 3D airway tree view, providing visual context for the current guidance position and the final endpoint of the valid region of interest along the tour path.

[0191] CT video companion tools, caption boxes, lymph node site mapping tools, and a 3D airway tree renderer form the main guidance pillars of the system, allowing physicians to determine the status of the bronchoscope in both the virtual guidance space and the real surgical space while tracking the progress of the procedure.

[0192] The Lymph Node Site Mapping tool displays a progress log along the way through the chest. This is a very helpful and reassuring tool for the physician as it shows the progress during the procedure, the last area of ​​interest visited, and the remaining lymph node sites to be visited.

[0193] Other tools help inform the physician of their current location and assist the physician in deciding to follow a recommended guided maneuver. All previously calculated region of interest planning and labeling information is transferred to the guidance system.

[0194] The operations described in the instruction box are automatically calculated. The operation suggestions are as follows:

[0195] 1. Advance: Move the bronchoscope deeper into the lungs to the indicated airway location. The guidance system will dynamically play the suggested actions and end the playback at the next bifurcation.

[0196] 2. Pull back: Pull the endoscope back to a more proximal airway branch in the airway tree. This maneuver is useful when moving from a deeper area of ​​interest to a more superficial airway, such as the trachea and right upper lobe (RUL) bronchus. The virtual maneuver animation allows you to smoothly pull the virtual endoscope back to the LCA bifurcation / branch in one motion.

[0197] 3. Positioning and sampling: This is the final instruction when the current region of interest is reached. This instruction is used when the system reaches the terminal guidance posture to guide the doctor to position and sample the lymph nodes as needed.

[0198] For multi-destination booting, such as Figure 8 As shown, navigation begins at the trachea of ​​lymph node 1 and guides the physician sequentially along the tour to the next region of interest. During the guidance process, all tools are synchronized with the global position of the guidance system. Upon reaching the destination of a specific region of interest, the system invokes several actions: 1) a bright green display appears indicating the optimal sampling location for the current lymph node; 2) a "locate and sample" command is issued; and 3) arrows are used on the site map to indicate the site just solved and the currently solved site. Fig. 9 An example is given when the system reaches lymph node 1.

[0199] As a second example of the composite guidance system, Figure 10 shows the use of the system in case 21405-3a, a patient with a right-sided tumor. As shown in the site progress map, the physician is tasked with locating lymph nodes in IASLC sites 12L, 4R, 4L, 7, 10R, and 12R. For the composite system view shown, the physician has just confirmed that the system has reached the first left-sided lymph node at the 4th LLLB (left lower lobe bronchus) site 12L. The 3D airway tree view depicts the physician's current position with red crosshairs. The companion live video bronchoscopy view and the corresponding CT-based virtual bronchoscopy view are displayed on the CT-video tool, which shows the lymph nodes with bright green areas. Finally, the site progress map shows the physician's approximate airway location on the lung map and checks the box for site 12L to confirm that all pre-planned site 12L lymph nodes have been inspected.

[0200] Next, Fig.11 Some example system views are shown for a patient with extensive IASLC nodal sites with 38 thoracic lymph nodes. This tour was calculated in 1 second using this method, but would not have been derived using other existing planning methods. This situation is common in true comprehensive nodal staging. Moreover, surgeries of similar complexity are occurring in other areas.

[0201] We tested the two-part image-guided system using a planning and guidance approach using retrospective human case series, controlled live animal studies, a multiphysician-guided study, and an ongoing prospective human study

[36] .

[0202] As a natural extension useful for endoscopic surgery planning, we also have a method to find the canonical view pose p c, allowing access to landmarks such as the airway tree within branching structures. This approach can be easily extended to define such endoluminal views of other branching anatomical structures such as the vasculature within the colon, brain, or liver.

[0203] For the airway tree, we created a rule-based decision tree approach to locate the canonical site pose p in the airway tree. c . Given a labeled tracheal tree and prior knowledge of IASLC lymph node site specifications, this method can move the virtual camera to a position close to a given IASLC lymph node site. These airway labels and lymph node sites are derived by the LNS Planner system from the patient's CT scan. The desired view pose is defined by a 3D position within the airway tree and a series of orientations (pitch, roll, yaw, and up vector) that define the 3D viewpoint required for endoscope positioning. The algorithm that needs to be run to find this information is as follows:

[0204] 1. Identify the trachea as branch 1 and current branch b m .

[0205] 2. Jump to b m The last viewing position in v end .

[0206] 3. Identify b m Sub-branch of: b a and b b .

[0207] 4. Label the sub-branches b according to the explicit knowledge related to each site a and b b .

[0208] 5. By moving back to the branch length 0.8·b i 80% of the viewing area to provide posture p c , forming an appropriate intracavitary view of the site, thus defining the site view.

[0209] 6. Repeat from step 2, and change b m Set each b a and b b , until all postures are derived.

[0210] This approach allows for easy derivation of a normative lymph node site view for each IASLC lymph node site. Figures 12A-12I Examples of canonical view poses derived around three example sites are given, and Fig.13 Figure 2 shows a tool in our software that can automatically derive these poses for multiple lymph node sites and airway tree locations; other poses may also be derived from this tool.

[0211] This approach can be easily extended to define similar view poses for major branching points in the airway tree that are commonly used as landmarks in the examination of the patient's airway. These landmarks include the aorta, left and right mainstem bronchi, and segmental airway bifurcations. In general, these view poses can serve as additional "general" diagnostic sites to help organize the examination of tissue areas that delineate lymph node sites, airway wall locations, the location of diffuse or ground-glass nodules, or the location of lobes / segments, etc. These view poses are particularly useful in situations where specific suspicious sites such as lymph nodes or tumors are difficult to define.

[0212] Finally, the guidance system includes a tool that can save detailed surgical history as the live surgery progresses, e.g. Fig.14 As the procedure progresses, complete status information related to all equipment operations performed, all guidance display views, all precise locations visited within the equipment travel space (e.g., airway during bronchoscopy), and all region of interest imaging features (e.g., CT data and volumes, etc.) can be recorded and saved. Our tool can reload the history to allow for post-operative review. The physician can call up our guidance system, which, as mentioned earlier, is equipped with a comprehensive graphic viewer for surgical reconstruction and study. In addition, the surgical endoscopy video can be linked to the surgery. Standard playback controls such as fast forward and rewind are also available. This is particularly helpful for reviewing specific regions of interest under consideration during surgery.

[0213] This information can be of great value in the following ways:

[0214] 1. The doctor can use this guidance system to reconstruct the entire surgery and focus on specific areas. This allows the doctor to evaluate the surgical results by completing a detailed review and plan follow-up surgery if necessary.

[0215] 2. The data can be used for subsequent surgeries. Thus, the treatment and / or disease progression of previously examined sites can be evaluated. In addition, sites can be compared in real time during subsequent surgeries, and entire previously completed surgeries can be replayed in real time.

[0216] 3. The tool may facilitate long-term monitoring of drug / treatment delivery and testing.

[0217] 4. It can monitor the occurrence of lung tumors shown in chest CT and / or PET imaging over a long period of time.

[0218] Applications such as those listed above are particularly important in the management of lung cancer patients, who must often be followed long-term over many years, as well as in the monitoring of patients with heart disease and colon cancer.

[0219] In summary, the lack of appropriate tools prevents physicians from planning complex, multi-site endoscopic surgeries in branching, hollow network organ systems in an organized manner. Additionally, there is a lack of mechanisms to guide physicians through the entire organ system during the procedure. Finally, there is a lack of tools to perform effective follow-up surgeries. As a result, physicians are forced to perform either inadequate, simplified surgeries targeting a few sites or none at all.

[0220] To address this issue, the current method and system implementation facilitates planning and guiding complex multi-site endoscopic surgical procedures within branched hollow network organ systems. As demonstrated in the lung cancer staging task, the present method facilitates planning of comprehensive lymph node staging bronchoscopy. Results from human CT data show that the present method can provide optimal or near-optimal inspection through the lungs within seconds, regardless of the number of selected lymph nodes. This is advantageous over other methods, which either give unacceptable inspections or are unable to provide inspections within practical time limits. In addition, the present method combines IASLC lymph node site labels and staging zone constraints to provide surgical plans that comply with state-of-the-art clinical staging guidelines.

[0221] This planning method could be incorporated into an image-guided bronchoscopy system tailored for staging bronchoscopy. Laboratory, animal, and human studies have shown that this system has great potential for efficient comprehensive lymph node staging, regardless of physician experience / skill level

[36] .

[0222] By using this approach, physicians can perform more complex, multi-site surgeries that were previously not possible or impossible to perform. With the efficiency provided by this approach and the intuitive cues provided by our guidance strategy, physicians can perform complex surgeries with less stress, work faster, and complete tasks with greater accuracy. These guidance aids provide a more thorough approach while reducing potential confusion about the completeness of the procedure. Finally, the path history feature provides a way to review complex surgeries and improve planning for future follow-up surgeries. Overall, this approach can provide more accurate and complete outcomes for patients.

[0223] In the emerging field of robotic-driven bronchoscopy [18, 67], the current approach could provide important long-term value. Automated robotic devices can also follow guidance strategies and quantitative information. In addition, the robotic system can utilize visualization aids from the guidance system so that the operator can monitor the robot's progress. It is possible to use the robotic system to perform detailed staging procedures involving more than 20 lymph nodes, which may be too tedious for humans to perform. Although we focused on staging bronchoscopy, our approach could be applied to other multi-destination tasks without the constraints of staging zones. Examples of pulmonary applications include: 1) guiding multimodal airway examinations to detect early mucosal cancer lesions along the airway wall

[61] ; 2) guiding asthma or cancer treatment delivery at multiple airway sites; and 3) coordinating airway examinations during repeat follow-up visits to monitor ongoing or recurrent lung disease. The approach is also applicable to other organ areas involving complex branching structures, such as the lung or renal vasculature. References [1] Fumihiro Asano, Ralf Eberhardt, and Felix JF Herth. Virtual bronchoscopic navigation for peripheral pulmonary lesions. Respiration, 88(5): 430–40, November 2014. [2] F. Asano. Practical application of virtual bronchoscopic navigation. In A. Mehta and P. Jain, editors, Interventional Bronchoscopy, Respir. Med., vol. 10, pp. 121-140, Humana Press, 2013. [3] J. Bentley. Fast algorithms for geometric traveling salesman problems. ORSA J. Comput., 4(4): 387–411, 1992. [4] Christopher Burrows, Fangde Liu, Alexander Leibinger, Riccardo Secoli, and Ferdinando Rodriguez y Baena. Multi-target planar needle steering with abio-inspired needledesign. In Giovanni Boschetti and Alessandro Gasparetto, editors, Advances in Italian Mechanism Science, pages 51-60, 2017. [5] Sharad Chandrika and Lonny Yarmus. Recent developments in advanced diagnostic bronchoscopy. European Respiratory Review, 29(157): 1–11, September 2020. [6] Gerard J Criner, Ralf Eberhardt, Sebastian Fernandez-Bussy, Momen M. Wahidi, et al., Interventional bronchoscopy: State-of-the-art review. Am. J. Respir. Crit. Care Med., vol. (1): 29–50, July 2020. [7]GACroes.A method for solving traveling-salesmanproblems.Oper.Research, Issue 6(6): Pages 791-812. [8] Frank C Detterbeck, Daniel J Boffa, Anthony W Kim, and Lynn T Tanoue. The eighth edition lung cancer stage classification. Chest, 151(1): 193–203, January 2017. [9] Frank Detterbeck, Jonathan Puchalski, Ami Rubinowitz, and David Cheng. Classification of the thoroughness of mediastinal staging of lung cancer. Chest, 137(2): 436–442, February 2010.

[10] MY Dolina, DC Cornish, SAM Merritt, L. Rai, R. Mahraj, WE Higgins, and R. Bascom. Interbronchoscopist variability in endobronchial path selection: an asimulation study. Chest, 133(4): 897–905, April 2008.

[11] Marco Dorigo, Mauro Birattari, and Thomas Stutzle. Ant colony optimization. IEEE Comput. Intell. Mag., 1(4): 28–39, November 2006.

[12] Marco Dorigo and Luca Maria Gambardella. Anti colonies for the traveling salesman problem. Biosystems, 43(2): 73–81, July 1997.

[13] Marco Dorigo, Vittorio Maniezzo, and Alberto Colorni. Ant system: optimization by a colony of cooperating agents. IEEE Trans. Sys., Man, Cyber., Pt. B, 26(1): 29–41, February 1996.

[14] M. Dorigo and LMGambardella. Ant colony system: a cooperative learning approach to the traveling salesman problem. IEEE Trans. Evol. Comput., 1(1): 53–66, April 1997.

[15] AH El-Sherief, CT Lau, CC Wu, RL Drake, GF Abbott, and TW Rice. International association for the study of lung cancer (IASLC) lymphnode map: radiologic review with CT illustration. Radiographics, 34(6): 1680–1691, October 2014.

[16] A. Ernst, G. A. Silvestri, and D. Johnstone. Interventional pulmonary procedures: guidelines from the American College of Chest Physicians. Chest, 123(5): 1693–1717, May 2003.

[17] John L. Falcone, Xiaotian Chen, and Giselle G. Hamad. The traveling salesman problem in surgery: economy of motion for the FLS peg transfer task. Surg. Endosc., 27(5): 1636–1641, May 2013.

[18] David Fielding and Masahide Oki. Technologies for targeting the peripheral pulmonary nodule including robotics. Respirol., 25: 914–923, September 2020.

[19] James H Finigan and Jeffrey A Kern. Lung cancer screening: past, present and future. Clin. Chest Med., 34(3): 365-371, September 2013.

[20] Erik Folch, Daniel B Costa, Jeffrey Wright and Paul A VanderLaan. Lung cancer diagnosis and staging in the minimally invasive age with increasing demands for tissue analysis. Transl. Lung Cancer Res., Vol. 4(4): pp. 392 - 403, August 2015.

[21] Erik Folch and Adnan Majid. Point: are >50 supervised procedures required to develop competency in performing endobronchial ultrasound-guided transbronchial needle aspiration for mediastinal staging? Yes. Chest, Vol. 143(4): pp. 888 - 891, April 2013.

[22] Michael R. Garey and David S. Johnson. Computers and Intractability: A Guide to the Theory of NP-Completeness. W.H. Freeman, San Francisco, 1979.

[23] R. Gayle, P. Segars, M.C. Lin and D. Manocha. Path planning for deformable robots in complex environments. Robotics: Science and Systems, pp. 225 - 232, June 2005.

[24] J.D. Gibbs, M.W. Graham, R. Bascom, D.C. Cornish, R. Khare and W.E. Higgins. Optimal procedure planning and guidance system for peripheral bronchoscopy. IEEE Trans. Biomed. Engin., Vol. 61(3): pp. 638 - 657, March 2014.

[25] JD Gibbs, MW Graham, and WE Higgins. 3D MDCT-based system for planning peripheral bronchoscopic procedures. Comp. Biol. Med., 39(3): 266–279, March 2009.

[26] Peter Goldstraw, Kari Chansky, John Crowley, Ramon Rami-Porta, et al., The IASLC lung cancer staging project: proposals for revision of the TNM stagegroupings in the forthcoming (eighth) edition of the TNM classification for lung cancer. J.Thor.Oncol., Issue 11(1): pp. 39-51, January 2016.

[27] B.E. Goyette and C.N. Riviere. Reducing operating time of a crawling robot for epicardial surgery. In Proc. 2010 IEEE 36th Ann. Northeast Bioeng. Conf., pp. 1–2, 2010.

[28] MW Graham, JD Gibbs, DC Cornish, and WE Higgins. Robust 3D Airway-Tree Segmentation for Image-Guided Peripheral Bronchoscopy. IEEE Trans. Medical Imaging, 29(4): 982–997, April 2010.

[29] Guennebaud, Benoit Jacob et al., Eigen.URl: http: / / eigen.tuxfamily.org, 3, 2010.

[30] WE Higgins, T. Kuhlengel and R. Bascom. Automated 3D CT-Based lymphnode station definition for the IASLC 8th edition guidelines. Am. J. Respir. Crit. Care Med., Issue 201: Page A4447, May 2020.

[31] Karla L Hoffman, Manfred Padberg and Giovanni Rinaldi. Traveling salesman problem. In Encyclopedia of operations research and management science, pp. 1573-1578, Springer, 2013.

[32] David S Johnson and Lyle A McGeoch. The traveling salesman problem: Acase study in local optimization. In EHLAarts and JK Lenstra, editors, LocalSearch in Combinatorial Optimization, pages 215-310, John Wiley & Sons, 1997.

[33] R. Khare, R. Bascom, and WE Higgins. Hands-free system for bronchoscopy planning and guidance. IEEE Trans. Biomed. Eng., 62(12): 2794–2811, December 2015.

[34] C Matthew Kinsey and Douglas A Arenberg. Endobronchial ultrasound-guided transbronchial needle aspiration for non-small cell lung cancer staging. Am. J. Resp. Crit. Care Med., 189(6): 640–649, March 2014.

[35] Trevor K. Kuhlengel and William E. Higgins. Bronchoscopic procedure planning for systematic lymph node analysis. In Med. Imag.: Image-Guided Proc., Robotic Interven. and Modeling, Volume 10951, page 109511O, International Society for Optics and Photonics.

[36] T. Kuhlengel. Planning and Guidance of Bronchoscopy for Comprehensive Lymph Node Staging. PhD thesis, The Pennsylvania State University, Department of Computer Science and Engineering, 2022.

[37] Vijay Kunadian, Alaide Chieffo, Paolo G Camici, Colin Berry, Javier Escaned, Angela HEM Maas, Eva Prescott, Nicole Karam, Yolande Appelman, Chiara Fraccaro et al., An EAPCI expert consensus document on ischaemia with non-obstructive coronary arteries in collaboration with european society of cardiology working group on coronary pathophysiology and microcirculation endorsed by coronary vasomotor disorders international study group. Europ. Heart J., Volume 41(37): pages 3504 - 3520, 2020.

[38] Didier Lardinois, Paul De Leyn, Paul Van Schil, Ramon Rami Porta, et al., ESTS guidelines for intraoperative lymph node staging in non-small cell lung cancer. Eur. J. Cardio-thoracic Surg., 30(5): 787–792, 2006.

[39] P. CMH Kuijpers, RH Murga, I. Inza, and S. Dizdarevic. Genetical algorithms for the travelling salesman problem: A review of representations and operators. Artif. Intell. Rev., 13(2): 129-170, 1999.

[40] S. Lin and BW Kernighan. An effective heuristic algorithm for the traveling-salesman problem. 21(2): 498–516, 1973.

[41] Jiamin Liu, Joanne Hoffman, Ronald M Summers, et al., Mediastinal lymphnode detection and station mapping on chest CT using spatial priors and random forest. Med. Phys., 43(7): 4362–4374, July 2016.

[42] Xiongbiao Luo and Kensaku Mori. Real-time bronchoscope three-dimensional motion estimation using multiple sensor-driven alignment of CTimages and electromagnetic measurements. Comp. Med. Imaging Graphics, Issue 38(6): Pages 540-548, September 2014.

[43] K. Lu, P. Taeprasartsit, R. Bascom, RPM Mahraj, and W.E. Higgins. Automatic definition of the central-chest lymph-node stations. Int. J. Computer Assisted Radiol. Surgery, 6(4): 539–555, 2011.

[44] K. Lu, P. Taeprasartsit, R. Bascom, RPM Mahraj, and W.E. Higgins. Automatic definition of the central-chest lymph-node stations. Int. J. Computer Assisted Radiol. Surgery, 6(4): 539–555, 2011.

[45] J. F. Aguilar-Madeira, H. L. Pinina, E. B. Pires, and J. Monteiro. Surgical correction of scoliosis: numerical analysis and optimization of the procedure. Int. J. Numer. Meth. Biomed. Eng., 26(9): 1087–1098, September 2010.

[46] Makoto Matsumoto and Takuji Nishimura. Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Transactions on Modeling and Computer Simulation, 8(1): 3–30, January 1998.

[47] S. A. Merritt, J. D. Gibbs, K. C. Yu, V. Patel, L. Rai, D. C. Cornish, R. Bascom, and W. E. Higgins. Image-guided bronchoscopy for peripheral lung lesions: A phantom study. Chest, 134(5): 1017–1026, November 2008.

[48] ​​S. A. Merritt, R. Khare, R. Bascom, and W. E. Higgins. Interactive CT-video registration for image-guided bronchoscopy. IEEE Trans. Med. Imaging 32(8): 1376–1396.

[49] Russell J Miller, Lakshmi Mudambi, Macarena R Vial, Mike Hernandez, and George A Papen. Evaluation of appropriate mediastinal staging among endobronchial ultrasound bronchoscopists. Ann. Am. Thorac. Soc., 14(7): 1162–1168, July 2017.

[50] H. Minami, Y. Ando, ​​F. Nomura, S. Sakai, and K. Shimokata. Interbronchoscopist variability in the diagnosis of lung cancer by flexible bronchoscopy. Chest, 105(2): 1658–1662, June 1994.

[51] DP Naidich and TJ Harkin. Airways and lung: correlation of CT with fiberoptic bronchoscopy. Radiol., 197(1): 1–12, October 1995.

[52] National Lung Screening Trial Research Team.Results of initiallow-dose computed tomographic screening for lung cancer.New Engl.J.Med., Issue 368(21): Pages 1980-1991, May 23, 2013.

[53] Carl A Nelson, David J Miller, and Dmitry Oleynikov. Modeling surgical tool selection patterns as a "traveling salesman problem" for optimizing amodular surgical tool system. Stud. Health Technol. Inform., Issue 132: Pages 322-326, 2008.

[54] David E Ost, Armin Ernst, Xiudong Lei, David Feller-Kopman, et al., Diagnostic yield of endobronchial ultrasound-guided transbronchial needle aspiration: results of the AQuIRE bronchoscopy registry. Chest, 140(6): 1557–1566, December 2011.

[55] S.Y. Park, D. Sargent, I. Spofford, K.G. Vosburgh, et al., A colon video analysis framework for polyp detection. IEEE Trans. Biomed. Eng., 59(5): 1408–1418, May 2012.

[56] Ramón Rami-Porta, Sergi Call, Christophe Dooms, Carme Obiols, Marcelo Sánchez, William D Travis, and Ivan Vollmer. Lung cancer staging: a concise update. Eur Respir J, 51(5): 1–17, May 2018.

[57] PJ Reynisson, H.O. Leira, T.N. Hornes, E.F. Hofstad, et al., Navigated bronchoscopy: a technical review. J. Bronchology Interv. Pulmonol., 21(3): 242–264, July 2014.

[58] AM Reynolds. Chemotaxis can provide biological organisms with good solutions to the travelling salesman problem. Phys. Rev. E, 83(5): 052901-1-052901-4, May 2011.

[59] D. Rosenkrantz, R. Stearns, and P. Lewis. An analysis of several heuristics for the traveling salesman problem. SIAM J. Comput., 6(3): 563–581, 1977.

[60] Valerie W Rusch, Hisao Asamura, Peter Goldstraw et al., The IASLC lungcancer staging project: a proposal for a new international lymph node map in the forthcoming seventh edition of the TNM classification for lungcancer. J. Thor. Oncol., Issue 4(5): pp. 568-577, April 2009.

[61] Alejandro H Sardi and Shaheen Islam. Early lung cancer detection, mucosal and alveolar imaging. Curr. Opinion Pulm. Med., 22(3): 271–280, May 2016.

[62] Rebecca L. Siegel, Kimberly D. Miller, and Ahmedin Jemal. Cancer statistics. CA Cancer J. Clin. 2020, 70(1): 7–30, January / February 2020.

[63] GA Silvestri,AV Gonzalez,MA Jantz,ML Margolis, et al., Methods forstaging non-small cell lung cancer: Diagnosis and management of lung cancer, 3rd ed: American College of Chest Physicians evidence-based clinical practice guidelines. Chest, 143(5):e211S–e250S, May 2013.

[64] D. Sudholt and C. Thyssen. Running time analysis of ant colony optimization for shortest path problems. J. Disc. Alg., 10(7): 165–180, January 2012.

[65] Lynn T Tanoue, Nichole T Tanner, Michael K Gould, and Gerard A Silvestri. Lung cancer screening. Am. J. Respir. Crit. Care Med., 191(1): 19–33, 1 January 2015.

[66] Huai-Kuang Tsai, Jinn-Moon Yang, Yuan-Fang Tsai, and Cheng-Yan Kao. Anevolutionary algorithm for large traveling salesman problems. IEEE Trans. Syst. Man Cybern. Part B Cybern., 34(4): 1718–1729, August 2004.

[67] M. M. Wahidi, F. J. F. Herth, A. Chen, G. Cheng, and L. Yarmus. State of the art: Interventional pulmonology. Chest, 157(3): 724–736, March 2020.

[68] KC Yu, EL Ritman, and WE Higgins. System for the analysis and visualization of large 3D anatomical trees. Comp. Biol. Med., 37(12): 1802–1820, December 2007.

[69] Xiaonan Zang, Jason D. Gibbs, Ronnarit Cheirsilp, Patrick D. Byrnes, Jennifer Toth, Rebecca Bascom, and William E. Higgins. Optimal route planning for image-guided EBUS bronchoscopy. Comp. in Biol. and Med., Issue 112: Page 103361, September 2019.

[70] Wen-Zhao Zhong, Si-Yang Liu and Yi-Long Wu. Numbers or stations: Fromsystematic sampling to individualized lymph node dissection in “Small-cell lung cancer.” J. Clin. Oncol., 35(11): 1143-1145, April 2017.

[71] Y. Zhou. Runtime analysis of an ant colony optimization algorithm for TSP instances. IEEE Trans. Evol. Comp., 13(5): 1083-1092, October 2009.

Claims

1. A method for planning a tour through a hollow organ system to access a set of target diagnostic sites by endoscopic inspection, the method The following steps are involved: receiving a 3D preoperative imaging scan depicting the diagnostic site and the hollow organ system; Receiving information of the target diagnosis part in the 3D imaging scan; calculating a 3D virtual space of the hollow organ system based on the 3D imaging scan; calculating a series of preliminary endoscopic routes to each diagnostic site, each preliminary endoscopic route consisting of a set of consecutive device poses through the hollow organ system, the poses connecting each target diagnostic site to all other target diagnostic sites, each pose consisting of its coordinates in the 3D virtual space and a set of vectors indicating a direction of travel through the hollow organ system; defining the cost between each pair of diagnostic sites as the accumulated distance of a minimum set of continuous postures connecting each pair of diagnostic sites together within the hollow organ system; as well as An optimal tour with minimum cost is calculated, which corresponds to the minimum set of consecutive postures that visits all target diagnostic sites once and only once.

2. The method according to claim 1, in, The step of calculating the optimal patrol comprises: Constructing a graph structure defined by a set of N vertices, a set of edges, and a cost matrix, wherein each of the N vertices corresponds to a diagnosis site, each of the N vertices has N-1 edges with other N-1 vertices, and the cost matrix has a plurality of entries, each entry of the cost matrix is ​​defined as a distance between two diagnosis sites; and Compute the shortest tour corresponding to a minimum-cost sequence of vertices that visits each vertex once and only once.

3. The method according to claim 1, in, The 3D pre-operative imaging scan is a CT or MRI scan.

4. The method according to claim 1, in, Such hollow organ systems include the lung airways, the vasculature in the heart, brain, liver, kidneys, or hollow areas in the colon, stomach, bladder, pelvis / abdomen.

5. The method according to claim 1, in, The endoscope is a bronchoscope, a colonoscope, a laparoscope, an angioscope or a cystoscope.

6. The method according to claim 1, in, The diagnostic site is a lymph node, a suspicious lesion, or a therapeutic delivery site.

7. The method according to claim 1, further comprising: include: Disjoint node sites representing tissue regions in the chest 3D virtual space are defined following predefined guidelines.

8. The method according to claim 7, further comprising: include: The node sites are mapped to staging areas.

9. The method according to claim 8, further comprising: include: The constraints imposed by the staging areas are incorporated in calculating the optimal tour.

10. The method according to claim 7, in, The predefined guidelines are the IASLC guidelines.

11. An image-guided endoscopy method for use during live medical procedures to guide an operator along a pre-planned tour of a patient's target diagnostic site within a hollow organ system, said method include: Designing a pre-planned tour of the target diagnostic site of the patient according to the method of claim 1; providing a live video endoscopic view within the real space of said live medical procedure; linking the planning data obtained in the 3D virtual space with the real space of the live medical procedure and enabling image guidance based on a virtual endoscopic endoluminal view and the live video endoscopic view of the hollow organ system; Provides suggested device maneuvers to reach each diagnostic site; indicating when to arrive at a particular site during said live medical procedure; as well as A surgical history is tracked indicating which areas have been accessed and which areas have not been accessed during the live medical procedure.

12. A system for continuously directing endoscopy to multiple target diagnostic sites along a pre-planned tour of a patient's hollow organ system during live surgery, include: an endoscope operable to navigate within said hollow organ system; a display device operable to display live, real endoscopic (RE) images and live video endoscopic views obtained by said endoscope; A memory for storing pre-calculated planning data obtained in a 3D virtual space constructed from the 3D image data; A processor is in communication with the memory and the display, the processor being operable to perform the following steps: Continuously linking the pre-computed planning data obtained in the 3D virtual space with the real space of the live surgery, and implementing image guidance based on the virtual endoscopic lumen view and the live video endoscopic view of the hollow organ system when the endoscope passes through the hollow organ system, thereby continuously tracking and guiding the endoscope to various diagnostic sites; indicating when a specific location on the display device is reached; as well as A surgical history is tracked indicating which sites on the display device have been visited and which sites have not been visited.