Auxiliary navigation method and system for bronchoscope and storage medium

Abstract

The invention relates to the technical field of image processing, and discloses an auxiliary navigation method and system for a bronchoscope and a storage medium. The method comprises the following steps: performing bronchial branch identification according to CT image data of a patient to obtain a three-dimensional bronchial tree semantic graph structure; performing cavity opening detection on the endoscope image through a lightweight detection network to obtain a detection result and confidence; performing trajectory association by adopting a multi-target tracking algorithm to obtain a tracking sequence; performing graph matching association based on the track sequence and the semantic identifier to obtain a matching result; and performing path planning in combination with a matching result to obtain real-time three-dimensional navigation guidance information. According to the invention, the navigation precision and the operation efficiency of bronchoscopy are improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an auxiliary navigation method, system and storage medium for bronchoscopy. Background Technology

[0002] Current bronchoscopy-assisted navigation methods primarily rely on electromagnetic navigation systems and traditional image-guided techniques. These methods involve placing a magnetic field generator outside the patient's body and using a bronchoscope equipped with sensors to position the lens in real time. This is combined with 3D reconstruction from preoperative CT images for path planning and location guidance. These traditional navigation systems provide basic spatial positioning capabilities, helping doctors reach the target lesion to some extent, and have already seen initial application in clinical practice, providing technical support for bronchoscopy.

[0003] However, traditional electromagnetic navigation systems rely on external hardware, which is costly and complex to operate, and electromagnetic interference can affect positioning accuracy. Secondly, existing image matching methods are mainly based on simple feature point matching or template matching, lacking targeted processing of the special visual features of bronchial endoscopy images, and are prone to matching failures under rapid lens movement and complex branching structures. Thirdly, existing navigation systems lack intelligent path planning and real-time tracking capabilities, and cannot provide the optimal navigation path based on bronchial anatomy and operational constraints.

[0004] The lack of semantic modeling of CT image data makes it impossible to establish a structured anatomical reference that includes branching levels and topological relationships, thus affecting the accuracy of subsequent matching. In the endoscopic image processing stage, the lack of a lightweight detection network specifically optimized for bronchial lumen features leads to insufficient target detection accuracy and poor real-time performance. In the target tracking stage, the lack of a multi-target tracking algorithm adapted to the rapid movement of the bronchoscope causes target loss and trajectory interruption. Finally, in the location matching and path planning stages, the lack of graph matching algorithms and intelligent path planning methods that consider the anatomical characteristics of the bronchi prevents accurate spatial positioning and optimal navigation guidance. Summary of the Invention

[0005] This application provides an auxiliary navigation method, system, and storage medium for bronchoscopy, addressing the shortcomings of existing bronchoscopy navigation technologies, such as the lack of intelligent matching and path planning capabilities based on purely visual information, reliance on external hardware, and insufficient navigation accuracy. By establishing a semantic bronchial tree structure and a multi-algorithm fusion visual navigation system, the navigation accuracy and operational efficiency of bronchoscopy are improved.

[0006] In a first aspect, this application provides an auxiliary navigation method for bronchoscopy, the auxiliary navigation method for bronchoscopy comprising:

[0007] Based on the patient's CT image data, bronchial branch identification is performed through CT imaging to obtain a three-dimensional bronchial tree semantic graph structure and semantic identifiers of branch nodes.

[0008] Based on the bronchoscopic images, a lightweight detection network is used to perform target detection processing of the bronchial lumen, and the bronchial lumen detection results and confidence scores are obtained.

[0009] The bronchial lumen detection results are processed by a multi-target tracking algorithm to obtain the lumen tracking trajectory sequence and motion state information.

[0010] Based on the cavity tracking trajectory sequence and the semantic identifier of the branch node, a graph matching algorithm is used to associate the detection-airway matching result and the matching confidence.

[0011] Based on the detection-airway matching results and the motion state information, path planning is performed to obtain real-time three-dimensional navigation guidance information and spatial positioning coordinates.

[0012] Optionally, the step of performing bronchial branch identification processing based on patient CT image data through CT imaging to obtain a three-dimensional bronchial tree semantic map structure and branch node semantic identifiers includes:

[0013] Adaptive threshold segmentation was performed on the patient's CT image data to obtain the bronchial lumen boundary contour and background separation results;

[0014] The bronchial cavity boundary contour is input into morphological opening and closing operations for noise reduction processing to obtain a continuous bronchial cavity structure.

[0015] Based on the bronchial cavity structure, a region growing algorithm is used to identify the branch hierarchy, resulting in a branch sequence from the first-level main bronchus to the eighth-level terminal bronchus.

[0016] Each bronchial node is labeled with attributes according to the branch sequence to obtain node attribute information including spatial coordinates, branch level, diameter parameters and connection relationships;

[0017] The node attribute information is organized and stored according to a tree data structure to obtain a three-dimensional bronchial tree semantic graph structure and branch node semantic identifiers with unique semantic identifiers.

[0018] Optionally, the step of performing bronchial orifice target detection processing based on bronchial endoscope images through a lightweight detection network to obtain bronchial orifice detection results and confidence scores includes:

[0019] The bronchoscope images were normalized to obtain standardized color image data of 512×512 pixels.

[0020] The standardized color image data is input into the improved YOLO architecture for feature extraction processing to obtain a multi-scale feature map containing sixteen convolutional layers and four pooling layers.

[0021] Based on the multi-scale feature map, a depthwise separable convolutional structure is used to perform cavity shape recognition processing to obtain circular and elliptical cavity candidate regions;

[0022] Based on the candidate cavity region, a multi-scale feature fusion algorithm is used to perform target classification processing to obtain cavity type identifier and bounding box coordinate information;

[0023] The bounding box coordinate information is combined with the lumen type identifier to perform confidence calculation, resulting in bronchial lumen detection results and confidence scores that include lumen location, size, and classification.

[0024] Optionally, the step of using a multi-target tracking algorithm to perform trajectory association processing on the bronchial orifice detection results to obtain the orifice tracking trajectory sequence and motion state information includes:

[0025] Based on the bronchial lumen detection results, an independent tracking trajectory is established for each detected lumen to obtain trajectory status information including lumen position coordinates, movement speed, appearance features and existence time.

[0026] The trajectory state information is input into a Kalman filter for motion prediction processing to obtain the predicted coordinates and velocity components of the cavity position in the next frame.

[0027] Based on the predicted coordinates and the detection results of the current frame, the IoU distance and appearance similarity are calculated to obtain the cost matrix between the predicted position and the detection position;

[0028] Based on the cost matrix, the Hungarian algorithm is used to perform optimal matching and solution processing to obtain the association pairing results between the cavity detection results and the tracking trajectory;

[0029] The association pairing results are processed for state update according to the trajectory management strategy to obtain the cavity tracking trajectory sequence and motion state information containing tracking ID, position sequence and motion vector.

[0030] Optionally, the step of performing graph matching algorithm association processing based on the cavity tracking trajectory sequence and the semantic identifier of the branch node to obtain the detection-airway matching result and matching confidence includes:

[0031] Based on the cavity tracking trajectory sequence, the spatial distribution pattern of cavities within the current field of view is analyzed and processed to obtain the distribution characteristic parameters of the number of cavities, their relative positional relationship, and their size ratio.

[0032] The distribution feature parameters and the semantic identifiers of the branch nodes are used to perform candidate region search processing to obtain a set of candidate branch regions with similar distribution patterns;

[0033] Based on the candidate branch region set, geometric matching degree and topological consistency verification are calculated to obtain the similarity value between the visual observation result and the semantic graph node.

[0034] Based on the similarity values, a dynamic programming algorithm is used to solve for the optimal association, resulting in the optimal association combination that satisfies the one-to-one matching constraint.

[0035] The optimal association combination is combined with a confidence assessment mechanism for matching verification to obtain the detection-airway matching result and matching confidence, which includes node ID, matching confidence and branch level information.

[0036] Optionally, the step of performing candidate region search processing on the distribution feature parameters and the semantic identifier of the branch nodes to obtain a set of candidate branch regions with similar distribution patterns includes:

[0037] Based on the aforementioned distribution characteristic parameters, geometric feature calculations are performed on the relative angles and distance ratios between the cavities to obtain a geometric description vector of the spatial distribution of the cavities.

[0038] The geometric description vector is compared with the spatial coordinates in the semantic identifier of the branch node to obtain the matching degree score of the geometric relationship between the nodes.

[0039] Based on the matching score, a hierarchical search process is performed on the three-dimensional bronchial tree semantic graph structure to obtain a set of candidate nodes that meet the geometric similarity threshold.

[0040] Based on the candidate node set, the connection relationship and branch level are verified by topological structure to obtain a branch region consistent with the observed cavity topology.

[0041] The branch regions are sorted and filtered according to their similarity scores to obtain a set of candidate branch regions containing highly matched nodes and neighborhood information.

[0042] Optionally, the step of performing path planning processing based on the detection-airway matching result and the motion state information to obtain real-time three-dimensional navigation guidance information and spatial positioning coordinates includes:

[0043] Based on the detection-airway matching results, the current lens position and target lesion position are determined, and the spatial coordinate information of the starting node and the target node is obtained.

[0044] The spatial coordinate information is combined with the motion state information to perform A* search algorithm path calculation to obtain the optimal path sequence from the current node to the target node;

[0045] Based on the optimal path sequence, a safety assessment is performed on the bronchial diameter constraint and bending angle limitation to obtain a feasible path scheme that meets the operational safety requirements.

[0046] Based on the feasible path scheme, a three-dimensional visualization display and distance and angle measurement processing are generated to obtain navigation display information containing three-dimensional graphics and spatial references;

[0047] The navigation display information is combined with the current position deviation to generate operation instructions, resulting in real-time 3D navigation guidance information and spatial positioning coordinates that include position coordinates, path instructions, and visual images.

[0048] Secondly, this application provides an auxiliary navigation system for bronchoscopy, the auxiliary navigation system for bronchoscopy comprising:

[0049] The identification module is used to identify bronchial branches based on the patient's CT image data through CT imaging, and to obtain a three-dimensional bronchial tree semantic graph structure and semantic identifiers of branch nodes.

[0050] The detection module is used to perform target detection processing of bronchial lumen based on bronchoscope images using a lightweight detection network, and obtain bronchial lumen detection results and confidence scores.

[0051] The association module is used to perform trajectory association processing on the bronchial lumen detection results using a multi-target tracking algorithm to obtain the lumen tracking trajectory sequence and motion state information;

[0052] The matching module is used to perform graph matching algorithm association processing based on the cavity tracking trajectory sequence and the semantic identifier of the branch node to obtain the detection-airway matching result and the matching confidence.

[0053] The planning module is used to perform path planning processing based on the detection-airway matching results and the motion state information to obtain real-time three-dimensional navigation guidance information and spatial positioning coordinates.

[0054] Thirdly, an auxiliary navigation device for bronchoscopy is provided, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the auxiliary navigation device for bronchoscopy to perform the aforementioned auxiliary navigation method for bronchoscopy.

[0055] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the aforementioned assisted navigation method for bronchoscopy.

[0056] The technical solution provided in this application identifies bronchial branches through CT imaging based on patient CT image data, obtaining a three-dimensional bronchial tree semantic graph structure and semantic labels for branch nodes. This solves the problem of lacking systematic modeling of bronchial anatomy in existing technologies. The semantic graph structure not only contains geometric information but also integrates branch hierarchy and connection relationships, providing a structured reference benchmark for subsequent image matching. Based on bronchoscope images, a lightweight detection network is used for lumen target detection, obtaining bronchial lumen detection results and confidence scores. Compared to traditional threshold detection methods, the lightweight detection network is specifically optimized for the visual features of bronchoscope images, accurately identifying the shape features of circular and elliptical lumens, significantly improving detection accuracy and real-time performance. The bronchial lumen detection results are then processed using a multi-target tracking algorithm for trajectory association, obtaining lumen tracking trajectory sequences and motion state information. This effectively solves the problem of target loss caused by rapid bronchoscope movement. Through motion prediction using a Kalman filter and optimal matching using the Hungarian algorithm, a stable and continuous target tracking capability is established, providing a reliable data foundation for subsequent position association.

[0057] A graph matching algorithm, based on the correlation between the bronchial orifice tracking trajectory sequence and the semantic identifiers of branch nodes, addresses the challenge of matching bronchial endoscopy observations with CT anatomical structures. Through a dual verification mechanism of geometric matching degree and topological consistency, it achieves cross-patient generalization capability without individualized training. The graph matching algorithm considers the special geometric patterns and anatomical constraints of bronchial bifurcation, exhibiting higher matching accuracy and stronger robustness compared to general image registration methods. An intelligent navigation algorithm, combining detection-airway matching results with motion state information for path planning, fully utilizes the hierarchical structure of the bronchial tree and the physical constraints of lens motion. The A* search algorithm optimizes path length and operational difficulty while considering bronchial diameter limitations and bending angle constraints. The generated real-time 3D navigation guidance information intuitively displays the optimal path from the current location to the target lesion, significantly improving the navigation accuracy and operational efficiency of bronchoscopy and providing physicians with spatial positioning and path guidance. Attached Figure Description

[0058] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of one embodiment of the assisted navigation method for bronchoscopy in this application.

[0060] Figure 2 This is a schematic diagram of one embodiment of the auxiliary navigation system for bronchoscopy in this application.

[0061] Figure 3 This is a schematic block diagram of the structure of the auxiliary navigation device for bronchoscope in an embodiment of the present invention. Detailed Implementation

[0062] This application provides an auxiliary navigation method, system, and storage medium for bronchoscopy. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0063] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the assisted navigation method for bronchoscopy in this application includes:

[0064] Step S101: Based on the patient's CT image data, bronchial branch identification is performed through CT imaging to obtain a three-dimensional bronchial tree semantic graph structure and semantic identifiers of branch nodes.

[0065] Step S102: Based on the bronchoscope endoscope image, perform lumen target detection processing through a lightweight detection network to obtain the bronchial lumen detection result and confidence score;

[0066] Step S103: The bronchial lumen detection results are processed by a multi-target tracking algorithm to obtain the lumen tracking trajectory sequence and motion state information;

[0067] Step S104: Based on the cavity tracking trajectory sequence and the semantic identifier of the branch node, a graph matching algorithm is used to associate the results to obtain the detection-airway matching result and the matching confidence.

[0068] Step S105: Based on the detection-airway matching results and motion state information, perform path planning processing to obtain real-time three-dimensional navigation guidance information and spatial positioning coordinates.

[0069] It is understood that the executing entity of this application can be an auxiliary navigation system for bronchoscopy, a terminal, or a server; no specific limitation is made here. This application's embodiments use a server as an example for illustration.

[0070] Specifically, the construction of the three-dimensional bronchial tree semantic graph structure is based on the depth processing of patient CT image data. The adaptive threshold segmentation algorithm first classifies the pixels in the CT image into binary categories according to their gray values. The pixel gray values ​​of the bronchial lumen are usually lower than those of the surrounding tissues, so they are identified as foreground regions. The background separation result generates a clear bronchial lumen boundary contour. The morphological opening operation eliminates small noise points and broken connections through an operation of erosion followed by dilation. The closing operation fills the empty areas inside the lumen through an operation of dilation followed by erosion. The region growing algorithm expands outward from the seed point of the tracheal trunk to identify connected bronchial regions. The algorithm labels the bronchial branches at each level layer by layer according to the connectivity and gray value similarity between pixels. The semantic identifier of the branch node includes the three-dimensional spatial coordinates of each node, its hierarchical position in the bronchial tree, the size of the lumen diameter, and the topological connection relationship with adjacent nodes. The tree data structure organizes all nodes hierarchically according to the parent-child relationship, with the root node representing the tracheal trunk and the leaf nodes representing the terminal bronchioles.

[0071] The lightweight detection network is optimized for the unique visual features of bronchoscopic images. In endoscopic images, the bronchial lumen appears as a dark, circular or elliptical area surrounded by a pink bronchial wall. Standardized color image data unifies input images of different resolutions to 512×512 pixels to ensure network input consistency. The improved YOLO architecture uses depthwise separable convolutions instead of traditional convolutional layers to reduce computational parameters. Sixteen convolutional layers extract visual features from edges to complex shapes layer by layer. Four pooling layers reduce feature map resolution while expanding the receptive field. Multi-scale feature maps contain lumen feature information at different resolutions to adapt to lumen targets of various sizes. The depthwise separable convolutional structure decomposes standard convolution into two steps: depthwise convolution and pointwise convolution to reduce computational complexity. The lumen candidate region is removed by non-maximum suppression algorithm to retain the optimal result. The confidence score reflects the network's evaluation of the accuracy of the detection result; a higher value indicates a more reliable detection.

[0072] The multi-target tracking algorithm solves the problem of target loss at the bronchus opening caused by rapid movement of the bronchoscope. The trajectory state information uses a state vector to record the motion characteristics of each opening. The Kalman filter predicts the position of the opening in the next frame image based on a constant velocity motion model. The prediction process estimates the future state based on the current position and velocity while considering the influence of motion noise. The IoU distance calculates the overlap ratio between the predicted box and the detection box to measure the degree of position matching. The appearance similarity is evaluated by comparing the color histogram and texture features of the opening region to assess appearance consistency. The cost matrix organizes the association costs between all predicted targets and detection targets into a matrix form. The Hungarian algorithm solves the bipartite graph optimal matching problem to find the association scheme with the minimum total cost. The trajectory management strategy includes the creation of new trajectories, the updating of existing trajectories, the deletion of lost trajectories, and the recovery of reappearing trajectories. The motion state information includes the position change trajectory of the opening, the motion velocity vector, and the motion direction angle.

[0073] The graph matching algorithm establishes a correspondence between endoscopic observations and CT anatomical structures. Distributed feature parameters describe the spatial arrangement of multiple cavities within the current field of view. Geometric description vectors quantify the relative positional relationships between cavities, including angular differences and distance ratios. Spatial coordinates in the semantic identifiers of branch nodes provide precise positional information of each branch in the CT anatomical structure. Similarity comparison assesses the matching probability by calculating the geometric differences between the observation pattern and the anatomical pattern. Hierarchical search searches for matching candidates at different levels of the bronchial tree to avoid the computational overhead of global search. Topological structure verification checks whether the branch connectivity of the candidate region is consistent with the observation results. The dynamic programming algorithm solves the globally optimal matching scheme through state transitions and optimal substructures. The matching confidence score combines geometric similarity and topological consistency to give the credibility of the matching results.

[0074] The path planning process generates navigation guidance information. The A* search algorithm searches for the shortest path from the current position to the target position in the bronchial tree semantic graph. The algorithm uses a heuristic function to estimate the distance from the current node to the target node to guide the search direction. The path cost function comprehensively considers the path length, bronchial diameter, and curvature to select the most suitable path for the operation. Safety assessment ensures that the planned path meets the physical constraints of bronchoscopy. 3D visualization transforms abstract path information into intuitive stereoscopic images to help doctors understand spatial relationships. Distance and angle measurements calculate precise values ​​between the current position and the target position to provide quantitative reference for the operation. Operation command generation outputs specific lens movement suggestions based on the path planning results, including advance distance and rotation angle.

[0075] In one specific embodiment, the process of performing step S101 may specifically include the following steps:

[0076] Adaptive threshold segmentation was performed on the patient's CT image data to obtain the bronchial lumen boundary contour and background separation results;

[0077] The bronchial cavity boundary contour is input into morphological opening and closing operations for noise reduction processing to obtain a continuous bronchial cavity structure.

[0078] Based on the bronchial cavity structure, a region growing algorithm is used to identify the branch hierarchy, resulting in a branch sequence from the first-level main bronchus to the eighth-level terminal bronchus.

[0079] Each bronchial node is labeled with attributes based on the branch sequence to obtain node attribute information including spatial coordinates, branch level, diameter parameters and connection relationships;

[0080] The node attribute information is organized and stored according to a tree data structure to obtain a three-dimensional bronchial tree semantic graph structure with unique semantic identifiers and semantic identifiers of branch nodes.

[0081] Specifically, the adaptive threshold segmentation process intelligently segments the bronchi and surrounding tissues based on the density difference in the patient's CT image data. The adaptive threshold algorithm dynamically adjusts the segmentation threshold according to the statistical characteristics of the local image region to avoid oversegmentation or undersegmentation caused by a globally fixed threshold. The bronchial cavity appears as a low-density region in CT images, with gray values ​​typically between -1000 and -200 HU. The algorithm calculates the mean and standard deviation within the neighborhood window of each pixel as the basis for local threshold judgment. When the pixel gray value is lower than the local threshold, it is marked as a foreground pixel belonging to the bronchial cavity, and pixels with a gray value higher than the threshold are marked as background pixels representing the surrounding lung tissue. Boundary contour extraction uses an edge detection algorithm to identify the boundary between the foreground and background to generate the outer contour of the bronchial cavity. The background separation result generates a binary image in which the bronchial region is white and the background region is black for easy subsequent processing. Morphological opening and closing operations address noise and discontinuities in the segmentation results. The opening operation first erodes the image to shrink the foreground region and eliminate small noise points and burrs, then performs a dilation operation to restore the original size of the target. The closing operation uses the reverse order, first dilating and then eroding to fill the voids inside the target and connect broken parts. The choice of structuring element affects the effect of morphological operations, and circular or cross-shaped structuring elements are usually used. The erosion operation performs a logical AND operation between the structuring element and the image, and only pixels completely contained in the foreground are retained. The dilation operation performs a logical OR operation to expand the boundary of the foreground region. The continuous bronchial cavity structure eliminates artifacts and breaks generated during the segmentation process and ensures the connectivity of the bronchi.

[0082] The region growing algorithm identifies the branching hierarchy of the bronchi based on pixel connectivity and similarity criteria. Starting from a manually or automatically selected seed point, the algorithm expands outwards to search for neighboring pixels that meet the growth conditions. These conditions include pixel grayscale values ​​within a preset range and connectivity with the seed point. The first-level main bronchus is identified from the bifurcation point, with the left and right main bronchi as root nodes. The algorithm traces the branch paths along the bronchus, recording the position of each bifurcation point. The second-level lobar bronchus continues to expand downwards from the bifurcation point of the main bronchus. Branch identification uses a skeleton extraction algorithm to simplify the three-dimensional bronchial structure to a centerline representation. Bifurcation points on the centerline are marked as branch nodes, and connecting line segments represent bronchial segments. The hierarchy numbering increases from coarse to fine, with the eighth-level terminal bronchus being the finest terminal bronchioles. The branch sequence records the complete topological structure of the bronchial tree, including all branch nodes and connections. The attribute annotation process assigns detailed geometric and topological attributes to each bronchial node. The spatial coordinates use a three-dimensional Cartesian coordinate system to record the precise position of the node in the CT image space. The branch level indicates the hierarchical depth of the node in the bronchial tree. The diameter parameter is obtained by measuring the maximum inner diameter of the bronchial cross-section at the node. The connection relationship records the topological connections between the node and its parent and child nodes, including the branch angle and the number of branches. The node attribute information also includes geometric features such as the length, orientation angle, and radius of curvature of the bronchial segment.

[0083] The tree-like data structure organizes all nodes according to a hierarchical parent-child relationship. The root node represents the main trachea and has the highest priority. Each internal node represents a branch point connecting the upper and lower branches. Leaf nodes represent terminal bronchioles without child branches. The tree structure uses an adjacency list or adjacency matrix to represent the connection relationship between nodes. A unique semantic identifier is assigned a unique code to each node to facilitate fast retrieval and matching. The identifier coding rules include branch level, branch sequence number, and spatial location information. The three-dimensional bronchial tree semantic graph structure integrates geometric and semantic information to generate a bronchial anatomical model. The semantic identifiers of branch nodes establish a bridge between endoscopic observation and CT anatomical structures.

[0084] In one specific embodiment, the process of performing step S102 may specifically include the following steps:

[0085] The bronchoscope images were normalized to obtain standardized color image data of 512×512 pixels.

[0086] Standardized color image data is input into the improved YOLO architecture for feature extraction, resulting in a multi-scale feature map containing sixteen convolutional layers and four pooling layers.

[0087] Based on multi-scale feature maps, a depthwise separable convolutional structure is used to perform cavity shape recognition processing to obtain circular and elliptical cavity candidate regions;

[0088] Based on the candidate cavity region, a multi-scale feature fusion algorithm is used to classify the target to obtain the cavity type identifier and bounding box coordinate information.

[0089] The bounding box coordinate information is combined with the lumen type identifier to calculate the confidence score, resulting in bronchial lumen detection results and confidence scores that include the lumen location, size, and classification.

[0090] Specifically, the original resolution of bronchoscopic images varies depending on the equipment model and acquisition settings. Normalization processing uses bilinear interpolation to resample the input image to a standard size of 512×512 pixels. Bilinear interpolation determines the gray value of the new pixel by calculating the weighted average of the four neighboring pixels around the target pixel position. The weighting coefficient is calculated inversely proportional to the distance between the target position and the neighboring pixels. Standardized color image data maintains the color information of the RGB three channels while unifying the spatial resolution. The pixel value range is normalized to between zero and one to facilitate numerical calculation by the neural network. The normalization formula divides the original pixel value by the maximum possible value of 255 to obtain a floating-point representation. The improved YOLO architecture optimizes the structure of the original YOLO detection framework for bronchial lumen features. Feature extraction uses a convolutional neural network to abstract image features layer by layer. Sixteen convolutional layers contain convolutional kernels of different sizes to extract local and global features respectively. The first eight convolutional layers use smaller convolutional kernels to extract fine-grained features such as edges and textures, while the last eight convolutional layers use larger convolutional kernels to integrate complex shape features. Four pooling layers are located after the fourth, eighth, twelfth, and sixteenth convolutional layers, respectively. The pooling operation reduces the spatial dimension of the feature map by selecting the maximum value or calculating the average value. The multi-scale feature map contains feature information at different resolution levels to adapt to lumen targets of various sizes.

[0091] The depthwise separable convolutional structure decomposes the standard convolution operation into two consecutive steps: depthwise convolution and pointwise convolution. Depthwise convolution independently performs spatial convolution operations on each input channel to extract spatial features within the channel. Pointwise convolution uses a 1x1 convolution kernel to linearly combine the output of depthwise convolution between channels. This decomposition method significantly reduces the number of computational parameters while maintaining feature extraction capabilities. The lumen shape recognition is specifically designed for the circular main lumen and elliptical lateral lumen at the bifurcation of the bronchus. Features of the circular lumen include central symmetry, smooth edges, and dark internal regions. Features of the elliptical lumen include the ratio of major and minor axes, tilt angle, and gradient boundaries. Candidate region generation adopts an anchor box mechanism to pre-set candidate boxes of different sizes and aspect ratios on the feature map. Each candidate box is adjusted through regression to obtain the position of the lumen boundary. The multi-scale feature fusion algorithm integrates information from feature maps at different levels to enhance the accuracy of target detection. The fusion process performs upsampling and downsampling operations on high-level semantic features and low-level detail features. Upsampling uses bilinear interpolation or transposed convolution to enlarge low-resolution feature maps to high resolution, while downsampling uses pooling or strided convolution to shrink high-resolution feature maps to low resolution. The feature pyramid network structure achieves effective integration of features at different scales. The lumen type identifier distinguishes between three categories: main bronchial lumen, branch lumen, and abnormal lumen. The bounding box coordinate information includes the coordinates of the lumen center point and the width and height parameters of the rectangular bounding box.

[0092] The confidence score calculation evaluates the reliability of the detection results by combining the probability of target presence and classification accuracy. The probability of target presence indicates the likelihood that a lumen target actually exists within the bounding box, while the classification accuracy reflects the correctness of the lumen type judgment. The confidence score maps the network output value to a probability range of zero to one using the sigmoid activation function. The closer the score is to one, the more reliable the detection result. The lumen location information includes the horizontal and vertical coordinates of the center point in the image coordinate system, the size information records the pixel width and height of the lumen region, and the classification category identifies the specific type and medical significance of the lumen. The bronchial lumen detection results integrate information from four dimensions: location, size, type, and confidence score to generate the detection output.

[0093] In one specific embodiment, the process of executing step S103 may specifically include the following steps:

[0094] Based on the bronchial lumen detection results, an independent tracking trajectory is established for each detected lumen to obtain trajectory status information including lumen position coordinates, movement speed, appearance features and existence time;

[0095] The trajectory state information is input into the Kalman filter for motion prediction processing to obtain the predicted coordinates and velocity components of the cavity position in the next frame.

[0096] Based on the predicted coordinates and the detection results of the current frame, the IoU distance and appearance similarity are calculated to obtain the cost matrix between the predicted position and the detection position;

[0097] The optimal matching solution is obtained by using the Hungarian algorithm based on the cost matrix, and the correlation pairing results between the cavity detection results and the tracking trajectory are obtained.

[0098] The association pairing results are processed for state update according to the trajectory management strategy to obtain the cavity tracking trajectory sequence and motion state information containing tracking ID, position sequence and motion vector.

[0099] Specifically, independent tracking trajectory establishment creates a dedicated tracking object for each detected bronchial lumen. The trajectory state information uses a state vector to record the multidimensional dynamic features of the lumen. The lumen position coordinates are contained in the horizontal and vertical coordinate values ​​in the image coordinate system. The motion velocity is obtained by calculating the position changes between consecutive frames to obtain the horizontal and vertical velocity components. The appearance features are extracted from the color histogram, texture descriptor, and shape parameters in the lumen region as visual identifiers. The existence time records the continuous tracking duration of the lumen target from its first detection to the current frame. The trajectory initialization requires that the newly detected target maintain stable detection for three consecutive frames before a formal trajectory is established to avoid interference from false targets. The Kalman filter predicts the cavity motion based on a linear dynamic model. The state vector contains four components: position coordinates, velocity coordinates, and so on. The state transition matrix describes the position update law of the target under the assumption of constant velocity motion. The observation matrix maps the internal state to the observable position measurement. The process noise covariance matrix models the uncertainty of the motion model, and the observation noise covariance matrix describes the measurement error of the detection result. The prediction step calculates the prior state of the current frame based on the state estimate of the previous frame and the motion model. The update step combines the observation results of the current frame to correct the prior estimate and obtain the posterior state. The predicted coordinates represent the most likely position of the cavity in the current frame estimated by the filter, and the velocity component reflects the direction and speed of the cavity motion.

[0100] The Intersection over Union (IoU) distance calculates the degree of overlap between the predicted bounding box and the detected bounding box. IoU is defined as the ratio of the area of ​​the intersection to the area of ​​the union of two rectangular regions. The intersection area is calculated by multiplying the width and height of the overlapping rectangles, and the union area is equal to the sum of the areas of the two rectangles minus the intersection area. IoU values ​​range from zero to one; a higher value indicates a higher degree of overlap and a greater probability of matching. Appearance similarity compares the historical appearance features of the predicted trajectory with the appearance features of the current detection result. Color histogram similarity measures the color distribution difference by calculating the Bach distance of the histograms of each RGB channel. Texture similarity uses the Local Binary Mode Descriptor to compare the texture characteristics of the cavity region. Shape similarity analyzes the geometric features of the cavity boundary, including roundness, aspect ratio, and edge smoothness. The cost matrix organizes the association costs between all predicted trajectories and the current detection result into a two-dimensional matrix. Rows represent predicted trajectories, and columns represent detection results. The matrix element values ​​are calculated by combining IoU distance and appearance similarity; a lower cost value indicates a higher probability of association. The Hungarian algorithm solves the maximum weighted matching problem in a bipartite graph by finding the globally optimal association scheme with the minimum total cost. The algorithm uses an augmenting path search strategy to gradually build the optimal match. The augmenting path starts from the unmatched predicted trajectory and finds an alternating path to the unmatched detection result. Matched edges and unmatched edges appear alternately on the path. After finding an augmenting path, the matching state is flipped along the path to increase the number of matches. This process is repeated until no new augmenting path can be found, resulting in the maximum match. The association pairing results determine the best detection target corresponding to each predicted trajectory. Unmatched predicted trajectories are marked as occluded or disappeared, and unmatched detection results are used as new target candidates. The trajectory management strategy controls the lifecycle of tracking trajectories, including four state transitions: creation, confirmation, update, and deletion. Newly detected targets first enter the trial state and require stable detection for multiple consecutive frames before transitioning to the confirmation state. Confirmed trajectories update their state information, including position, velocity, and appearance features, upon successful matching. Occluded trajectories lose matching within a short period but retain their trajectory information, awaiting re-association. Deleted trajectories are removed from the tracking list after prolonged periods without matching to release computational resources. State updates use an exponential moving average method to smooth changes in trajectory parameters. Tracking IDs assign unique identifiers to each trajectory for easy identification and retrieval. Position sequences record the complete motion path of the trajectory in the time dimension. Motion vectors describe the velocity direction and acceleration changes of the trajectory. Cavity tracking trajectory sequences generate multi-target tracking results. Motion state information includes real-time motion parameters of all active trajectories.

[0101] In one specific embodiment, the process of executing step S104 may specifically include the following steps:

[0102] Based on the cavity tracking trajectory sequence, the spatial distribution pattern of cavities within the current field of view is analyzed and processed to obtain the distribution characteristic parameters of the number of cavities, their relative positional relationship, and their size ratio.

[0103] The distribution feature parameters and the semantic identifiers of the branch nodes are used to perform candidate region search processing to obtain a set of candidate branch regions with similar distribution patterns;

[0104] Based on the candidate branch region set, geometric matching degree and topological consistency verification are calculated to obtain the similarity value between the visual observation result and the semantic graph node;

[0105] Based on the similarity values, a dynamic programming algorithm is used to solve for the optimal association, and the optimal association combination that satisfies the one-to-one matching constraint is obtained.

[0106] The optimal association combination is combined with a confidence assessment mechanism for matching verification to obtain the detection-airway matching result and matching confidence, which includes node ID, matching confidence and branch level information.

[0107] Specifically, the geometric arrangement features of multiple cavities within the current field of view are extracted based on the cavity tracking trajectory sequence. The number of cavities is counted to determine the total number of cavity targets corresponding to all active tracking trajectories in the current frame. The relative position relationship is calculated by determining the Euclidean distance and the angle between the center points of each cavity. Distance normalization is performed to eliminate the influence of image scale, and the maximum distance is used as the benchmark for scaling. The angle is calculated by measuring the angle between the lines connecting cavities clockwise with the horizontal axis of the image as the benchmark. The size ratio analysis shows that the area ratio and diameter ratio of different cavities reflect the hierarchical relationship of the bronchial branches. The distribution feature parameters are stored in vector form, including the number of cavities, distance matrix, angle matrix, and size ratio matrix. The dimension of the feature vector is dynamically adjusted according to the number of cavities to adapt to different branch complexities. Candidate region search identifies anatomical locations matching the observed pattern within the 3D bronchial tree semantic graph. The search strategy employs a hierarchical traversal, expanding progressively from coarse to fine branches to avoid the computational overhead of a global search. Branch node semantic identifiers provide spatial coordinates, branch level, and connectivity information for each anatomical location. Similarity distribution pattern judgment is based on an initial screening of the number of luminal openings; candidate regions must have the same or similar number of branches as the observed results. Spatial geometric comparison calculates the distance ratio and angular relationship between branches in the semantic graph. The candidate branch region set contains all anatomical locations that passed the initial screening and their neighborhood information. Geometric matching calculates the geometric similarity between the observed luminal opening distribution and the candidate anatomical locations. Matching evaluation includes three dimensions: distance similarity, angular similarity, and size similarity. Distance similarity is calculated by comparing corresponding elements of the normalized distance matrix to determine the root mean square error. Angle similarity analyzes the difference between the angles of the lines connecting luminal openings and the angles of the anatomical branches. Size similarity compares the consistency between the proportion of luminal opening area and the proportion of bronchial diameter. Geometric matching uses a weighted average method to synthesize the similarity scores of the three dimensions, with weighting coefficients determined based on the stability and discriminative power of each dimension's features. The topology consistency verification checks whether the observed cavity connection relationships conform to the topology of the anatomical branches. The verification process constructs an adjacency graph of the observed cavities and a branch connection graph of the candidate positions. The graph matching algorithm compares the structural isomorphism of the two graphs. The node correspondence must maintain the edge connection attribute. Candidate positions with inconsistent topology are excluded to reduce the risk of mismatch. The similarity value combines geometric matching degree and topology consistency to obtain a comprehensive score. The score ranges from zero to one. The higher the value, the better the matching quality.

[0108] Dynamic programming is used to find the optimal combination of lumbar cavities and anatomical nodes. The algorithm's state is defined as an intermediate result where some lumbar cavities have been assigned to some nodes. State transitions consider all possible choices for assigning the next lumen to an available node. The transition cost equals the negative of the similarity score, allowing the algorithm to find the maximum sum of similarities. A one-to-one matching constraint requires that each lumen is associated with at most one node, and each node is associated with at most one lumen. The dynamic programming table records the optimal solution and corresponding allocation scheme in different states. The recursive relationship starts from an empty allocation state and gradually adds lumen allocations until all lumbar cavities are processed. The optimal combination of lumbar cavities is obtained by backtracking the optimal path in the dynamic programming table. The algorithm's complexity increases exponentially with the number of lumbar cavities and candidate nodes, but its scale is controllable in practical applications. A confidence assessment mechanism quantifies the reliability of the optimal combination of lumbar cavities. Evaluation indicators include matching similarity distribution, geometric consistency, and topological verification results. Similarity distribution analysis checks whether the similarity of each pair in the optimal match is evenly distributed. Geometric consistency checks whether the geometric transformation of the overall matching result is reasonable. Topological verification confirms that the connection relationship after matching conforms to the bronchial anatomy. The confidence calculation uses a Bayesian method to synthesize multiple evaluation indicators to obtain the overall credibility.

[0109] The matching verification process determines whether to accept the current matching result based on the confidence threshold. High-confidence matches are directly accepted and the final result is output. Medium-confidence matches are added to the candidate list for further verification. Low-confidence matches are rejected and trigger a relocation process. The node ID identifies the unique encoding of the successfully matched anatomical location in the semantic graph. The matching confidence quantifies the reliability of the matching result to guide subsequent decisions. The branch level information indicates the hierarchical depth of the current location in the bronchial tree. The detection-airway matching result establishes a precise correspondence between endoscopic observation and CT anatomical structure. The output format of the matching result includes a pairing list of lumen IDs and node IDs, the overall matching confidence, and the location hierarchy information.

[0110] In one specific embodiment, the process of performing candidate region search processing by combining the distribution feature parameters and the semantic identifiers of the branch nodes can specifically include the following steps:

[0111] Based on the distribution characteristic parameters, the relative angles and distance ratios between the cavities are calculated to obtain the geometric description vector of the spatial distribution of the cavities.

[0112] The geometric description vector is compared with the spatial coordinates in the semantic identifier of the branch node to obtain the matching degree score of the geometric relationship between the nodes.

[0113] Based on the matching score, a hierarchical search process is performed on the three-dimensional bronchial tree semantic graph structure to obtain a set of candidate nodes that meet the geometric similarity threshold.

[0114] Based on the candidate node set, the connection relationship and branch level are verified by topological structure processing to obtain the branch region consistent with the observed cavity topology.

[0115] The branch regions are sorted and filtered according to their similarity scores to obtain a set of candidate branch regions containing highly matched nodes and neighborhood information.

[0116] Specifically, the relative angle calculation uses a vector geometry method to establish a local coordinate system with the main cavity as the origin. The azimuth angles of other cavities are obtained by calculating the ratio of the difference between the horizontal and vertical coordinates using the arctangent function. The angle measurement range is from 0 to 360 degrees, with clockwise direction as the positive value. In the case of multiple cavities, an angle matrix is ​​constructed to record the relative angle relationship between any two cavities. The distance ratio calculation normalizes the Euclidean distance between cavities to eliminate the influence of image scale. The normalization method uses the maximum distance as the benchmark and divides all distances by the maximum value to obtain a ratio coefficient of zero to one. The distance matrix stores the normalized distance ratio between all cavity pairs. The geometric description vector integrates the angle information and distance information into a fixed-length feature vector. The vector dimension is determined according to the number of cavities and includes all independent angle and distance features. The arrangement order of the feature vectors is organized according to the lexicographical order of the cavity numbers to ensure the consistency of the calculation results at different times. Similarity comparison processing matches and evaluates the observed geometric description vectors with the spatial geometric relationships of branch nodes in the semantic graph. The semantic identifiers of branch nodes store the three-dimensional coordinate information of each anatomical location. Coordinate transformation projects the three-dimensional anatomical coordinates onto the two-dimensional image plane to simulate the observation perspective of the endoscope. The projection transformation considers the direction angle and field of view parameters of the lens. The geometric relationship between nodes is constructed by calculating the relative angle and distance ratio between branches to construct the anatomical geometric description vector. The vector comparison uses cosine similarity to measure the directional consistency of two high-dimensional vectors. The cosine similarity is calculated by dividing the inner product of the two vectors by the product of the vector magnitudes. The matching score ranges from negative one to positive one. The closer the value is to positive one, the more similar the geometric relationship.

[0117] The hierarchical search process employs a breadth-first strategy to find geometrically matching candidate locations in the three-dimensional bronchial tree semantic graph structure. The search starts from the tracheal trunk and expands to the lower-level branches layer by layer. Each layer of the search calculates the local geometric pattern formed by the current node and its sub-branches. A geometric similarity threshold of 0.7 is set to filter candidate locations with low matching degree. The threshold judgment is based on the comparison between the matching degree score and the set threshold. Nodes that meet the threshold condition are added to the candidate node set to continue searching to the lower level. Branch paths that do not meet the condition are pruned to reduce computational overhead. The search depth is limited to prevent over-expansion and is usually set to the two levels of branches above and below the current estimated location. The candidate node set records all anatomical locations that have passed the geometric screening and their local branch information. The size of the set reflects the richness of matching candidates. Too many candidates indicate insufficient geometric feature discrimination, while too few candidates may miss the correct location. Topological verification ensures that the branch connections of candidate nodes are consistent with the observed cavity connections. The verification process constructs an adjacency graph of the observed cavities to record which cavities have direct connections. The branch level checks whether candidate nodes are located at a reasonable anatomical level. The correspondence between the number of cavities and the number of branches must satisfy anatomical constraints. The connection verification compares the structural isomorphism between the cavity graph and the branch graph. Graph isomorphism judgment adopts two levels: node degree matching and edge connection pattern matching. Degree matching requires that corresponding nodes have the same number of adjacent nodes. Edge connection pattern matching ensures that the connection relationship between nodes is consistent in the two graphs. Branch regions with consistent topology are retained as candidate positions that conform to anatomical rules through verification screening.

[0118] The sorting and filtering process prioritizes verified branch regions based on their similarity scores. The sorting algorithm uses quicksort or mergesort to ensure computational efficiency. The similarity score is used as the sorting key; higher scores rank higher. The sorting results reflect the matching quality of candidate positions; higher-ranked positions correspond to more likely true anatomical locations. The filtering strategy retains the top-ranked candidates to avoid excessive interference from low-quality candidates in subsequent processing. The number of candidates to be filtered is typically three to five, depending on application requirements. High-matching nodes are those ranked high and whose similarity scores exceed a preset threshold. Neighborhood information includes the branch structure and connections around the candidate node, encompassing its parent, sibling, and child nodes. The candidate branch region set integrates high-quality candidate positions after sorting and filtering, along with their detailed anatomical information. The set structure uses a priority queue or ordered list for storage, facilitating subsequent matching decisions.

[0119] In one specific embodiment, the process of executing step S105 may specifically include the following steps:

[0120] Based on the detection-airway matching results, the current lens position and target lesion position are determined, and the spatial coordinate information of the starting node and the target node is obtained.

[0121] By combining spatial coordinate information with motion state information, the A* search algorithm is used to calculate the path and obtain the optimal path sequence from the current node to the target node.

[0122] Based on the optimal path sequence, a safety assessment is performed on the bronchial diameter constraint and bending angle limitation to obtain a feasible path scheme that meets the operational safety requirements.

[0123] Based on feasible path solutions, a 3D visualization display and distance and angle measurement processing are generated to obtain navigation display information containing 3D graphics and spatial references;

[0124] The navigation display information is combined with the current position deviation to generate operation instructions, resulting in real-time 3D navigation guidance information and spatial positioning coordinates that include position coordinates, path instructions, and visual images.

[0125] Specifically, the current camera position and target lesion position are determined based on node association information from the detection-airway matching results. The starting node determines the current anatomical position of the camera in the bronchial tree by matching the lumen-node with the highest confidence in the matching results. Node ID parsing extracts the unique identifier and hierarchical information of the matching node in the semantic graph. The target lesion position is determined by the anatomical area to be reached through pre-labeling by the doctor or imaging diagnosis. The target node selects the nearest bronchial branch node near the lesion as the navigation endpoint. Spatial coordinate information is extracted from the node attributes of the semantic graph, including the three-dimensional Cartesian coordinates of the starting node and the target node. The coordinate system adopts the standard anatomical coordinate system of CT images to ensure the accuracy and consistency of the position information. The A* search algorithm combines spatial coordinate information and motion state information to find the optimal navigation path. Motion state information provides the initial conditions for path planning, including the current direction of camera movement and velocity constraints. The search algorithm uses a heuristic function to estimate the cost from the current node to the target node. The heuristic function calculates the Euclidean distance as the shortest path length under ideal conditions. The actual cost function comprehensively considers path distance, bronchial diameter, and operational difficulty. An open list stores candidate nodes to be explored, sorted by total cost, while a closed list records explored nodes to avoid duplicate calculations. Path expansion extends gradually from the current node to adjacent branch nodes. The selection of adjacent nodes is based on the connectivity and branch direction of the bronchus. The optimal path sequence constructs a complete path from the target node to the starting node by backtracking the parent node pointer. Safety assessment ensures the planned route meets the physical constraints and medical safety requirements of bronchoscopy. Bronchial diameter constraints check whether the lumen diameter at each node of the route is sufficient to accommodate the bronchoscope. The lens diameter is determined based on the bronchoscope model used, typically three to six millimeters. Diameter comparison requires the bronchial inner diameter to be greater than the lens diameter plus a safety margin. The safety margin is set at 20-30% of the lens diameter to prevent jamming or damage. Bending angle limitations assess whether the angle changes at route bends exceed the lens's operational limits. The maximum bending angle of the bronchoscope is typically 120-180 degrees. Angle calculation is done by measuring the vector angle between three consecutive nodes to determine the degree of bending. Bending sections exceeding the limit are marked as high-risk areas requiring special handling. Feasible route options select the route with the lowest cost while meeting all constraints. In cases with multiple route candidates, a comprehensive scoring system is used to rank and select the optimal solution. The scoring criteria include route length, operational difficulty, and time cost.

[0126] 3D visualization transforms abstract path planning results into intuitive stereoscopic images to help doctors understand spatial relationships. The visualization rendering uses a 3D graphics engine to construct a stereoscopic model of the bronchial tree. The model texture and lighting settings simulate the effect of real endoscopic observation. The current position and target position are highlighted with markers of different colors. The planned path is represented by a highlighted curve to show the complete trajectory from the starting point to the end point. Distance measurement calculates the cumulative distance along the path to provide quantitative spatial reference. The measurement unit uses millimeters to ensure the accuracy requirements of clinical applications. Angle measurement displays the bending angle and turning direction of key turning points. The stereoscopic graphics support interactive operations such as rotation and zoom to facilitate multi-angle observation. Spatial reference includes anatomical landmarks and distance scales to help doctors establish spatial concepts. The navigation display integrates visualization images, measurement data, and operation prompts to generate a navigation interface.

[0127] The operation command generation provides specific lens operation suggestions based on navigation display information and current position deviation. The current position deviation is calculated by comparing the difference between the actual lens position and the planned path. Deviation measurement includes two dimensions: position offset and direction deflection. Position offset uses Euclidean distance to measure the vertical distance between the lens center and the planned path. Direction deflection calculates the angular difference between the lens orientation and the ideal forward direction. The operation command generates corresponding adjustment suggestions based on the magnitude and type of deviation. Position correction commands include translation operations such as forward, backward, left, and right. Direction adjustment commands cover rotation operations such as pitch, yaw, and roll. The command accuracy is measured in millimeters and degrees to ensure the accuracy of the operation. The real-time update mechanism dynamically adjusts the navigation commands according to changes in lens position. Position coordinates provide the precise location of the lens in the bronchial tree. Path commands give the specific direction and distance of the next operation. The visualized image displays the current field of view and navigation information in real time. Spatial positioning coordinates establish the correspondence between the lens position and anatomical structures.

[0128] The above describes the assisted navigation method for bronchoscopy in the embodiments of this application. The following describes the assisted navigation system for bronchoscopy in the embodiments of this application. Please refer to [link to relevant documentation]. Figure 2 One embodiment of the assisted navigation system for bronchoscopy in this application includes:

[0129] The identification module is used to identify bronchial branches based on the patient's CT image data through CT imaging, and to obtain a three-dimensional bronchial tree semantic graph structure and semantic identifiers of branch nodes.

[0130] The detection module is used to perform target detection processing of bronchial lumen based on bronchoscope images using a lightweight detection network, and obtain bronchial lumen detection results and confidence scores.

[0131] The association module is used to perform trajectory association processing on the bronchial lumen detection results using a multi-target tracking algorithm to obtain the lumen tracking trajectory sequence and motion state information;

[0132] The matching module is used to perform graph matching algorithm association processing based on the cavity tracking trajectory sequence and the semantic identifier of the branch node to obtain the detection-airway matching result and the matching confidence.

[0133] The planning module is used to perform path planning processing based on the detection-airway matching results and the motion state information to obtain real-time three-dimensional navigation guidance information and spatial positioning coordinates.

[0134] above Figure 2 The auxiliary navigation system for bronchoscopy in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The auxiliary navigation device for bronchoscopy in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0135] Reference Figure 3 This invention also provides an auxiliary navigation device for bronchoscopy, which can be a server, and its internal structure can be as follows: Figure 3 As shown. The bronchoscope-assisted navigation device includes a processor, memory, display screen, input device, network interface, and database connected via a system bus. The processor, designed as a computer, provides computing and control capabilities. The memory of the bronchoscope-assisted navigation device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the bronchoscope-assisted navigation device stores the data corresponding to this embodiment. The network interface of the bronchoscope-assisted navigation device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the above-described method.

[0136] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the auxiliary navigation device for bronchoscopy to which the present invention is applied.

[0137] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the assisted navigation method for bronchoscopy.

[0138] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an auxiliary navigation device for bronchoscopy (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for assisted navigation in bronchoscopy, characterized in that, The method includes: Based on the patient's CT image data, bronchial branch identification is performed through CT imaging to obtain a three-dimensional bronchial tree semantic graph structure and semantic identifiers of branch nodes. Based on the bronchoscopic images, a lightweight detection network is used to perform target detection processing of the bronchial lumen, and the bronchial lumen detection results and confidence scores are obtained. The bronchial lumen detection results are processed by a multi-target tracking algorithm to obtain the lumen tracking trajectory sequence and motion state information. Based on the cavity tracking trajectory sequence and the semantic identifier of the branch node, a graph matching algorithm is used to associate the detection-airway matching result and the matching confidence. Based on the detection-airway matching results and the motion state information, path planning is performed to obtain real-time three-dimensional navigation guidance information and spatial positioning coordinates.

2. The assisted navigation method for bronchoscopy according to claim 1, characterized in that, The process of identifying bronchial branches through CT imaging based on patient CT image data to obtain a three-dimensional bronchial tree semantic map structure and semantic identifiers for branch nodes includes: Adaptive threshold segmentation was performed on the patient's CT image data to obtain the bronchial lumen boundary contour and background separation results; The bronchial cavity boundary contour is input into morphological opening and closing operations for noise reduction processing to obtain a continuous bronchial cavity structure. Based on the bronchial cavity structure, a region growing algorithm is used to identify the branch hierarchy, resulting in a branch sequence from the first-level main bronchus to the eighth-level terminal bronchus. Each bronchial node is labeled with attributes according to the branch sequence to obtain node attribute information including spatial coordinates, branch level, diameter parameters and connection relationships; The node attribute information is organized and stored according to a tree data structure to obtain a three-dimensional bronchial tree semantic graph structure and branch node semantic identifiers with unique semantic identifiers.

3. The assisted navigation method for bronchoscopy according to claim 1, characterized in that, The process of detecting bronchial orifices using a lightweight detection network based on bronchoscopic images yields bronchial orifice detection results and confidence scores, including: The bronchoscope images were normalized to obtain standardized color image data of 512×512 pixels. The standardized color image data is input into the improved YOLO architecture for feature extraction processing to obtain a multi-scale feature map containing sixteen convolutional layers and four pooling layers. Based on the multi-scale feature map, a depthwise separable convolutional structure is used to perform cavity shape recognition processing to obtain circular and elliptical cavity candidate regions; Based on the candidate cavity region, a multi-scale feature fusion algorithm is used to perform target classification processing to obtain cavity type identifier and bounding box coordinate information; The bounding box coordinate information is combined with the lumen type identifier to perform confidence calculation, resulting in bronchial lumen detection results and confidence scores that include lumen location, size, and classification.

4. The assisted navigation method for bronchoscopy according to claim 1, characterized in that, The step of using a multi-target tracking algorithm to perform trajectory association processing on the bronchial orifice detection results to obtain the orifice tracking trajectory sequence and motion state information includes: Based on the bronchial lumen detection results, an independent tracking trajectory is established for each detected lumen to obtain trajectory status information including lumen position coordinates, movement speed, appearance features and existence time. The trajectory state information is input into a Kalman filter for motion prediction processing to obtain the predicted coordinates and velocity components of the cavity position in the next frame. Based on the predicted coordinates and the detection results of the current frame, the IoU distance and appearance similarity are calculated to obtain the cost matrix between the predicted position and the detection position; Based on the cost matrix, the Hungarian algorithm is used to perform optimal matching and solution processing to obtain the association pairing results between the cavity detection results and the tracking trajectory; The association pairing results are processed according to the trajectory management strategy to update the status, resulting in a cavity tracking trajectory sequence and motion status information containing tracking ID, position sequence and motion vector.

5. The assisted navigation method for bronchoscopy according to claim 1, characterized in that, The graph matching algorithm association processing based on the cavity tracking trajectory sequence and the semantic identifier of the branch node is used to obtain the detection-airway matching result and matching confidence, including: Based on the cavity tracking trajectory sequence, the spatial distribution pattern of cavities within the current field of view is analyzed and processed to obtain the distribution characteristic parameters of the number of cavities, their relative positional relationship, and their size ratio. The distribution feature parameters and the semantic identifiers of the branch nodes are used to perform candidate region search processing to obtain a set of candidate branch regions with similar distribution patterns; Based on the candidate branch region set, geometric matching degree and topological consistency verification are calculated to obtain the similarity value between the visual observation result and the semantic graph node. Based on the similarity values, a dynamic programming algorithm is used to solve for the optimal association, resulting in the optimal association combination that satisfies the one-to-one matching constraint. The optimal association combination is combined with a confidence assessment mechanism for matching verification to obtain the detection-airway matching result and matching confidence, which includes node ID, matching confidence and branch level information.

6. The assisted navigation method for bronchoscopy according to claim 5, characterized in that, The step of performing candidate region search processing on the distribution feature parameters and the semantic identifier of the branch node to obtain a set of candidate branch regions with similar distribution patterns includes: Based on the aforementioned distribution characteristic parameters, geometric feature calculations are performed on the relative angles and distance ratios between the cavities to obtain a geometric description vector of the spatial distribution of the cavities. The geometric description vector is compared with the spatial coordinates in the semantic identifier of the branch node to obtain the matching degree score of the geometric relationship between the nodes. Based on the matching score, a hierarchical search process is performed on the three-dimensional bronchial tree semantic graph structure to obtain a set of candidate nodes that meet the geometric similarity threshold. Based on the candidate node set, the connection relationship and branch level are verified by topological structure to obtain a branch region consistent with the observed cavity topology. The branch regions are sorted and filtered according to their similarity scores to obtain a set of candidate branch regions containing highly matched nodes and neighborhood information.

7. The assisted navigation method for bronchoscopy according to claim 1, characterized in that, The step of performing path planning based on the detection-airway matching results and the motion state information to obtain real-time three-dimensional navigation guidance information and spatial positioning coordinates includes: Based on the detection-airway matching results, the current lens position and target lesion position are determined, and the spatial coordinate information of the starting node and the target node is obtained. The spatial coordinate information is combined with the motion state information to perform A* search algorithm path calculation to obtain the optimal path sequence from the current node to the target node; Based on the optimal path sequence, a safety assessment is performed on the bronchial diameter constraint and bending angle limitation to obtain a feasible path scheme that meets the operational safety requirements. Based on the feasible path scheme, a three-dimensional visualization display and distance and angle measurement processing are generated to obtain navigation display information containing three-dimensional graphics and spatial references; The navigation display information is combined with the current position deviation to generate operation instructions, resulting in real-time 3D navigation guidance information and spatial positioning coordinates that include position coordinates, path instructions, and visual images.

8. An auxiliary navigation system for bronchoscopy, characterized in that, For implementing the assisted navigation method for bronchoscopy as described in any one of claims 1-7, the assisted navigation system for bronchoscopy comprises: The identification module is used to identify bronchial branches based on the patient's CT image data through CT imaging, and to obtain a three-dimensional bronchial tree semantic graph structure and semantic identifiers of branch nodes. The detection module is used to perform target detection processing of bronchial lumen based on bronchoscope images using a lightweight detection network, and obtain bronchial lumen detection results and confidence scores. The association module is used to perform trajectory association processing on the bronchial lumen detection results using a multi-target tracking algorithm to obtain the lumen tracking trajectory sequence and motion state information; The matching module is used to perform graph matching algorithm association processing based on the cavity tracking trajectory sequence and the semantic identifier of the branch node to obtain the detection-airway matching result and the matching confidence. The planning module is used to perform path planning processing based on the detection-airway matching results and the motion state information to obtain real-time three-dimensional navigation guidance information and spatial positioning coordinates.

9. An auxiliary navigation device for bronchoscopy, characterized in that, It includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the computer program to implement the assisted navigation method for bronchoscopy as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it causes the processor to perform the assisted navigation method for bronchoscopy as described in any one of claims 1 to 7.

Images