Bronchoscopy view-based airway navigation system and method

Abstract

The application provides an airway navigation system and method based on a bronchoscope view, which comprises a reconstruction module, a model construction module, a path extraction module, a path selection module, a navigation module and a navigation assistance module. The reconstruction module reconstructs a bronchial tree 3D mask through CT images. The model construction module obtains a three-dimensional virtual map model comprising a 3D bronchial tree according to the bronchial tree 3D mask. The path extraction module extracts all possible navigation paths from a bronchial entrance to a target site based on the 3D bronchial tree. The path selection module selects a target navigation path in a navigation path. The navigation module guides the bronchoscope to move along the target navigation path in the three-dimensional virtual map model based on a visualized guide path. The navigation assistance module maps a real-time site corresponding to the bronchoscope to the 3D bronchial tree for matching, so as to realize dynamic synchronization of the real-time site corresponding to the bronchoscope and the target navigation path. The system can reliably and conveniently navigate in a complex airway network, effectively avoid misoperation, save time and reduce operation difficulty.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical navigation planning, and in particular to a bronchoscopic view-based airway navigation system and method. BACKGROUND

[0002] Generally, when conducting an examination of a human airway, a medical practitioner will need to examine a large portion or all of the lung tree, such as the trachea, left and right bronchi, and their respective bronchioles and alveoli, to check for any abnormalities. In other cases, it is desirable to investigate a particular portion of a human airway, such as a particular bronchiole, based on, for example, magnetic resonance (MR) or computed tomography (CT) scan results, including bronchoscopic diagnosis and post-biopsy and post-puncture pathological diagnosis.

[0003] However, when navigating through portions of a human airway, a medical practitioner typically navigates a bronchoscope through the human airway based on experience, for example, reaching a large portion / all of the lung tree, and / or a particular portion based on camera images from the bronchoscope. Since the portions of a human airway, such as the respective bronchi or bronchioles, often look very similar, there is a risk of making errors, for example, not reaching a desired portion of the human airway or mistaking a portion of the airway for a different portion of the airway. This in turn increases the risk of the patient not being properly examined.

[0004] To improve the accuracy of the examination, existing solutions establish a three-dimensional virtual navigation map based on bronchial CT images through computer graphics technology, and during the operation, the bronchoscope operation process is guided by comparing the three-dimensional virtual navigation map. This virtual navigation cannot automatically correspond the bronchoscopic image with the position in the three-dimensional virtual navigation map, and the doctor must rely on experience to judge the position of the bronchoscope by comparing the three-dimensional virtual navigation map and the current view of the bronchoscope. The operation process needs to be operated manually following the guidance of virtual roaming. This way requires a higher real-time reaction ability and operation accuracy of the doctor. At the same time, during the operation, when the patient's anatomical structure changes due to factors such as breathing and tissue displacement, the real-time position of the bronchoscope cannot be identified, and the navigation path cannot be adjusted in real time to adapt to these changes. Therefore, how to realize the continuous real-time synchronous update of the target navigation path based on the three-dimensional virtual navigation map and the real-time position recognized by the bronchoscopic view is still a challenge, especially in dynamic operation, how to maintain the accuracy and real-time nature of the real-time navigation route of the bronchoscope during the operation is still to be solved.

[0005] The prior art CN112598639A discloses using a stitching point algorithm for bronchial tree path planning, which can quickly and conveniently plan an automatic roaming path to reach any branch end point after obtaining a complete center line of the bronchial tree, but does not solve the problem that the target navigation path of bronchoscope navigation in the three-dimensional virtual navigation map picture and the real-time position recognized under the actual visual angle when operating the bronchoscope cannot be continuously and real-time synchronized.

[0006] The prior art WO2023124978A1 determines the real-time pose of the virtual bronchoscope in the real bronchial tree based on the similarity between the virtual bronchoscope image corresponding to the virtual bronchoscope and the real bronchoscope image collected by the real bronchoscope, and quickly and effectively realizes the synchronization of the virtual bronchoscope image and the real bronchoscope image, but the direct comparison of image similarity has low reliability, especially when the patient's anatomical structure changes due to factors such as respiration and tissue displacement, which can cause matching failure, image synchronization failure, or inaccurate display results. SUMMARY

[0007] To solve the problem in the prior art that the position segment of the bronchial tree during the travel of the bronchoscope according to the target navigation path and the real-time position recognized under the actual visual angle when operating the bronchoscope cannot be continuously and real-time synchronized in the three-dimensional virtual map model interface, the present application provides a bronchoscope view-based airway navigation system and method to help doctors navigate more reliably and conveniently in complex airway networks.

[0008] The first aspect of the present application provides a bronchoscope view-based airway navigation system, which comprises:

[0009] a reconstruction module that reconstructs a bronchial tree 3D mask from CT images;

[0010] a model generation module that obtains a three-dimensional virtual map model including a 3D bronchial tree corresponding to the bronchial tree 3D mask;

[0011] a path extraction module that extracts a navigation path from a bronchial entrance to a target position based on the 3D bronchial tree;

[0012] a path selection module that selects one of the navigation paths as a target navigation path based on preset planning parameters;

[0013] a navigation module that guides the bronchoscope to move along the target navigation path in the three-dimensional virtual map model based on the visualized guide path;

[0014] A navigation assistance module, which maps the recognized real-time position corresponding to the bronchoscope to the 3D bronchial tree for matching, so as to realize dynamic synchronization of the real-time position corresponding to the bronchoscope and the target navigation path.

[0015] Further, the reconstruction module comprises:

[0016] An input preparation unit configured to select a data block subjected to blocking and normalization processing from a thin-slice CT image sequence corresponding to a bronchus as an input object;

[0017] A prediction unit configured to input the input object into a pre-trained prediction model to predict a bronchus voxel;

[0018] An output unit configured to obtain a bronchial tree 3D mask by splicing and restoring based on the bronchus voxel.

[0019] Further, the model generation module comprises:

[0020] A preprocessing unit configured to perform morphological repair and boundary smoothing preprocessing on the bronchial tree 3D mask;

[0021] A visualization unit configured to realize visualization of the bronchial tree and the three-dimensional virtual map model based on the bronchial tree 3D mask after preprocessing and VTK visualization technology.

[0022] Further, the airway navigation system further comprises a path planning module configured to generate the preset planning parameter based on at least two of a path length of the navigation path, an entering angle when contacting the target position, an entering distance when contacting the target position, an airway diameter of a terminal position of the navigation path, and a blood vessel density of the terminal position of the navigation path.

[0023] Further, the path planning module comprises at least two of the following units:

[0024] A path length calculation unit configured to calculate a shortest path from the bronchus entrance to the target position in the navigation path;

[0025] An entering angle calculation unit configured to calculate an included angle between an airway direction of the terminal position of the navigation path and a surface normal of the target position;

[0026] An entering distance calculation unit configured to calculate a shortest distance from the terminal position of the navigation path to the target position;

[0027] An airway diameter calculation unit configured to calculate a cross-sectional diameter of an airway of the terminal position of the navigation path;

[0028] a blood vessel density calculation unit configured to calculate a blood vessel density value of an end position of the navigation path.

[0029] Further, the path selection module comprises:

[0030] a path evaluation unit configured to evaluate a path score of each of the navigation paths based on the weights of the preset planning parameters, and select the navigation path with the highest path score as the target navigation path.

[0031] Further, the navigation module comprises:

[0032] a structuring unit configured to structure the set of centerline points of the 3D bronchial tree to generate a centerline tree;

[0033] a guiding unit configured to generate a guiding path from a root node of the centerline tree to the target site according to the tree structure information of the centerline tree.

[0034] Further, the navigation assistance module comprises:

[0035] a site identification unit configured to identify a real-time site corresponding to a real-time image collected by the bronchoscope at a current real-time position based on a pre-trained site identification model;

[0036] a segment identification unit configured to identify a site segment of the 3D bronchial tree based on a pre-trained point cloud classification model;

[0037] a site mapping unit configured to map the identified real-time site to the site segment of the 3D bronchial tree corresponding thereto;

[0038] a synchronization unit configured to dynamically synchronize a mapping result of the site mapping unit during movement of the bronchoscope along the target navigation path.

[0039] Further, the segment identification unit comprises:

[0040] a point cloud generation part configured to generate a set of point clouds based on the extracted centerline of the 3D bronchial tree;

[0041] a classification part configured to perform point cloud classification on the preprocessed set of point clouds based on a pre-trained point cloud classification model to obtain a classification result as an identification result of the site segment.

[0042] Further, the site mapping unit comprises:

[0043] a coordinate conversion part configured to convert a real-time position of the bronchoscope identified to the real-time site to a display position of the site segment in a view coordinate system based on a relative relationship between a real-time coordinate system and the view coordinate system;

[0044] an angle-of-view adjusting unit configured to adjust a display angle of view of the real-time site in the view coordinate system based on display position information of the site segment corresponding to the real-time site in the view coordinate system.

[0045] A second aspect of the present application provides a bronchoscopic view-based airway navigation method, applied in the bronchoscopic view-based airway navigation system, the method comprising:

[0046] reconstructing a 3D mask of the bronchial tree from the CT images;

[0047] obtaining a three-dimensional virtual map model corresponding to the 3D bronchial tree according to the 3D mask of the bronchial tree;

[0048] extracting a navigation path from a bronchial inlet to a target site based on the 3D bronchial tree;

[0049] selecting one of the navigation paths as a target navigation path based on preset planning parameters;

[0050] guiding the bronchoscope to move along the target navigation path in the three-dimensional virtual map model based on the visualized guide path;

[0051] mapping the recognized real-time site corresponding to the bronchoscope to the 3D bronchial tree for matching, so as to realize dynamic synchronization of the real-time site corresponding to the bronchoscope and the target navigation path.

[0052] The airway navigation system disclosed in the present application can reconstruct a 3D bronchial tree based on CT images, generate a target navigation path based on a bronchial inlet and a target site, match real-time images acquired under the bronchoscope to structural information of the 3D bronchial tree reconstructed based on a conventional CT sequence, and establish dynamic synchronization of a real-time scene under the bronchoscope and the target navigation path in the three-dimensional virtual map model reconstructed based on CT, so as to realize movement along the target navigation path and real-time sensing of the real-time position of the bronchoscope in the process of bronchoscope navigation, and provide accurate navigation and positioning support for operation under the bronchoscope. Therefore, the doctor can accurately view the specific position recognized by the bronchoscope and the dynamic synchronization condition of the movement progress of the target navigation path on the three-dimensional virtual map model, which can effectively avoid misoperation and reduce damage to normal airway tissues, and is particularly suitable for surgical operation of complex airway structures.

[0053] Further, based on the three-dimensional virtual map model established based on the CT sequence images, an optimal navigation path from the current bronchoscope position to the target site is used as a target navigation path in the three-dimensional virtual navigation model by using preset planning parameters, the target navigation path planning taking into account the shape of the airway and possible obstacles, etc., so that the doctor can quickly reach the target site according to the planned optimal operation path under the guidance of the target navigation path when operating the bronchoscope, saving operation time and reducing operation complexity. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0055] Figure 1 A structural schematic diagram of a bronchoscope 1 is shown;

[0056] Figure 2 A computer device embedded with an airway navigation system based on bronchoscope view is shown;

[0057] Figure 3 A system diagram of an airway navigation system based on bronchoscope view is shown;

[0058] Figure 4 A system block diagram of a reconstruction module is shown;

[0059] Figure 5 A system block diagram of a model generation module is shown;

[0060] Figure 6 A system diagram of an airway navigation system based on another bronchoscope view is shown;

[0061] Figure 7 A system block diagram of a navigation module is shown;

[0062] Figure 8 A system block diagram of a navigation assistance module is shown;

[0063] Figure 9 A flowchart of an airway navigation method based on bronchoscope view is shown. DETAILED DESCRIPTION

[0064] The present disclosure will be further described below in conjunction with the drawings and embodiments.

[0065] It should be noted that the following detailed description is intended to provide further description of the present disclosure. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0066] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0067] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0068] A bronchoscope generally includes an image capturing device such as a camera at the front end of the bronchoscope to be inserted into the patient's body and connected to a display to provide the medical staff with a view of the airway portion where the front end of the bronchoscope is located.

[0069] Among them, the bronchial classification can be: after the trachea enters the chest cavity, it is divided into left and right main bronchus. The right main bronchus is divided into upper lobe bronchus and intermediate segment bronchus, and the intermediate segment bronchus is divided into middle lobe and lower lobe bronchus, and the left main bronchus is divided into upper lobe and lower lobe bronchus, and the left upper lobe bronchus is divided into lingual segment bronchus branch, so that the right lung is divided into upper, middle and lower three lobes, and the left lung is divided into upper and lower two lobes. These bronchi are further divided into segmental bronchi, subsegmental bronchi, terminal bronchioles, respiratory bronchioles, alveolar ducts, alveolar sacs and alveoli.

[0070] Please refer to Figure 1, a schematic view of a bronchoscope 1 is shown. The bronchoscope can be disposable or reusable. The bronchoscope 1 comprises an insertion portion 2. At the proximal end of the insertion portion 2 a handle 3 is arranged for operating the bronchoscope 1. At the distal end of the insertion portion 2 a tip portion 5 is provided. A camera assembly 6 is positioned in the tip portion 5 and configured to transmit image signals to a display 11 by a processing device 12 of the bronchoscope 1. The display 11 can allow an operator to view real-time images captured by the camera assembly 6 of the bronchoscope 1 as well as a target navigation path in a virtual navigation image of the bronchial interior based on CT images to assist bronchoscope operation.

[0071] The display 11 is coupled with the processor and operable to render images of the patient anatomy. Such images can be based on a set of preoperatively obtained images (e.g., CT or MRI scans, 3D maps, etc.). The view of the patient anatomy provided by the display 11 can also dynamically change based on the current identified real-time location. For example, as the bronchoscope 1 moves within the patient, data related to its real-time location can cause the processor to update the view of the patient anatomy in the display 11 in real-time to depict the region of the patient anatomy that the camera assembly 6 of the bronchoscope 1 is acquiring.

[0072] In an embodiment, referring to Figure 2 , the airway navigation system based on bronchoscope view can be embedded in a computer device as shown in Figure 2 . The computer device can be a separately arranged computer device, or a related computer device in the navigation system. The computer device comprises a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement the airway navigation method based on bronchoscope view. The display screen of the computer device can be a liquid crystal display screen or an electronic display screen, etc. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0073] Referring to Figure 3, shows a system diagram of the airway navigation system based on bronchoscope view. The airway navigation system 7 comprises: a reconstruction module 71 which reconstructs a 3D mask of the bronchial tree through segmented CT images; a model generation module 72 which constructs a three-dimensional virtual map model corresponding to the displayable 3D bronchial tree according to the 3D mask of the bronchial tree and displays the three-dimensional virtual map model; a path extraction module 73 which extracts a navigation path from the bronchial entrance to the target site based on the 3D bronchial tree; a path selection module 74 which selects one of the navigation paths as a target navigation path based on preset planning parameters; a navigation module 75 which guides the bronchoscope to move along the corresponding target navigation path in the three-dimensional virtual map model based on the visualized guide path; and a navigation assistance module 76 which maps the recognized real-time site corresponding to the bronchoscope to the 3D bronchial tree for matching, so as to realize dynamic synchronization of the real-time site corresponding to the bronchoscope and the target navigation path.

[0074] The 3D bronchial tree is a mask used to represent the structure of the bronchial tree in three-dimensional space. In the 3D mask of the bronchial tree, the mask is a three-dimensional data set, and the value of each voxel can be used to indicate whether it belongs to the structure of the bronchial tree.

[0075] The airway navigation system 7 segments and reconstructs the acquired CT sequence images based on the reconstruction module and the model generation module to obtain the 3D mask of the bronchial tree, and further performs morphological hollowing processing on the 3D mask of the bronchial tree to obtain a model of the 3D bronchial tree with a cavity. Then, the 3D bronchial tree with the cavity model is imported into the VTK world coordinate system, and the recognition results of the corresponding bronchial site segments are temporarily stored. At the same time, a three-dimensional virtual map model of the bronchial tree is constructed from the three-dimensional segmentation result using the VTK visualization construction tool. The positions of the bronchial sites in the reconstructed 3D bronchial tree form a one-to-one mapping relationship with the positions in the three-dimensional virtual map model. At the same time, the system receives the video stream output by the computer device (bronchoscope host), and uses deep learning video understanding technology to recognize the real-time site corresponding to the real-time image collected by the real-time position of the bronchoscope in real time. After recognizing the real-time site under the bronchoscope, a corresponding mapping relationship is established with the site segment recognition result of the 3D bronchial tree, thereby indirectly establishing linkage with the navigation path in the three-dimensional virtual map model. When the bronchoscope moves in the bronchial tree, the current position is highlighted or indicated on the corresponding area on the three-dimensional virtual map model, or the view of the current site segment is played in real time on the display interface, etc., to assist the navigation process of the bronchoscope, so that the doctor can perceive the current position of the bronchoscope during navigation.

[0076] Please refer to Figure 4 , a system block diagram of the reconstruction module is shown. The reconstruction module 71 comprises: an input preparation unit 711 for selecting the thin layer data after blocking and normalization processing from the thin layer CT image sequence corresponding to the bronchus as the input object; a prediction unit 712 for inputting the input object into the pre-trained prediction model to predict the bronchus voxel; an output unit 713 for splicing and restoring the bronchus region including the bronchus voxel to output the bronchus tree 3D mask. The model training and iteration process of the prediction model will be described in detail below.

[0077] The input preparation unit 711 is used to select the data block after blocking and normalization processing from the thin layer CT image sequence corresponding to the bronchus as the input object. The thin layer CT image sequence is formed by a plurality of slice images. Each frame of CT image obtained in the CT examination of the bronchus can be one cross section of the bronchus, and the thin layer CT image data thereof is first selected as the analysis object, each CT image is cut into a plurality of small blocks, two-dimensional blocking is performed along the x-y plane, and three-dimensional small blocks are formed in combination with a plurality of slices in the z direction, and the spatial proximity relationship and depth information of the CT data are fully considered, for example, the block division is performed in a block division manner of 64x64x64 size and a step length of 16 to form a plurality of sub-blocks, and the magnitude difference of the image gray value is eliminated by linear normalization or Z-Score normalization processing.

[0078] In addition, the bronchus in the CT image obtained by the image examination often occupies a small part of the CT image, and the CT sequence of the three-dimensional body data. If the entire CT image is directly processed, the calculation amount is large, and the CT image is divided into a plurality of sub-blocks (patches) in the present application, which can effectively reduce the calculation range and focus on the local area features, and facilitate further analysis.

[0079] The prediction unit 712 is used to input the data block into the pre-trained prediction model for inference and prediction, and the voxels that satisfy the bronchus part features in the prediction analysis process are selected based on the prediction model score. Specifically, for the voxels whose output results are greater than or equal to the preset threshold, they are determined as bronchus voxels, and for the voxels whose output results are less than the preset threshold, they are considered as other backgrounds. The corresponding output mask of the data block is obtained. Specifically, when the prediction score obtained by the prediction model is in the interval of 0-1, the preset threshold can be set to 0.5, when the prediction score is greater than or equal to 0.5, the corresponding voxel is considered as a bronchus voxel, otherwise it is considered as a background or other tissue related voxel.

[0080] The output unit 713 inputs all the input blocks of the relevant data of the output pre bronchus reconstruction into the prediction model before output and predicts the output blocks output from the prediction model including bronchus voxels, background, other tissues, etc. according to the spatial positions of the original CT data after the prediction is completed, and restores and splices the output blocks to obtain the prediction result of the bronchus tree 3D mask based on 3D sequences. The bronchus tree 3D mask can include the branch structure of the entire bronchus system from the main bronchus to the segmental bronchus and even finer bronchus branches.

[0081] In an embodiment, when the overlapping area is introduced in the sub-blocking process of the input preparation unit 711, different weight values can be assigned to the overlapping parts in different sub-blocks to restore by weighted average. For example, the center of the block is taken as the reference point, and the part close to the block center is given a large weight, and the part close to the edge is given a small weight, so as to ensure that the spliced 3D mask image has no obvious boundary.

[0082] The thin layer data refers to the thickness of each slice in the CT scan being very thin (usually not more than 1 mm), and the slice containing at least the main bronchus and smaller airways. The thin layer data is helpful to obtain a high-resolution image of the details inside the bronchus, and by selecting the thin layer data, the redundant information in the data can be reduced, and the accuracy of the local features can be enhanced.

[0083] The training process of the prediction model mainly includes selecting the entire bronchus structure CT data, screening the CT data meeting the layer thickness branch (thin layer) as the labeled object, manually labeling the main bronchus, secondary bronchus and its branches, bronchus region boundary, etc. that can form the bronchus tree mask, and dividing the training set, validation set and test set in units of CT sequences after the manual labeling is completed. The number of training set, validation set and test set is 8:1:1, a 3D segmentation network model Vnet is constructed, the input size of the network model is set to 64x64x64, the output layer size is also 64x64x64, the last output layer of the network is activated by a sigmoid function, and the output range of the output layer output value is between 0 and 1, representing the probability value of the corresponding voxel of the input data belonging to a part of the bronchus structure.

[0084] Since the dimension of a CT sequence is generally 512x512xZ or 1024x1024xZ (Z is the sequence depth), it cannot be input into the network model for one-time calculation and training, and needs to be cut into blocks. The entire process can refer to the network input size of Vnet, and the original 3D CT data and the corresponding labeled 3D mask are divided into blocks according to the size of 64x64x64 and the step of 16, to obtain training data and corresponding labels that meet the network input, and improve the stability and generalization ability of the pre-trained prediction model. At the same time, the training data is normalized to 0-1 according to the window width WW=1500 and the window level WL=-400, and the normalization formula is as follows:

[0085] Where x is the original CT value, and f(x) is the normalized result:

[0086]

[0087] The entire model training process also includes randomly initializing the network model, setting the dice loss as the target loss function for network model training, using the Adam optimizer to optimize the model parameters, and obtaining the pre-trained prediction model.

[0088] Referring to Figure 5 , a system block diagram of the model generation module is shown. The model generation module 72 includes: a preprocessing unit 721 for morphological repair and boundary smoothing preprocessing of the bronchial tree 3D mask; a visualization unit 722 capable of visualizing a three-dimensional virtual map model including a 3D bronchial tree based on the preprocessed bronchial tree 3D mask. The 3D bronchial tree displayed in the three-dimensional virtual map model labels the structure and path information inside the bronchus, can directly display the anatomical structure of the 3D bronchus, and can perform path extraction, path planning generation, etc. on this basis.

[0089] The preprocessing unit 721 is used after the generation of the bronchial tree 3D mask to reduce the problem of noise or inaccurate segmentation in the mask data in the bronchial tree 3D mask. By using boundary smoothing processing such as Gaussian filtering, noise can be reduced, and the bronchial boundary can be smoothed. By morphological operation, the internal cavity (channel) part is extracted, and by inflation and corrosion morphological repair processing, small holes or small protrusions in the mask are corrected, the holes inside the bronchus are filled, and the bronchial structure is more coherent, to ensure the integrity of the reconstructed 3D bronchial tree.

[0090] The visualization unit 722 is configured to realize the visualization of the three-dimensional virtual map model including the 3D bronchial tree in a VTK visualization manner, and meanwhile, based on the specific position and shape information of the target site, all possible navigation paths that can reach the target site can be calculated by a VTK path planning algorithm, which can be one branch or multiple branches. By performing appropriate scaling and rotation adjustment to a suitable view angle and mapping into a VTK virtual coordinate system, and providing interactive operation, the user is allowed to rotate, scale and translate the bronchial tree model in the three-dimensional virtual map model, so as to observe the structure of the 3D bronchial tree from different angles and levels.

[0091] Specifically, the 3D bronchial tree is converted into a model in a VTK corresponding coordinate system in a VTK visualization manner, and all positions corresponding to the 3D bronchial tree are adjusted to form a three-dimensional virtual map model of the 3D bronchial tree by performing a corresponding scaling and rotation rule. When the coordinates of a certain point of the original 3D bronchial tree directly reconstructed are (x, y, z), the coordinates (x', y', z') in the VTK coordinate system are obtained through a rotation transformation matrix M, which includes the coefficients of rotation and scaling and the translation amount. The specific rotation transformation M can be set according to the actual display requirements of the VTK coordinate system. Thus, the display angle and scale of the 3D bronchial tree in the corresponding three-dimensional virtual map model in the VTK coordinate system are accurate and controllable.

[0092] Therefore, the bronchial tree 3D mask is generated by a surface reconstruction method, morphological hollowing treatment, etc., to include a geometric surface and internal structure three-dimensional model of the 3D bronchial tree that can have a lumen, and is imported into a VTK corresponding coordinate system to provide a three-dimensional virtual map model reaching the target site in a VTK visualization manner.

[0093] The path extraction module 73 automatically identifies and extracts a plurality of possible navigation paths from the bronchial entrance to the target site (such as a lesion, a biopsy point, etc.) in the three-dimensional virtual map model based on the reconstructed 3D bronchial tree, which can contain multiple branches and bending points. The extraction of these paths is based on the reconstructed 3D bronchial tree and the position data of the target site, and as long as the target site can be reached from the bronchial entrance.

[0094] In some embodiments, the path selection module 74 can also use path search algorithms such as Dijkstra algorithm and A* algorithm to identify all navigation paths of the target navigation path from the bronchial airway entrance to the target site, and when the navigation path exceeds one branch, the shortest path is selected as the standard for path selection by calculating the path length.

[0095] In an embodiment, please refer to Figure 6Fig. 7 shows a system diagram of an airway navigation system based on another bronchoscopic view. To further improve the reliability of the path selected by the path selection module, the airway navigation system 7 also provides a path planning module 77 which generates the preset planning parameters based on at least two of the path length of the navigation path to the target site, the entry angle of the target site, the entry distance of the target site, the airway diameter of the end position of the navigation path, and the blood vessel density of the end position of the navigation path.

[0096] The path planning module 77 generates the planning parameters required for the selected target navigation path by calculating the results of at least two of the following units in order to obtain the preset planning parameters, specifically including:

[0097] A path length calculation unit is configured to calculate the shortest path from the bronchial entrance to the target site in the navigation path. The calculation of the path length can be performed by using the K-shortest path algorithm to find other navigation paths based on the shortest path found, using a recursive method to find all paths in the graph, and using a backtracking method to return to the previous branching point to find other paths after finding one of the paths. This can effectively calculate all feasible paths from the bronchial entrance to the target site, and when multiple navigation paths are present, the length of each path can be calculated. A shorter path length usually means a shorter operation time, reducing patient discomfort and surgical risk.

[0098] An entry angle calculation unit is configured to calculate the included angle between the airway direction of the end position of the navigation path and the normal of the surface of the target site. This included angle can indicate whether the path to the target site is steep or smooth, and the included angle θ can be calculated according to the dot product of the path end airway direction vector and the target site surface normal vector and the length of the path direction vector and the target site surface normal. For each possible path, the airway direction vector Vpath at the path end is calculated, and the direction vector Vpath is defined by the previous point Pn-1 and the end point Pn of the path end:

[0099] Vpath = (Xn - Xn-1, Yn - Yn-1, Zn - Zn-1)

[0100] According to the target site extracted from the reconstructed 3D bronchial tree, the normal vector Nlesion of the surface of the target site is calculated, and the final angle θ at the position close to the path end of the target site (such as a lesion, a biopsy point, etc.) is calculated:

[0101]

[0102] wherein Vpath · Nlesion is the dot product of the path end direction vector and the target site surface normal vector;

[0103] |Vpath||Nlesion| are the lengths of the airway direction vector at the end of the path and the surface normal of the target site, respectively.

[0104] It can be understood that a smaller angle θ helps the bronchoscope or other tools to approach the target site in the right direction, increasing the success rate of biopsy. The smaller the angle θ, the closer the entry angle of the path to the surface normal of the target site (i.e., perpendicular to the lesion), and theoretically, the higher the success rate of operation. In general, an angle of no more than 30° is considered a small angle. For example, when using biopsy forceps to collect biopsy, the optimal entry angle of the biopsy forceps is 0°, at which the biopsy forceps enter the target site vertically, and the doctor's biopsy operation is the most convenient and simple.

[0105] In actual operation, if no path can reach the ideal angle, the doctor will usually choose a path with an angle θ of no more than 60 degrees to meet the operability and accuracy as much as possible.

[0106] The entry distance calculation unit is used to calculate the shortest distance from the end position of the navigation path to the target site. A smaller end distance generally represents a higher success rate of surgery and accuracy of biopsy, making it easier for the doctor to obtain lesion tissue samples.

[0107] The distance from the end of the navigation path to the target site (such as a lesion, biopsy point, etc.) can be achieved by calculating the Euclidean distance between the two, and the specific formula is:

[0108]

[0109] Where (x1, y1, z1) is the coordinate of the end of the path, and (x2, y2, z2) is the coordinate of the center point of the target site (such as a lesion, biopsy point, etc.).

[0110] In an embodiment, if the target site surface is complex and irregular, the end position of the navigation path can also be projected onto the target site surface, and the distance from the projection point to the target site can be calculated.

[0111] The airway diameter calculation unit is used to calculate the airway cross-sectional diameter of the end position of the navigation path. A larger airway diameter helps the bronchoscope and other tools to pass smoothly, reducing the difficulty of operation.

[0112] In an embodiment, when the cross-sectional shape of the target site is relatively regular as a whole, the calculation of the airway cross-sectional diameter of the end position of the navigation path can be achieved by measuring the maximum diameter at the end of the navigation path, i.e., the width of the airway, as the airway cross-sectional diameter.

[0113] In an embodiment, when the cross-sectional shape of the target site is irregular or complex, the calculation of the airway cross-sectional diameter of the end position of the navigation path can also be achieved by obtaining the maximum diameter and the minimum diameter of the airway cross-section measured in the vertical direction along the airway, calculating the average of the maximum diameter and the minimum diameter as the airway diameter at the end position.

[0114] A blood vessel density calculation unit is configured to calculate a blood vessel density value of the end position of the navigation path. A high blood vessel density can indicate a high risk of the operation area, as puncture or operation can cause bleeding; a lower density means a lower risk of operation. After blood vessel reconstruction based on the CT image, the calculation of the blood vessel density is achieved by the number and distribution of blood vessels near the end position of the navigation path, and the specific calculation method is as follows:

[0115] A three-dimensional volume V is set near the end position of the navigation path total (such as a sphere or a cube), the total number of detected blood vessels or the volume ratio of blood vessels in the region is counted, and the density value D can represent the number of blood vessels or the volume ratio of blood vessels per unit volume; when the density value is not more than 5%, it is considered to be a low blood vessel density area, the density value is in the range of 10%-30%, it is considered to be a medium density area, and the density value is more than 30%, it is considered to be a high density area. High-density blood vessels can increase the risk of operation, so the low blood vessel density area has higher path safety.

[0116] In this application, the blood vessel reconstruction based on the CT image can be achieved in the same way as the reconstruction of the 3D bronchial tree, which will not be described here.

[0117] The path selection module 74 further includes a path evaluation unit configured to evaluate a path score of each of the navigation paths based on the weights of the preset planning parameters, and select the navigation path with the highest path score as the target navigation path. Generating a target navigation path generally requires comprehensive evaluation of the values of each preset planning parameter in the navigation path, and by evaluating the feasibility and safety of the path, it is ensured that the selected target navigation path avoids passing through fragile tissues and important blood vessels, while maximizing the linearity and visibility of the path.

[0118] For example, when the preset planning parameters are selected for evaluation, the path length and the entry angle are set as follows: the path length weight is set to 60%, the entry angle weight is set to 40%, the path length is less than 50 mm, the score is 100, the path length is in the range of 50-100 mm, the score is 80, the path length is in the range of 100-150 mm, the score is 60, the path length is in the range of 150-200 mm, the score is 40, the path length is in the range of 200-250 mm, the score is 20; the entry angle is in the range of 0-20°, the score is 100, the entry angle is in the range of 20-30°, the score is 80, the entry angle is in the range of 30-40°, the score is 60, the entry angle is in the range of 40-60°, the score is 40, and the entry angle is greater than 60°, the score is 20.

[0119] When the three target paths A, B, C are extracted, the path length of path A is 188 mm and the entry angle is 18 degrees; the path length of path B is 136 mm and the entry angle is 45 degrees; the path length of path B is 207 mm and the entry angle is 32 degrees, which are evaluated according to the path evaluation unit 741:

[0120] Path A score = (40 x 60%) + (100 x 40%) = 64

[0121] Path B score = (60 x 60%) + (40 x 40%) = 52

[0122] Path C score = (20 x 60%) + (60 x 40%) = 36

[0123] Therefore, at this time, path A is selected as the target navigation path, although the entry angle is slightly larger during biopsy, but the overall path length is the shortest.

[0124] Further, when the scores of multiple navigation paths are the same, the target navigation path is determined according to the priority of the preset planning parameters during evaluation, such as when the path length, entry distance, and blood vessel density of path A, path B, and path C are considered simultaneously, the scores of path A and path C are the same, at this time, the priority of the blood vessel density among the three preset planning parameters is the highest, and the blood vessel density value of path C is the lowest, path C is selected as the selected target navigation path.

[0125] In some embodiments, the preset planning parameters further include: the number of narrow sections in the navigation path, the number of sharp turns, the user's input or preference, the patient's physiological parameters, the actual characteristics of the target site (such as lesions, biopsy points, etc.) to ensure that the actual operability of the target navigation path is optimized.

[0126] Please refer to Figure 7 , which shows a system block diagram of the navigation module. Further, the navigation module 75 includes: a structuring unit 751 for structuring the center line point set of the reconstructed 3D bronchial tree to generate a center line tree; a guiding unit 752 for generating a guiding path from the root node of the center line tree to the target site according to the tree structure information of the center line tree.

[0127] The structuring unit 751 is configured to extract a discrete set of centerline points of the 3D bronchial tree by calculating the distance of each bronchial voxel to the boundary of the bronchial tree, and to obtain a centerline tree by structuring the set of centerline points to explicitly the bifurcation nodes and branches of the bronchial tree, each node representing a bifurcation point of the bronchial tree, and each branch representing a centerline segment connecting two nodes. In the centerline tree, all non-leaf nodes have child nodes, and all nodes except the root node have parent nodes. In the bronchial tree, a leaf node usually represents a terminal branch of the bronchial tree, or a bronchial endpoint that has no further branching, and a non-leaf node usually represents a bifurcation point or a connection point of the bronchial tree, which has a position of bronchial bifurcation, and has two or more bronchial branches extending from the bifurcation point, which are child nodes of the bifurcation point.

[0128] It can be understood that the point on the centerline closest to the CT head direction is selected as the starting point of the bronchial tree, as the root node R of the 3D mask centerline tree of the bronchial tree, and all centerline points connected to the root node are continuously added to the tree structure using a depth-first strategy, until all points are included in the tree, and finally a tree data storage structure is formed. In this process, all non-leaf nodes have child nodes, and all nodes except the root node have parent nodes, and finally a standard tree structure is formed.

[0129] Generally, the human airway tree has about 24 levels of branches from the first level to the alveoli, and the shape is like an inverted tree, so the concept of tree structure can be extended to the data structure. The bronchus is divided into left and right main bronchi at the bifurcation, and the bronchus entrance can be regarded as a root node.

[0130] The guiding unit 752 is configured to select the starting point of the main bronchus centerline as the root node R of the centerline tree, continuously add all points to the centerline tree using a depth-first strategy, and finally form a tree data storage structure, select a target site A on the center point in the tree data storage structure, and generate a guiding path L from the root node R to the target site A according to the tree structure, and the guiding path L is used to guide the bronchoscope 1 when traveling along the target navigation path in the three-dimensional virtual map model.

[0131] Please refer to Figure 8, a system block diagram of the navigation assistance module is shown. The navigation assistance module 76 comprises: a site identification unit 761, which identifies the real-time site corresponding to the real-time image collected by the current real-time position of the bronchoscope based on a pre-trained site identification model; a section identification unit 762, which identifies the site section of the 3D bronchial tree based on a pre-trained point cloud classification model; a site mapping unit 763, which is used to map the identified real-time site to the corresponding site section of the 3D bronchial tree; and a synchronization unit 764, which is used to dynamically synchronize the mapping result of the site mapping unit during the movement of the bronchoscope along the target navigation path in the three-dimensional virtual map model. It can enable the position update / inspection site update in the bronchoscope inspection process to be displayed synchronously in the three-dimensional virtual map model, and the real-time position under the current field of view of the bronchoscope can be dynamically synchronized and updated during the movement of the bronchoscope along the target navigation path in the three-dimensional virtual map model, thereby realizing the dynamic synchronization of the specific position between the bronchoscope and the three-dimensional virtual map model. It helps doctors to track the current position of the bronchoscope and display the complete operation path, ensuring that the doctor can always accurately know the position and direction of travel of the bronchoscope.

[0132] The identification process of the site identification unit 761 includes initializing the site identification model based on the trained model parameters during system operation, inputting the continuously collected current 20 frames (sampling mode: 4 frames per second, a total of 5 seconds) of images into the trained site identification model for classification and identification, and obtaining the identification result of the current bronchoscope for the current real-time site. The site identification model can be trained using a multi-frame convolution feature fusion classification network.

[0133] Further, the training process of the site identification model includes the following specific steps: using a CNN to extract features from a plurality of (20 frames) of video frames, introducing an Attention structure, calculating the importance weight of each frame image through global average pooling and a Sigmoid activation function, weighting and summing the feature vectors using the Attention weight, obtaining a fusion feature vector, classifying the fusion features through a fully connected layer and a Softmax activation function, outputting the confidence of the classification result, using randomly initialized weights and an Adam optimizer, combining a cross-entropy loss function to train and optimize the model, and obtaining a pre-trained site identification model.

[0134] The synchronization unit 764 can finally realize real-time linkage between the bronchoscope operation process and the three-dimensional virtual map model during display on the display interface, and dynamically synchronize and display the real-time position and / or site section of the bronchoscope in the target navigation path in the three-dimensional virtual map model.

[0135] The segment recognition unit 762 further includes: a point cloud generation unit 7621 configured to generate a point cloud set based on the extracted center line of the 3D bronchial tree; and a classification unit 7622 configured to perform point cloud classification on the preprocessed point cloud set based on a pre-trained point cloud classification model, and obtain a classification result as the recognition result of the segment of the part.

[0136] The part segment recognition process includes: extracting a center line of a bronchial tree 3D mask according to a skeleton extraction algorithm during system operation, and performing a farthest point uniform sampling algorithm to select uniformly distributed points on the entire center line length to form a point cloud set, and inputting the point cloud set into a pre-trained point cloud classification model to obtain corresponding bronchial part classification information of each point.

[0137] The point cloud generation process includes performing a farthest point uniform sampling algorithm on the bronchial center line, taking a point on the center line as the first sampling point, selecting the farthest point in each iteration, calculating a new distance and updating the nearest distance each time to ensure that each point can find the nearest sampling point, thereby finding the farthest point, and when the required number of sampling points (2048) is reached, the algorithm ends. The attribute of each point is a three-tuple (x, y, z) of its spatial coordinates, and the category information is the part classification information of the bronchial tube.

[0138] Further, before inputting the point cloud data into the pre-trained point cloud classification model, the data usually needs to be preprocessed, such as point cloud downsampling and point cloud normalization. The preprocessed point cloud set is input into the pre-trained point cloud classification model. Based on the received three-dimensional point cloud data such as the coordinates of the point clouds in the point cloud set and possible additional features, the point cloud classification model outputs the probability distribution of the bronchial part segment shown by each point in the point cloud set based on the classification logic in the point cloud classification model. The final classification result of the part segment of the bronchial tube is determined by screening the maximum probability category of each point, or according to the model pre-design, an overall classification result may also be output, which identifies the part segment of the entire bronchial region.

[0139] The classification logic can include a PointMLP model that sequentially calculates through each layer to gradually extract and aggregate the features of the point cloud for the input point cloud data. The finally output feature vector is used for classification, and the model outputs the category of each point. The last layer of the model is usually a fully connected layer (or several layers) that maps the extracted features to the category space to produce the category prediction result of each point.

[0140] Specifically, the process of training the point cloud classification model includes: initializing the point cloud data input model, the model extracts the features of the point cloud through a neural network structure such as MLP, and finally obtains the class prediction value of each point or the entire point cloud. The class prediction value includes the probability distribution of the output of each class of bronchus, for example, the output can be normalized to probability using the softmax function; the difference between the model prediction value and the true label is calculated using the cross-entropy loss function; the class probability distribution output by the model and the real one-hot label are compared to generate a cross-entropy loss value; the gradient of the model parameters (i.e. the derivative of the loss function with respect to the model parameters) is calculated based on the value of the loss function through the back propagation algorithm. Among them, by adjusting the adjustment method of the weight parameters and bias parameters of the model to reduce the loss value. Specifically, the Adam optimizer is used, and the parameters of the model are updated based on the calculated gradient and momentum term and variance term; since Adam can dynamically adjust the learning rate, the model training is more stable and converges faster; the training process will be iterated multiple times. In each iteration, the model will be trained through multiple small batches of point cloud data, constantly optimizing the parameters so that the prediction results output by the model become more and more close to the true value.

[0141] For example, taking the PointMLP as the point cloud classification model for bronchial part segment classification as an example, its training process includes: inputting point cloud data, passing through the network layer of PointMLP, generating class prediction for each point, calculating the difference between the model prediction value and the true label using the cross-entropy loss, calculating the derivative of the loss function with respect to the network parameters, determining the adjustment direction of the parameters, and the Adam optimizer updates the weight parameters and bias parameters of the model based on the results of back propagation. Repeat iteration until the loss function value is reduced to a satisfactory level, so that the classification performance of the cloud classification model reaches the expectation.

[0142] The part mapping unit 763 further includes: a coordinate conversion unit 7631 that converts the real-time position of the bronchoscope identified to the real-time part into the display position of the corresponding part segment in the view coordinate system based on the relative relationship between the real-time coordinate system and the view coordinate system; and a view angle adjustment unit 7632 that adjusts the display view angle of the real-time part in the view coordinate system in which the part segment corresponding to the real-time part is located in the 3D bronchial tree based on the display position information of the part segment corresponding to the real-time part in the view coordinate system.

[0143] The site mapping unit 7631 includes a bronchoscope for real-time site recognition based on deep learning video understanding to realize site recognition under the bronchoscope. The specific implementation process includes pre-processing and feature extraction of images in the real-time video of the bronchoscope, then using a convolutional neural network (CNN) to capture spatial and temporal information in the video, and finally realizing real-time recognition and positioning of different parts of the bronchus. The position of the real-time site recognized under the bronchoscope in the real-time coordinate system is (x cam ,y cam ,z cam ), and the relative relationship between the real-time coordinate system and the VTK coordinate system displaying the 3D bronchial tree is a rotation matrix M. The coordinates (x cam ,y cam ,z cam ) of the real-time site recognized under the bronchoscope are converted from the real-time coordinate system to the VTK view coordinate system (x vtk ,y vtk ,z vtk ) using the rotation matrix M.

[0144] The adjustment process of the view angle adjustment unit 7632 includes setting the view matrix of the camera in the view coordinate system according to the coordinates (x vtk ,y vtk ,z vtk ) of the display position of the site segment in the view coordinate system of the converted 3D bronchial tree, specifying the position, view point (target direction), and upward direction (usually vertical direction) of the camera, aligning it to the center line axis of the 3D bronchial tree, ensuring that the VTK camera view angle in the view coordinate and the view angle of the real-time site corresponding to the image collected under the bronchoscope are consistent, so that the rendered virtual navigation picture accurately corresponds to the observation view angle under the bronchoscope.

[0145] The application further provides a bronchoscope view-based airway navigation method, which is applied to a bronchoscope view-based airway navigation system, as shown in Figure 9As shown, a flowchart of a bronchoscope view based airway navigation method is shown. The specific steps of the method include: step S802: reconstructing a 3D bronchial tree mask from a segmented CT image; step S804: obtaining a three-dimensional virtual map model including a 3D bronchial tree from the 3D bronchial tree mask; step S806: extracting a navigation path from a bronchial entrance to a target site based on the 3D bronchial tree; step S808: when the number of navigation paths is more than one, selecting one of the navigation paths as a target navigation path based on preset planning parameters; step S810: guiding the bronchoscope to move along the target navigation path in the three-dimensional virtual map model based on the visualized guide path; and step S812: mapping the recognized real-time site corresponding to the bronchoscope to the 3D bronchial tree for matching, to achieve dynamic synchronization of the real-time site corresponding to the bronchoscope and the target navigation path. The process of the method is consistent with the implementation of each module / unit of the foregoing bronchoscope view based airway navigation, and will not be described here again.

[0146] Therefore, the airway navigation system and method can reconstruct a 3D bronchial tree based on a CT image, generate a target navigation path based on a bronchial entrance and a target site, and match real-time images acquired under a bronchoscope with structural information of a 3D bronchial tree reconstructed based on a conventional CT image sequence, and establish synchronization of a real-time scene under the bronchoscope and a virtual map reconstructed based on the CT image, so that the bronchoscope can be moved along the target navigation path and the real-time position of the bronchoscope can be perceived in real time during bronchoscope navigation, to provide accurate navigation and positioning support for operation under the bronchoscope. The current bronchial site (e.g., which branch, which section) can be identified through the view under the bronchoscope, and the bronchial site can be corresponded to a section of the 3D bronchial tree. Therefore, the doctor can accurately view the current position and navigation progress on the three-dimensional virtual map model, misoperation can be effectively avoided, damage to normal airway tissue can be reduced, and the method is particularly suitable for surgical operation of a complex airway structure.

[0147] Further, the virtual map established based on the CT sequence image is used as a basis, and the optimal navigation path from the current bronchoscope position to the target site is used as the target navigation path based on preset planning parameters. The target navigation path planning considers the shape of the airway and possible obstacles, so that the doctor can quickly reach the target site according to the planned optimal operation path under the guidance of the target navigation path when operating the bronchoscope, to save operation time and reduce operation complexity.

[0148] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications of the present application belong to the scope of the claims of the present application and their equivalents, the present application also intends to include these modifications and variations.

Claims

1. An airway navigation system based on bronchoscopic view, characterized in that: The system comprises: Reconstruction module, which reconstructs the bronchial tree 3D mask from CT images; a model generation module, which obtains a three-dimensional virtual map model including a 3D bronchial tree according to the bronchial tree 3D mask; a path extraction module, which extracts a navigation path from the bronchial entrance to the target site based on the 3D bronchial tree; a path selection module, which selects one of the navigation paths as a target navigation path based on preset planning parameters, the path selection module including a path evaluation unit, which evaluates a path score of each of the navigation paths based on a weight of the preset planning parameters and selects the navigation path with the highest path score as the target navigation path; a path planning module, which generates the preset planning parameters based on the path length of the navigation path, the entry angle when contacting the target part, the entry distance when contacting the target part, the airway diameter at the end position of the navigation path, and the blood vessel density at the end position of the navigation path; a navigation module, which guides the bronchoscope to move along the target navigation path of the three-dimensional virtual map model based on a visualized guidance path; A navigation auxiliary module maps the identified real-time position corresponding to the bronchoscope to the 3D bronchial tree for matching, so as to achieve dynamic synchronization between the real-time position corresponding to the bronchoscope and the target navigation path.

2. The bronchoscopic view-based airway navigation system according to claim 1, characterized in that: The reconstruction module includes: An input preparation unit is used to select a data block that has been divided and normalized from a CT image sequence corresponding to the bronchus as an input object; a prediction unit, configured to input the input object into a pre-trained prediction model to predict bronchial voxels; An output unit is used to perform splicing and restoration based on the bronchial voxels to obtain a 3D mask of the bronchial tree.

3. The bronchoscopic view-based airway navigation system according to claim 1, characterized in that: The model generation module includes: a preprocessing unit, configured to perform morphological restoration and boundary smoothing preprocessing on the bronchial tree 3D mask; A visualization unit is configured to visualize the 3D bronchial tree and the three-dimensional virtual map model based on the preprocessed bronchial tree 3D mask.

4. The bronchoscopic view-based airway navigation system according to claim 1, characterized in that: The path planning module includes the following units: a path length calculation unit, configured to calculate the shortest path from the bronchial entrance to the target site in the navigation path; An angle calculation unit is used to calculate the angle between the airway direction of the end point of the navigation path and the surface normal of the target part; Entering a distance calculation unit for calculating the shortest distance from the end point of the navigation path to the target part; an airway diameter calculation unit, configured to calculate the airway cross-sectional diameter at the end point of the navigation path; The blood vessel density calculation unit is used to calculate the blood vessel density value at the end point of the navigation path.

5. The bronchoscopic view-based airway navigation system according to claim 1, characterized in that: The navigation module includes: a structuring unit, configured to perform structural processing on the centerline point set of the 3D bronchial tree to generate a centerline tree; A guidance unit is used to generate a guidance path from the root node of the centerline tree to the target part according to the tree structure information of the centerline tree.

6. The bronchoscopic view-based airway navigation system according to claim 1, characterized in that: The navigation assistance module includes: a part recognition unit, which recognizes a real-time part corresponding to a real-time image acquired by the bronchoscope at a current real-time position based on a pre-trained part recognition model; a segment identification unit, which identifies the location segments of the 3D bronchial tree based on a pre-trained point cloud classification model; a part mapping unit, configured to map the identified real-time part to the part segment corresponding to the 3D bronchial tree; A synchronization unit is used to dynamically synchronize the mapping result of the part mapping unit when the bronchoscope moves along the target navigation path.

7. The bronchoscopic view-based airway navigation system according to claim 6, characterized in that: The segment identification unit includes: a point cloud generation unit, which generates a point cloud set based on the centerline of the extracted 3D bronchial tree; The classification unit classifies the preprocessed point cloud set based on the pre-trained point cloud classification model and obtains the classification result as the recognition result of the part segment.

8. The bronchoscopic view-based airway navigation system according to claim 6, characterized in that: The part mapping unit includes: a coordinate conversion unit, which converts the real-time position of the real-time part identified by the bronchoscope into a display position of the corresponding part segment in the view coordinate system based on the relative relationship between the real-time coordinate system and the view coordinate system; A viewing angle adjusting unit adjusts a display viewing angle of the real-time part in the view coordinate system where the 3D bronchial tree is located based on display position information of the part segment corresponding to the real-time part in the view coordinate system.

Images