Airway navigation system and method based on bronchoscope view

Abstract

The invention provides an airway navigation system and method based on a bronchoscope view, and the system comprises a reconstruction module which obtains a bronchial tree 3D mask through CT image reconstruction; the model building module is used for obtaining a three-dimensional virtual map model comprising a 3D bronchial tree according to the 3D mask of the bronchial tree; the path extraction module is used for extracting all possible navigation paths from the bronchus entrance to the target part based on the 3D bronchus tree; the path selection module is used for selecting one navigation path as a target navigation path; the navigation module is used for guiding the bronchoscope to move along a target navigation path in the three-dimensional virtual map model based on a visual guide path; and the navigation auxiliary module maps the identified real-time part corresponding to the bronchoscope to the 3D bronchial tree for matching so as to realize dynamic synchronization of the real-time part corresponding to the bronchoscope and the target navigation path. According to the invention, reliable and convenient navigation in a complex airway network can be realized, misoperation is effectively avoided, time is saved, and operation difficulty is reduced.

Description

Technical Field

[0001] The present invention relates to the field of medical navigation planning, and in particular to an airway navigation system and method based on a bronchoscope view. Background Art

[0002] Typically, when examining the human airway, medical personnel will need to check most or all parts of the lung tree, such as the trachea, left and right bronchi, and their respective bronchioles and alveoli, to check for any abnormalities. In other cases, it is desirable to study specific parts of the human airway (such as specific bronchioles), such as based on magnetic resonance (MR) or computed tomography (CT) scan results, including diagnosis under a bronchoscopic lens and pathological diagnosis after biopsy and puncture.

[0003] However, when navigating through the various parts of the human airway, medical personnel usually rely on experience to navigate the bronchoscope through the human airway, for example, to reach most / all parts, and / or specific parts of the lung tree based on the camera image from the bronchoscope. Because the various parts of the human airway (such as each bronchi or each bronchiole) usually look very similar, there is a risk of errors, such as not reaching the desired part of the human airway or mistaking a part of the airway for a different part of the airway. This in turn increases the risk that the patient is not properly examined.

[0004] In order to improve the accuracy of the examination, the existing scheme uses computer graphics technology to establish a three-dimensional virtual navigation map based on bronchial CT images, and during the operation, the three-dimensional virtual navigation map is used to guide the bronchoscope operation process. This virtual navigation cannot automatically correspond the image under the bronchoscope to the position in the three-dimensional virtual navigation map. The doctor must rely on experience to judge the position of the bronchoscope by comparing the three-dimensional virtual navigation map and the current perspective of the bronchoscope. The operation process requires manual operation following the guidance of virtual roaming. This method requires high real-time response ability and operation accuracy of the doctor. At the same time, during the operation, the patient's anatomical structure may change due to factors such as breathing and tissue displacement. When the real-time position of the bronchoscope cannot be identified, there will be a situation where the navigation path cannot be adjusted in real time to adapt to these changes. Therefore, how to achieve continuous real-time synchronization of the target navigation path based on the three-dimensional virtual navigation map and the real-time position identified by the bronchoscope perspective is still a challenge, especially in dynamic operations. How to maintain the accuracy and real-time performance of the real-time navigation route of the bronchoscope during the entry process still needs to be solved.

[0005] The prior art CN112598639A discloses the use of a splicing point algorithm for bronchial tree path planning, which can quickly and conveniently plan an automatic roaming path to the end point of any branch after obtaining the complete centerline of the bronchial tree. However, it does not solve the problem of the inability to continuously synchronize the target navigation path of the bronchoscope navigation in the three-dimensional virtual navigation map screen with the real-time position identified in the actual viewing angle when operating the bronchoscope in real time.

[0006] The prior art WO2023124978A1 determines the real position of the real bronchoscope in the real bronchial tree based on the similarity between the virtual bronchoscope image corresponding to the virtual bronchoscope and the real bronchoscope image collected by the real bronchoscope, and determines the virtual bifurcation local image, so as to quickly and effectively synchronize the virtual bronchoscope image with the real bronchoscope image. However, the reliability of directly using the image similarity comparison method is low, especially during the operation, when the patient's anatomical structure may change due to factors such as breathing and tissue displacement, the matching fails, the image cannot be displayed synchronously or the display result is inaccurate. Summary of the invention

[0007] In order to solve the problem in the prior art of the inability to continuously synchronize in real time the location and segment of the bronchial tree and the real-time location identified by the actual viewing angle when operating the bronchoscope in the three-dimensional virtual map model interface during the movement of the bronchoscope according to the target navigation path, the present invention provides an airway navigation system and method based on bronchoscope view, so as to help doctors navigate more reliably and conveniently in a complex airway network.

[0008] The first aspect of the present application provides an airway navigation system based on a bronchoscopic view, the system comprising:

[0009] The reconstruction module reconstructs the bronchial tree 3D mask from the CT image;

[0010] A model generation module, which obtains a three-dimensional virtual map model corresponding to the 3D bronchial tree according to the bronchial tree 3D mask;

[0011] a path extraction module, which extracts a navigation path from the bronchial entrance to the target site based on the 3D bronchial tree;

[0012] A path selection module, which selects one of the navigation paths as a target navigation path based on preset planning parameters;

[0013] A navigation module, which guides the bronchoscope to move along the target navigation path in the three-dimensional virtual map model based on the visualized guidance path;

[0014] A navigation auxiliary module maps the identified real-time position corresponding to the bronchoscope to the 3D bronchial tree for matching, so as to achieve dynamic synchronization between the real-time position corresponding to the bronchoscope and the target navigation path.

[0015] Furthermore, the reconstruction module includes:

[0016] An input preparation unit is used to select a data block that has been divided and normalized from a thin-layer CT image sequence corresponding to the bronchus as an input object;

[0017] A prediction unit, used for inputting the input object into a pre-trained prediction model to predict bronchial voxels;

[0018] An output unit is used to perform splicing and restoration based on the bronchial voxels to obtain a 3D mask of the bronchial tree.

[0019] Furthermore, the model generation module includes:

[0020] A preprocessing unit, used for preprocessing the bronchial tree 3D mask by morphological restoration and boundary smoothing;

[0021] A visualization unit is used to realize the visualization of the bronchial tree and the three-dimensional virtual map model based on the preprocessed bronchial tree 3D mask and VTK visualization technology.

[0022] Furthermore, the airway navigation system also includes: a path planning module, which generates the preset planning parameters based on at least two of the path length of the navigation path, the entry angle when contacting the target part, the entry distance when contacting the target part, the airway diameter at the end position of the navigation path, and the blood vessel density at the end position of the navigation path.

[0023] Furthermore, the path planning module includes at least two of the following units:

[0024] A path length calculation unit, used for calculating the shortest path from the bronchial entrance to the target site in the navigation path;

[0025] Entering an angle calculation unit, used to calculate the angle between the airway direction of the end point position of the navigation path and the surface normal of the target part;

[0026] Entering a distance calculation unit, for calculating the shortest distance from the end position of the navigation path to the target part;

[0027] an airway diameter calculation unit, used to calculate the airway cross-sectional diameter at the end point of the navigation path;

[0028] The blood vessel density calculation unit is used to calculate the blood vessel density value at the end point of the navigation path.

[0029] Furthermore, the path selection module includes:

[0030] The path evaluation unit evaluates the path score of each of the navigation paths based on the weight of the preset planning parameters, and selects the navigation path with the highest path score as the target navigation path.

[0031] Furthermore, the navigation module includes:

[0032] A structuring unit, used for performing structural processing on the centerline point set of the 3D bronchial tree to generate a centerline tree;

[0033] A guidance unit is used to generate a guidance path from the root node of the centerline tree to the target part according to the tree structure information of the centerline tree.

[0034] Furthermore, the navigation assistance module includes:

[0035] A part recognition unit, which recognizes a real-time part corresponding to a real-time image acquired by the bronchoscope at a current real-time position based on a pre-trained part recognition model;

[0036] A segment identification unit, which identifies the location segment of the 3D bronchial tree based on a pre-trained point cloud classification model;

[0037] A part mapping unit, configured to map the identified real-time part to the part segment corresponding to the 3D bronchial tree;

[0038] A synchronization unit is used to dynamically synchronize the mapping result of the part mapping unit during the movement of the bronchoscope along the target navigation path.

[0039] Furthermore, the segment identification unit includes:

[0040] A point cloud generation unit, for generating a point cloud set based on the centerline of the extracted 3D bronchial tree;

[0041] The classification unit is used to classify the preprocessed point cloud set based on the pre-trained point cloud classification model, and obtain the classification result as the recognition result of the part segment.

[0042] Further, the part mapping unit includes:

[0043] A coordinate conversion unit, which converts the real-time position of the real-time part identified by the bronchoscope into a display position of the corresponding part segment in the view coordinate system based on the relative relationship between the real-time coordinate system and the view coordinate system;

[0044] A viewing angle adjustment unit adjusts a display viewing angle of the real-time part in the view coordinate system where the real-time part is located in the 3D bronchial tree based on display position information of the part segment corresponding to the real-time part in the view coordinate system.

[0045] A second aspect of the present application provides an airway navigation method based on a bronchoscope view, which is applied to the aforementioned airway navigation system based on a bronchoscope view, and the method comprises:

[0046] Reconstruct the bronchial tree 3D mask from CT images;

[0047] Obtaining a three-dimensional virtual map model corresponding to the 3D bronchial tree according to the bronchial tree 3D mask;

[0048] Extracting a navigation path from the bronchial entrance to the target site based on the 3D bronchial tree;

[0049] Selecting one of the navigation paths as a target navigation path based on preset planning parameters;

[0050] Guiding the bronchoscope to move along the target navigation path in the three-dimensional virtual map model based on the visualized guidance path;

[0051] The real-time position corresponding to the identified bronchoscope is mapped to the 3D bronchial tree for matching, so as to achieve dynamic synchronization between the real-time position corresponding to the bronchoscope and the target navigation path.

[0052] The airway navigation system disclosed in the present application can reconstruct a 3D bronchial tree based on CT images, generate a target navigation path based on the bronchial entrance and the target site, and match the real-time image acquired under the bronchoscope with the structural information of the 3D bronchial tree reconstructed based on the traditional CT sequence, and establish dynamic synchronization between the real-time scene under the bronchoscope and the target navigation path in the three-dimensional virtual map model reconstructed based on CT, so as to realize the ability to move along the target navigation path and perceive the real-time position of the bronchoscope in real time during the bronchoscope navigation process, and provide accurate navigation and positioning support for the bronchoscope operation. Therefore, it can ensure that the doctor can accurately view the dynamic synchronization status of the specific position identified by the current bronchoscope and the moving progress of the target navigation path on the three-dimensional virtual map model, which can effectively avoid misoperation and reduce damage to normal airway tissue, and is particularly suitable for surgical operations with complex airway structures.

[0053] Furthermore, a three-dimensional virtual map model based on CT sequence images is used as a basis, and the optimal navigation path from the current bronchoscope position to the target site using preset planning parameters in the three-dimensional virtual navigation model is used as the target navigation path. The target navigation path planning takes into account the shape of the airway and possible obstacles, etc., so that the doctor can quickly reach the target site according to the planned optimal surgical path under the guidance of the target navigation path when operating the bronchoscope, thereby saving operation time and reducing operation complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 A schematic structural diagram of a bronchoscope 1 is shown;

[0056] Figure 2 A computer device in which a bronchoscopic view-based airway navigation system is embedded is shown;

[0057] Figure 3 A system diagram of a bronchoscopic view-based airway navigation system is shown;

[0058] Figure 4 shows a system block diagram of the reconstruction module;

[0059] Figure 5 A system block diagram of a model generation module is shown;

[0060] Figure 6 A system diagram of an airway navigation system based on another bronchoscopic view is shown;

[0061] Figure 7 shows a system block diagram of a navigation module;

[0062] Figure 8 A system block diagram of a navigation assistance module is shown;

[0063] Fig. 9 A schematic flow chart of the bronchoscopic view-based airway navigation method is shown. DETAILED DESCRIPTION

[0064] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0065] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0066] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0067] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0068] A bronchoscope typically includes an image capture device, such as a camera, located at the front end of the bronchoscope to be inserted into a patient and connected to a display to provide medical personnel with a view of the portion of the airway where the front end of the bronchoscope is located.

[0069] Among them, the bronchial classification can be specifically: after the trachea enters the chest cavity, it is divided into the left and right main bronchi. The right main bronchus is divided into the upper lobe bronchus and the middle segment bronchus, the middle segment bronchus is further divided into the middle lobe and lower lobe bronchus, the left main bronchus is divided into the upper lobe and lower lobe bronchus, and the left upper lobe bronchus branches into the lingual segment bronchus. In this way, the right lung is divided into three lobes, the upper, middle and lower lobes, and the left lung is divided into two lobes, the upper and lower lobes. These bronchi are further divided into segmental and subsegmental bronchi, terminal bronchioles, respiratory bronchioles, alveolar ducts, alveolar sacs and alveoli.

[0070] See also Figure 1, shows a schematic structural diagram of a bronchoscope 1. The bronchoscope may be disposable or reusable. The bronchoscope 1 includes an insertion portion 2. A handle 3 for operating the bronchoscope 1 is arranged at the base end of the insertion portion 2. A head portion 5 is provided at the front end of the insertion portion 2. The camera assembly 6 is positioned in the end portion 5 and is configured to transmit an image signal to a display 11 through a processing device 12 of the bronchoscope 1. The display 11 may allow an operator to view a real-time image captured by the camera assembly 6 of the bronchoscope 1 and a navigation path in a virtual navigation image of the inside of the bronchus generated based on a CT image to assist in the target navigation path of the bronchoscope operation.

[0071] The display 11 is coupled to the processor and is operable to render an image of the patient's anatomy. Such an image may be based on a set of images obtained before surgery (e.g., CT or MRI scans, 3D mapping, etc.). The view of the patient's anatomy provided by the display 11 may also be dynamically changed based on the current identified real-time location. For example, as the bronchoscope 1 moves within the patient's body, data related to its real-time position may cause the processor to update the view of the patient's anatomy in the display 11 in real time to depict the area of ​​the patient's anatomy captured by the camera assembly 6 of the bronchoscope 1.

[0072] In one embodiment, see Figure 2 , the airway navigation system based on bronchoscopic view can be embedded in Figure 2 In the computer device shown. Among them, the computer device can be a separately set computer device, or it can be a related computer device in the navigation system. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an airway navigation method based on a bronchoscopic view is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic display screen, etc., and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0073] See also Figure 3, showing a system diagram of an airway navigation system based on a bronchoscope view. The airway navigation system 7 includes: a reconstruction module 71, which reconstructs a 3D mask of the bronchial tree through segmented CT images; a model generation module 72, which constructs a three-dimensional virtual map model corresponding to a displayable 3D bronchial tree according to the bronchial tree 3D mask and displays the three-dimensional virtual map model; a path extraction module 73, which extracts a navigation path from the bronchial entrance to the target site based on the 3D bronchial tree; a path selection module 74, which selects one of the navigation paths as the target navigation path based on preset planning parameters; a navigation module 75, which guides the bronchoscope to move along the corresponding target navigation path in the three-dimensional virtual map model based on a visualized guidance path; and a navigation assistance module 76, which maps the identified real-time site corresponding to the bronchoscope to the 3D bronchial tree for matching, so as to achieve dynamic synchronization between the real-time site corresponding to the bronchoscope and the target navigation path.

[0074] The 3D bronchial tree is a mask used to represent the bronchial tree structure in three-dimensional space. In the bronchial tree 3Dmask, the mask is a three-dimensional data set, in which the value of each volume pixel can be used to indicate whether it belongs to the bronchial tree structure.

[0075] The airway navigation system 7 performs segmentation processing on the acquired CT sequence images based on the reconstruction module and the model generation module to reconstruct the bronchial tree 3D mask, and further performs morphological hollowing processing on the bronchial tree 3D mask to obtain a 3D bronchial tree model with a cavity. The 3D bronchial tree with the cavity model is then imported into the VTK world coordinate system, and then multiple part segments in the reconstructed 3D bronchial tree are identified on the three-dimensional segmentation results, and the corresponding bronchial part segment identification results are temporarily stored. At the same time, the VTK visualization construction tool is used to construct a three-dimensional virtual map model of the bronchial tree from the three-dimensional segmentation results, and the positions of each bronchial part in the reconstructed 3D bronchial tree form a one-to-one mapping relationship with the positions in the three-dimensional virtual map model. At the same time, the system receives the video stream output by the computer device (bronchoscope host), and through deep learning video understanding technology, it identifies the real-time position corresponding to the real-time image collected at the current real-time position of the bronchoscope in real time. After identifying the real-time position under the bronchoscope, it establishes a corresponding mapping relationship with the position segment recognition result of the 3D bronchial tree, thereby indirectly establishing a linkage with the navigation path in the three-dimensional virtual map model. When the bronchoscope moves in the bronchial tree, the corresponding area on the three-dimensional virtual map model is highlighted or indicated, or the view of the current position segment is played in real time in the display interface to assist the navigation process of the bronchoscope, so that the doctor can perceive the current position of the bronchoscope during the navigation process.

[0076] See also Figure 4 , showing a system block diagram of the reconstruction module. The reconstruction module 71 includes: an input preparation unit 711, which is used to select thin-layer data that has been divided and normalized from the thin-layer CT image sequence corresponding to the bronchus as an input object; a prediction unit 712, which is used to input the input object into a pre-trained prediction model to predict bronchial voxels; an output unit 713, which splices and restores the bronchial area including the bronchial voxels to output a bronchial tree 3D mask. The model training and iteration process of the prediction model will be described in detail below.

[0077] The input preparation unit 711 is used to select a data block that has been divided and normalized from the thin-layer CT image sequence corresponding to the bronchus as an input object. The thin-layer CT image sequence is formed by multiple slice images. Each frame of CT image obtained in the CT examination of the bronchus can be a cross-section of the bronchus. First, its thin-layer CT image data is selected as the analysis object, and each CT image is divided into multiple small blocks, and two-dimensional blocks are performed along the xy plane. At the same time, multiple slices in the z direction are combined to form three-dimensional small blocks, and the spatial proximity relationship of the CT data and the depth information are fully considered. For example, the block division is performed in a 64x64x64 size and a step size of 16 to form multiple sub-blocks. After the block division, the magnitude difference of the image gray value is eliminated by normalization processing methods such as linear normalization or Z-Score.

[0078] Moreover, in the CT images obtained by imaging examinations, the bronchi often occupy a small area of ​​the CT images. If the CT sequence of three-dimensional volume data is directly processed in the entire CT image, the amount of calculation is very large. However, the present application adopts the method of dividing the CT image into multiple sub-blocks (patches), which can effectively reduce the calculation scope and focus on local area features for further analysis.

[0079] The prediction unit 712 is used to input the data block into the pre-trained prediction model for inference prediction, and screen out voxels that meet the bronchial location characteristics in the prediction analysis process based on the prediction model score. Specifically, for voxels whose output results are greater than or equal to the preset threshold, they are determined to be bronchial voxels, and for voxels less than the preset threshold, they are considered to be other backgrounds. The corresponding output mask of the data block is obtained. Specifically, when the prediction score obtained by the prediction model is in the range of 0 to 1, the preset threshold can be set to 0.5. When the prediction score is greater than or equal to 0.5, the corresponding voxel is considered to be a bronchial voxel, otherwise it is considered to be a background or other tissue-related voxel.

[0080] After all the input blocks of all the relevant data of the bronchial reconstruction before output are input into the prediction model and the prediction is completed, the output unit 713 recombines, restores and splices the output blocks including bronchial voxels, background, other tissues, etc. output in the prediction model according to the spatial position of the original CT data to obtain the prediction result of the bronchial tree 3D mask based on the 3D sequence. The bronchial tree 3D mask can include the branch structure of the entire bronchial system from the main bronchi to the lobar bronchi, segmental bronchi and even finer bronchial branches in a detailed three-dimensional representation.

[0081] In one embodiment, when an overlapping area is introduced in the block division process of the input preparation unit 711, the overlapping parts in different sub-blocks can be restored by assigning different weight values ​​to the overlapping parts in different sub-blocks through weighted averaging. For example, the center of the block is used as a reference point, and the part close to the center of the block is processed in a manner of having a large weight and a small weight close to the edge, so as to ensure that the spliced ​​3D mask image has no obvious boundaries.

[0082] Among them, thin-layer data means that the thickness of each slice in the CT scan is very thin (usually not more than 1 mm), and the slice contains at least the main bronchi and smaller airways. Thin-layer data helps to obtain high-resolution images of details within the bronchi. By selecting thin-layer data, redundant information in the data can be reduced and the accuracy of local features can be enhanced.

[0083] The training process of the prediction model mainly includes selecting the entire bronchial structure CT data, screening the CT data that meets the layer thickness support (thin layer) as the annotation object, and manually annotating the main bronchi, secondary bronchi and their branches, bronchial regional boundaries, etc. that can form the bronchial tree mask. After the manual annotation is completed, the training set, validation set, and test set are divided into training sets, validation sets, and test sets in units of CT sequences. The number of training sets, validation sets, and test sets is 8:1:1. The 3D segmentation network model Vnet is constructed. The input size of the network model is set to 64x64x64, and the size of the output layer is also 64x64x64. The final output layer of the network is activated by the sigmoid function, and the output range of the output layer output value is between 0 and 1, representing the probability value of the voxel corresponding to the input data belonging to a certain part of the bronchial structure.

[0084] Since the dimension of a CT sequence is generally 512x512xZ or 1024x1024xZ (Z is the sequence depth), it cannot be input into the network model for a row calculation and training, and needs to be cut into blocks. The entire process can refer to the network input size of Vnet, and the original 3DCT data and the corresponding annotated 3Dmask are divided into blocks according to the size of 64x64x64 and the step size of 16 to obtain the training data and corresponding labels that meet the network input, thereby improving the stability and generalization ability of the pre-trained prediction model. At the same time, the training data is normalized to between 0 and 1 according to the window width WW=1500 and the window level WL=-400. The normalization formula is as follows:

[0085] Where x is the original CT value and f(x) is the normalized result:

[0086]

[0087] The entire model training process also includes randomly initializing the network model, setting dice loss as the target loss function for network model training, and using the Adam optimizer to optimize the model parameters to obtain a pre-trained prediction model.

[0088] See also Figure 5 , showing a system block diagram of the model generation module. The model generation module 72 includes: a preprocessing unit 721, which is used to perform morphological repair and boundary smoothing preprocessing on the bronchial tree 3D mask; a visualization unit 722 can realize the visualization of a three-dimensional virtual map model including a 3D bronchial tree based on the preprocessed bronchial tree 3D mask. The 3D bronchial tree displayed in the three-dimensional virtual map model annotates the internal structure and path information of the bronchus, can intuitively display the anatomical structure of the 3D bronchus, and can perform path extraction, path planning generation, etc. on this basis.

[0089] The preprocessing unit 721 is used to reduce noise points and smooth the bronchial boundaries by using boundary smoothing processing such as smoothing filtering (such as Gaussian filtering) after the bronchial tree 3D mask is generated, in order to reduce the problem of noise or inaccurate segmentation in the mask data in the bronchial tree 3D mask; extract the internal cavity (cavity) part through morphological operations and correct the small holes or tiny protrusions in the mask through morphological repair processing such as expansion and corrosion, so that the holes inside the bronchus can be filled, making the bronchial structure more coherent, so as to ensure the integrity of the reconstructed 3D bronchial tree.

[0090] The visualization unit 722 is used to realize the visualization of the three-dimensional virtual map model including the 3D bronchial tree by means of VTK visualization, and can calculate all feasible navigation paths that can reach the target part based on the specific position and morphological information of the target part through the VTK path planning algorithm, and the navigation path can be one or more. By performing appropriate scaling and rotation adjustment to map to the VTK virtual coordinate system to a suitable perspective, and providing interactive operations, the user can rotate, scale and translate the bronchial tree model in the three-dimensional virtual map model, so as to observe the structure of the 3D bronchial tree at different angles and levels.

[0091] Specifically, the 3D bronchial tree is converted into a model in the corresponding coordinate system of VTK through VTK visualization. All corresponding positions of the 3D bronchial tree will be adjusted according to the corresponding scaling and rotation rules to form a three-dimensional virtual map model of the 3D bronchial tree. When the coordinates of a point of the original 3D bronchial tree directly reconstructed are (x, y, z), after the rotation transformation matrix M, M includes the rotation and scaling coefficients and the translation amount, the coordinates (x', y', z') in the VTK coordinate system are obtained. The specific rotation transformation M can be set according to the actual display requirements of the VTK coordinate system. This ensures that the display angle and scale of the corresponding three-dimensional virtual map model of the 3D bronchial tree in the VTK coordinate system are accurate and controllable.

[0092] Therefore, the bronchial tree 3D mask is generated through surface reconstruction methods, morphological hollowing processing, etc., including the geometric surface and internal structure three-dimensional model of the 3D bronchial tree with a cavity, which is imported into the coordinate system corresponding to VTK, and a three-dimensional virtual map model of the target part is provided in the VTK coordinate system in a visual way.

[0093] The path extraction module 73 automatically identifies and extracts multiple possible navigation paths from the bronchial entrance to the target part (such as lesions, biopsy points, etc.) in the three-dimensional virtual map model based on the reconstructed 3D bronchial tree. These navigation paths may contain multiple branches and bending points. The extraction of these paths is based on the reconstructed 3D bronchial tree and the position data of the target part, as long as it can reach the target part from the bronchial entrance.

[0094] In some embodiments, the path selection module 74 may also use a path search algorithm such as the Dijkstra algorithm, the A* algorithm, etc. to identify all navigation paths of the target navigation path from the bronchial airway entrance to the target site. When there is more than one navigation path, the shortest path is selected as the path selection criterion by calculating the path length.

[0095] In one embodiment, see Figure 6, showing a system diagram of an airway navigation system based on another bronchoscope view. In order to further improve the reliability of the path selected by the path selection module, the airway navigation system 7 also provides a path planning module 77, which generates the preset planning parameters based on at least two of the path length from the navigation path to the target part, the entry angle of the target part, the entry distance of the target part, the airway diameter at the end position of the navigation path, and the blood vessel density at the end position of the navigation path.

[0096] Path Planning Module The path planning module 77 generates the planning parameters required for the selected target navigation path by including at least two calculation results of the following units in order to obtain the preset planning parameters, specifically including:

[0097] The path length calculation unit is used to calculate the shortest path from the bronchial entrance to the target site in the navigation path. The calculation of the path length can be performed by using the K shortest path algorithm to recursively find other navigation paths based on the shortest path found, or by using a depth-first search (DFS) to traverse all paths in the graph, and at the same time, combining the backtracking method to return to the previous bifurcation point after finding one of the paths to find other paths. This can effectively calculate all feasible paths from the bronchial entrance to the target site, and when multiple navigation paths appear, calculate the length of each path. A shorter path length usually means a shorter operation time, reducing patient discomfort and surgical risks.

[0098] The entry angle calculation unit is used to calculate the angle between the airway direction at the end position of the navigation path and the surface normal of the target part. The angle can indicate whether the path to the target part is steep or smooth. The angle θ can be calculated based on the dot product of the airway direction vector at the end of the path and the surface normal vector of the target part, and the modulus length of the path direction vector and the surface normal of the target part. For each possible path, the airway direction vector Vpath at the end of the path is calculated. The direction vector Vpath is defined by the point before the end of the path Pn-1 and the end point Pn:

[0099] Vpath=(Xn-Xn-1,Yn-Yn-1,Zn-Zn-1)

[0100] According to the target part extracted from the reconstructed 3D bronchial tree, the normal vector Nlesion of the target part surface is calculated, and the final angle θ at the position close to the end point of the path at the target part (such as lesion, biopsy point, etc.) is calculated:

[0101]

[0102] Among them, Vpath·Nlesion is the dot product of the path end point direction vector and the surface normal vector of the target part;

[0103] |Vpath||Nlesion| are the modulus lengths of the airway direction vector at the end of the path and the surface normal of the target part, respectively.

[0104] It is understandable that a smaller angle θ helps the bronchoscope or other tools approach the target site in the appropriate direction, increasing the success rate of biopsy. The smaller the angle θ, the closer the path entry angle is to the surface normal of the target site (i.e., vertical entry into the lesion), and theoretically the success rate of the operation is higher. Generally, an angle of no more than 30° is considered a smaller angle. For example, when using biopsy forceps to collect biopsies, the best entry angle of the biopsy forceps is 0°. At this time, the biopsy forceps enters the target site vertically, which is the most convenient and simple for doctors to perform biopsy operations.

[0105] In actual operation, if no path can reach the ideal angle, the doctor will usually choose a path with an angle θ not exceeding 60 degrees to meet the operability and accuracy as much as possible.

[0106] Enter the distance calculation unit, which is used to calculate the shortest distance from the end position of the navigation path to the target site. A smaller end distance usually represents a higher surgical success rate and biopsy accuracy, making it easier for doctors to obtain diseased tissue samples.

[0107] The distance from the end point of the navigation path to the target site (such as lesions, biopsy points, etc.) can be achieved by calculating the Euclidean distance between the two. The specific formula is:

[0108]

[0109] Where (x1, y1, z1) are the coordinates of the end point of the path, and (x2, y2, z2) are the coordinates of the center point of the target site (such as lesion, biopsy point, etc.).

[0110] In one embodiment, if the surface of the target part is complex and irregular, the end point of the navigation path may be projected onto the surface of the target part, and the distance from the projection point to the target part may be calculated.

[0111] The airway diameter calculation unit is used to calculate the airway cross-sectional diameter at the end position of the navigation path. A larger airway diameter facilitates the smooth passage of a bronchoscope and other tools, reducing the difficulty of operation.

[0112] In one embodiment, when the cross-sectional shape of the target part is relatively regular as a whole, the airway cross-sectional diameter at the end point of the navigation path can be calculated by measuring the maximum diameter at the end point of the navigation path, that is, the width of the airway as the airway cross-sectional diameter.

[0113] In one embodiment, when the cross-sectional shape of the target part is irregular or complex as a whole, the calculation of the airway cross-sectional diameter at the end point of the navigation path can also be performed by obtaining the maximum diameter and minimum diameter of the airway cross-sectional area measured along the vertical direction of the airway, and calculating the average of the maximum diameter and the minimum diameter as the airway diameter at the end point.

[0114] The vascular density calculation unit is used to calculate the vascular density value of the end point of the navigation path. High vascular density may indicate a high risk in the surgical area because puncture or operation may cause bleeding; lower density means a lower surgical risk. After vascular reconstruction based on CT images, the vascular density is calculated by the number and distribution of blood vessels near the end point of the navigation path. The specific calculation method is:

[0115] Set a 3D volume V near the end point of the navigation path total The total number of blood vessels detected in the area or the proportion of the volume occupied by the blood vessels are counted. The density value D can be expressed as the number of blood vessels per unit volume or the proportion of the blood vessel volume. When the density value does not exceed 5%, it is considered to be a low blood vessel density area. When the density value is within the range of 10%-30%, it is considered to be a medium-density area. If the range exceeds 30%, it is considered to be a high-density area. High-density blood vessels may increase the risk of surgery. Therefore, the path in the low blood vessel density area is safer.

[0116] In the present application, the blood vessel reconstruction based on the CT image can be implemented in the same manner as the reconstruction of the 3D bronchial tree, which will not be described in detail here.

[0117] The path selection module 74 further includes a path evaluation unit, which evaluates the path score of each navigation path based on the weight of the preset planning parameters, and selects the navigation path with the highest path score as the target navigation path. Generating a target navigation path usually requires a comprehensive evaluation of the values ​​of each preset planning parameter in the navigation path, and by evaluating the feasibility and safety of the path, ensures that the selected target navigation path avoids passing through fragile tissues and important blood vessels, while maximizing the linearity and visibility of the path.

[0118] For example, the preset planning parameters selected during evaluation include: path length and entry angle, the path length weight is set to 60%, the entry angle weight is set to 40%, the path length is less than 50mm, the score is 100, the score is 80 in the range of 50mm-100mm, the score is 60 in the range of 100mm-150mm, the score is 40 in the range of 150mm-200mm, and the score is 20 in the range of 200-250mm; the score is 100 in the range of entry angle 0°-20°, 80 in the range of 20°-30°, 60 in the range of 30°-40°, 40 in the range of 40°-60°, and 20 greater than 60°.

[0119] When three target paths A, B, and C are extracted, the path length of path A is 188 mm, and the entry angle is 18 degrees; the path length of path B is 136 mm, and the entry angle is 45 degrees; the path length of path B is 207 mm, and the entry angle is 32 degrees. The evaluation is performed according to the path evaluation unit 741:

[0120] Path A score = (40 × 60%) + (100 × 40%) = 64

[0121] Path B score = (60 × 60%) + (40 × 40%) = 52

[0122] Path C score = (20 × 60%) + (60 × 40%) = 36

[0123] Therefore, path A is selected as the target navigation path at this time. Although the entry angle is slightly larger during biopsy, the overall path length is the shortest.

[0124] Furthermore, when multiple navigation paths have the same score, the target navigation path is determined according to the priority of the preset planning parameters during the evaluation. For example, when the three preset planning parameters of path length, entry distance, and blood vessel density of path A, path B, and path C are considered at the same time, path A and path C have the same score. At this time, the blood vessel density has the highest priority among the three preset planning parameters, and the blood vessel density value in path C is the lowest. Path C is selected as the selected target navigation path.

[0125] In some embodiments, the preset planning parameters also include: the number of narrow sections in the navigation path, the number of sharp turns, user input or preferences, the patient's physiological supports, the actual characteristics of the target site (such as lesions, biopsy points, etc.), etc. to ensure that the actual operability of the target navigation path is optimized.

[0126] See also Figure 7 , showing a system block diagram of the navigation module. Further, the navigation module 75 includes: a structuring unit 751, used to perform structural processing on the centerline point set of the reconstructed 3D bronchial tree to generate a centerline tree; a guidance unit 752, used to generate a guidance path from the root node of the centerline tree to the target part according to the tree structure information of the centerline tree.

[0127] The structuring unit 751 is used to extract the discrete centerline point set of the 3D bronchial tree by calculating the distance from each bronchial voxel to the boundary of the bronchial tree, and to obtain the centerline tree by structuring the centerline point set to clarify the bifurcation nodes and branches of the bronchus, wherein each node represents the bifurcation point of the bronchus, and each branch edge represents the centerline segment connecting two nodes. In the centerline tree, all non-leaf nodes have child nodes, and all nodes except the root node have a parent node. In the bronchial tree, leaf nodes usually represent the terminal branches of the bronchus, or those tracheal endpoints without further branches, and non-leaf nodes usually represent the bifurcation points or connection points of the bronchus. At the location of the bronchial bifurcation, there will be two or more bronchial branches extending from the bifurcation point, and these branches are located at the child nodes of the bifurcation point.

[0128] It can be understood that the point on the centerline closest to the CT head direction is selected as the starting point of the bronchial tree, as the root node R of the bronchial tree 3D mask centerline tree, and all the centerline points connected to it are continuously added to the tree structure starting from the root node using a depth-first strategy until all points are included in the tree, and finally a tree-type data storage structure is formed. In this process, all non-leaf nodes will have child nodes, and all nodes except the root node will have parent nodes, and finally a standard tree structure is formed.

[0129] Generally speaking, the human airway tree has about 24 branches from the first level to the alveoli, which is shaped like an inverted tree, so it can be extended to the concept of tree structure in data structure. The trachea is divided into the left and right main bronchi at the bifurcation, and the bronchial entrance can be compared to the root node.

[0130] The guidance unit 752 is used to select the starting point of the main trachea centerline as the root node R of the centerline tree, use the depth-first strategy to continuously add all points to the centerline tree, and finally form a tree-type data storage structure, select the target part A at the center point in the tree-type data storage structure, and generate a guidance path L from the root node R to the target part A according to the tree structure. The guidance path L is used to guide the bronchoscope 1 when it moves along the target navigation path in the three-dimensional virtual map model.

[0131] See also Figure 8, showing a system block diagram of the navigation assistance module. The navigation assistance module 76 includes: a part recognition unit 761, which recognizes the real-time part corresponding to the real-time image collected by the bronchoscope at the current real-time position based on a pre-trained part recognition model; a segment recognition unit 762, which recognizes the part segment of the 3D bronchial tree based on a pre-trained point cloud classification model; a part mapping unit 763, used to map the recognized real-time part to the part segment corresponding to the 3D bronchial tree; a synchronization unit 764, used to dynamically synchronize the mapping result of the part mapping unit during the movement of the bronchoscope along the target navigation path in the three-dimensional virtual map model. It can make each position update / examination part update during the inspection process of the bronchoscope be able to be synchronously displayed in the three-dimensional virtual map model, and the bronchoscope can dynamically and synchronously update the real-time position of the bronchoscope under the current field of view when moving along the target navigation path in the three-dimensional virtual map model, thereby realizing the dynamic synchronization of the specific position between the bronchoscope and the three-dimensional virtual map model. Helps doctors track the current position of the bronchoscope and displays the complete operation path, ensuring that doctors always know the exact position and direction of the bronchoscope.

[0132] The recognition process of the part recognition unit 761 includes initializing the part recognition model based on the trained model parameters when the system is running, continuously collecting 20 consecutive frames of images (sampling method is 4 frames per second, sampling for 5 seconds in total) and inputting them into the trained part recognition model for classification and recognition, and obtaining the recognition result of the current bronchoscope for the current real-time part. The part recognition model can be trained by a multi-frame convolution feature fusion classification network.

[0133] Furthermore, the specific steps of the training process of the part recognition model include: using CNN to extract features from multiple consecutive (20 frames) video frames, introducing the Attention structure, calculating the importance weight of each frame image through global average pooling and Sigmoid activation function, using the Attention weight to weight and sum the feature vectors to obtain a fused feature vector, classifying the fused features through a fully connected layer and a Softmax activation function, outputting the confidence of the classification result, using randomly initialized weights and the Adam optimizer, combined with the cross entropy loss function for model training and optimization, and obtaining a pre-trained part recognition model.

[0134] When displayed on the display interface, the synchronization unit 764 can ultimately achieve real-time linkage with the three-dimensional virtual map model during the bronchoscope operation process, and dynamically and synchronously display the real-time position and / or location segment of the bronchoscope in the target navigation path in the three-dimensional virtual map model.

[0135] The segment identification unit 762 further includes: a point cloud generation unit 7621, which is used to generate a point cloud set based on the center line of the extracted 3D bronchial tree; a classification unit 7622, ​​which is used to classify the preprocessed point cloud set based on a pre-trained point cloud classification model, and obtain the classification result as the identification result of the part segment.

[0136] Among them, the part segment recognition process includes: when the system is running, the center line of the bronchial tree 3D mask is extracted according to the skeleton extraction algorithm, and the farthest point uniform sampling algorithm is performed to select evenly distributed points along the entire center line length to form a point cloud set, and the formed point cloud set is input into the pre-trained point cloud classification model to obtain the corresponding bronchial part classification information of each point.

[0137] The point cloud generation process includes sampling the farthest point uniform sampling algorithm on the bronchial centerline, starting from a point on the centerline as the first sampling point, selecting the farthest point in each iteration, calculating the new distance each time and updating the nearest distance, ensuring that each point can find the nearest sampling point, thereby finding the farthest point, and when the required number of sampling points (2048) is reached, the algorithm ends. The attribute of each point is the triplet (x, y, z) of its spatial coordinates, and its category information is the location classification information of the bronchus.

[0138] Furthermore, before inputting the point cloud data into the pre-trained point cloud classification model, it is usually necessary to perform pre-processing such as point cloud downsampling and point cloud normalization on the data. The pre-processed point cloud set is input into the pre-trained point cloud classification model, and the three-dimensional point cloud data such as the coordinates of the point clouds in the received point cloud set and possible additional features is output based on the classification logic in the point cloud classification model. The probability distribution of the bronchial site segment indicated by each point in the point cloud set is output, and the final classification result of the bronchial site segment is determined by screening the maximum probability category of each point. Alternatively, according to the pre-designed model, an overall classification result may also be output to identify the site segment of the entire bronchial area.

[0139] Among them, the classification logic can include that the PointMLP model will gradually extract and aggregate the features of the point cloud through the calculation of each layer for the input point cloud data. The final output feature vector is used for classification, and the model outputs the category of each point. The last layer of the model is usually a fully connected layer (or several layers), which maps the extracted features to the category space and generates the category prediction result for each point.

[0140] Specifically, the process of training the point cloud classification model includes: initializing the point cloud data input model, the model extracts the features of the point cloud through the neural network structure such as MLP, and finally obtains the category prediction value of each point or the entire point cloud. The category prediction value includes the probability distribution of each category of the output bronchus, for example, the output can be normalized to probability using the softmax function; the difference between the model prediction value and the true label is calculated using the cross entropy loss function; the category probability distribution output by the model is compared with the true one-hot label to generate the cross entropy loss value; based on the value of the loss function, the gradient of the model parameters (i.e., the derivative of the loss function to the model parameters) is calculated by the back propagation algorithm. Among them, the loss value is reduced by adjusting the weight parameters of the model and the adjustment method of the bias parameters. Specifically, the Adam optimizer is used to update the parameters of the model based on the calculated gradient, momentum term, and variance term; because Adam can dynamically adjust the learning rate, the model training is more stable and converges quickly; the training process will be iterated multiple times. In each iteration, the model will be trained with multiple small batches of point cloud data, and the parameters will be continuously optimized so that the prediction results output by the model are closer and closer to the true value.

[0141] For example, taking PointMLP as a point cloud classification model for bronchial segment classification, the training process includes: inputting point cloud data, passing through the PointMLP network layer, generating category predictions for each point, using cross entropy loss to calculate the difference between the model prediction value and the true label, calculating the derivative of the loss function with respect to the network parameters, and determining the adjustment direction of the parameters. The Adam optimizer updates the model's weight parameters and bias parameters according to the results of back propagation and repeats the iterations until the loss function value is reduced to a satisfactory level, so that the classification performance of the cloud classification model meets expectations.

[0142] The part mapping unit 763 further includes: a coordinate conversion unit 7631, which converts the real-time position of the real-time part identified by the bronchoscope into the display position of the corresponding part segment in the view coordinate system based on the relative relationship between the real-time coordinate system and the view coordinate system; and a viewing angle adjustment unit 7632, which adjusts the display viewing angle of the real-time part in the view coordinate system where the real-time part is located in the 3D bronchial tree based on the display position information of the part segment corresponding to the real-time part in the view coordinate system.

[0143] The part mapping unit 7631 includes the implementation of real-time part recognition of the bronchoscope based on deep learning video understanding to realize the part recognition under the bronchoscope. The specific implementation process includes preprocessing and feature extraction of the image in the real-time video of the bronchoscope, and then using the convolutional neural network (CNN) to capture the spatial and temporal information in the video, and finally realize the real-time recognition and positioning of different parts of the bronchus. The position of the real-time part recognized under the bronchoscope in the real-time coordinate system is (x cam ,y cam ,z cam ), the relative relationship between the real-time coordinate system and the VTK coordinate system showing the 3D bronchial tree is the rotation matrix M, which is used to transform the coordinates (x cam ,y cam ,z cam ) from the real-time coordinate system to the VTK view coordinate system (x vtk ,y vtk ,z vtk ).

[0144] The adjustment process of the viewing angle adjustment unit 7632 includes adjusting the coordinates (x vtk ,y vtk ,z vtk ), set the camera's view matrix in the view coordinate system, specify the camera's position, viewpoint (target direction), and upward direction (usually vertical direction) to align it with the centerline axis of the 3D bronchial tree, ensure that the VTK camera perspective in the view coordinates is consistent with the perspective of the real-time part corresponding to the image collected under the bronchoscope, and make the rendered virtual navigation picture accurately correspond to the observation perspective under the bronchoscope.

[0145] The present invention further provides an airway navigation method based on a bronchoscopic view, which is applied to an airway navigation system based on a bronchoscopic view, such as Fig. 9As shown, a flow chart of the airway navigation method based on bronchoscope view is shown. The specific steps of the method include: step S802: reconstructing the bronchial tree 3D mask through the segmented CT image; step S804: obtaining a three-dimensional virtual map model including a 3D bronchial tree according to the bronchial tree 3D mask; step S806: extracting the navigation path from the bronchial entrance to the target part based on the 3D bronchial tree; step S808: when the number of navigation paths is more than one, one of the navigation paths is selected as the target navigation path based on the preset planning parameters; step S810: guiding the bronchoscope to move along the target navigation path in the three-dimensional virtual map model based on the visualized guidance path, step S812: mapping the real-time part corresponding to the identified bronchoscope to the 3D bronchial tree for matching, so as to realize the dynamic synchronization of the real-time part corresponding to the bronchoscope and the target navigation path. The process of this method is consistent with the implementation method of each module / unit of the aforementioned airway navigation based on bronchoscope view, and will not be repeated here.

[0146] Therefore, the airway navigation system and method can reconstruct a 3D bronchial tree based on CT images, generate a target navigation path based on the bronchial entrance and the target part, and at the same time, can match the real-time image acquired under the bronchoscope with the structural information of the 3D bronchial tree reconstructed based on the traditional CT image sequence, and establish synchronization between the real-time scene under the bronchoscope and the virtual map reconstructed based on the CT image, so that during the bronchoscope navigation process, the bronchoscope can move along the target navigation path and sense the real-time position of the bronchoscope in real time to provide accurate navigation and positioning support for bronchoscope operations, and identify the specific part of the current bronchus (such as which branch, which section) through the view under the bronchoscope, and correspond it to the section of the 3D bronchial tree. Therefore, it can ensure that doctors can accurately view the current position and navigation progress on the three-dimensional virtual map model, which can effectively avoid misoperation and reduce damage to normal airway tissue, and is particularly suitable for surgical operations with complex airway structures.

[0147] Furthermore, a virtual map based on CT sequence images is used as a basis, and the optimal navigation path from the current bronchoscope position to the target site is determined using preset planning parameters as the target navigation path. The target navigation path planning takes into account the shape of the airway and possible obstacles, etc., so that when operating the bronchoscope, the doctor can quickly reach the target site according to the planned optimal surgical path under the guidance of the target navigation path, thereby saving operation time and reducing operation complexity.

[0148] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these changes and modifications.

Claims

1. An airway navigation system based on bronchoscopic view, characterized in that: The system comprises: The reconstruction module reconstructs the bronchial tree 3D mask from the CT image; A model generation module, which obtains a three-dimensional virtual map model including a 3D bronchial tree according to the bronchial tree 3D mask; a path extraction module, which extracts a navigation path from the bronchial entrance to the target site based on the 3D bronchial tree; A path selection module, which selects one of the navigation paths as a target navigation path based on preset planning parameters; a navigation module, which guides the bronchoscope to move along the target navigation path of the three-dimensional virtual map model based on a visualized guidance path; A navigation auxiliary module maps the identified real-time position corresponding to the bronchoscope to the 3D bronchial tree for matching, so as to achieve dynamic synchronization between the real-time position corresponding to the bronchoscope and the target navigation path.

2. The airway navigation system based on bronchoscopic view according to claim 1, characterized in that: The reconstruction module comprises: An input preparation unit, used for selecting a data block after block division and normalization processing from a CT image sequence corresponding to the bronchus as an input object; A prediction unit, used for inputting the input object into a pre-trained prediction model to predict bronchial voxels; An output unit is used to perform splicing and restoration based on the bronchial voxels to obtain a 3D mask of the bronchial tree.

3. The airway navigation system based on bronchoscopic view according to claim 1, characterized in that: The model generation module includes: A preprocessing unit, used for preprocessing the bronchial tree 3D mask by morphological restoration and boundary smoothing; A visualization unit is used to realize visualization of the 3D bronchial tree and the three-dimensional virtual map model based on the preprocessed bronchial tree 3D mask.

4. The airway navigation system based on bronchoscopic view according to claim 1, characterized in that: The system further comprises: A path planning module generates the preset planning parameters based on at least two of the path length of the navigation path, the entry angle when contacting the target part, the entry distance when contacting the target part, the airway diameter at the end position of the navigation path, and the blood vessel density at the end position of the navigation path.

5. The airway navigation system based on bronchoscopic view according to claim 4, characterized in that: The path planning module includes at least two of the following units: A path length calculation unit, used for calculating the shortest path from the bronchial entrance to the target site in the navigation path; Entering an angle calculation unit, used to calculate the angle between the airway direction of the end point position of the navigation path and the surface normal of the target part; Entering a distance calculation unit, for calculating the shortest distance from the end position of the navigation path to the target part; an airway diameter calculation unit, used to calculate the airway cross-sectional diameter at the end point of the navigation path; The blood vessel density calculation unit is used to calculate the blood vessel density value at the end point of the navigation path.

6. The bronchoscopic view-based airway navigation system according to claim 1, characterized in that: The path selection module includes: A path evaluation unit evaluates the path score of each of the navigation paths based on the weight of the preset planning parameters, and selects the navigation path with the highest path score as the target navigation path.

7. The bronchoscopic view-based airway navigation system according to claim 1, characterized in that: The navigation module includes: A structuring unit, used for performing structural processing on the centerline point set of the 3D bronchial tree to generate a centerline tree; A guidance unit is used to generate a guidance path from the root node of the centerline tree to the target part according to the tree structure information of the centerline tree.

8. The bronchoscopic view-based airway navigation system according to claim 1, characterized in that: The navigation assistance module comprises: A part recognition unit, which recognizes a real-time part corresponding to a real-time image acquired by the bronchoscope via a current real-time position based on a pre-trained part recognition model; A segment identification unit, which identifies the location segment of the 3D bronchial tree based on a pre-trained point cloud classification model; A part mapping unit, configured to map the identified real-time part to the part segment corresponding to the 3D bronchial tree; A synchronization unit is used to dynamically synchronize the mapping result of the part mapping unit when the bronchoscope moves along the target navigation path.

9. The bronchoscopic view-based airway navigation system according to claim 8, characterized in that: The segment identification unit comprises: A point cloud generation unit, which generates a point cloud set based on the center line of the extracted 3D bronchial tree; The classification unit classifies the preprocessed point cloud set based on the pre-trained point cloud classification model, and obtains the classification result as the recognition result of the part segment.

10. The bronchoscopic view-based airway navigation system according to claim 8, characterized in that: The part mapping unit comprises: A coordinate conversion unit, which converts the real-time position of the real-time part identified by the bronchoscope into a display position of the corresponding part segment in the view coordinate system based on the relative relationship between the real-time coordinate system and the view coordinate system; A viewing angle adjustment unit adjusts a display viewing angle of the real-time part in the view coordinate system where the 3D bronchial tree is located based on display position information of the part segment corresponding to the real-time part in the view coordinate system.

11. An airway navigation method based on bronchoscopic view, characterized in that: The method is applied to the airway navigation system based on bronchoscopic view as described in any one of claims 1 to 10, and the method comprises: Reconstruct the bronchial tree 3D mask from the segmented CT image; Obtaining a three-dimensional virtual map model including a 3D bronchial tree according to the bronchial tree 3D mask; Extracting a navigation path from the bronchial entrance to the target site based on the 3D bronchial tree; Selecting one of the navigation paths as a target navigation path based on preset planning parameters; Guiding the bronchoscope to move along the target navigation path of the three-dimensional virtual map model based on a visualized guidance path; The real-time position corresponding to the identified bronchoscope is mapped to the 3D bronchial tree for matching, so as to achieve dynamic synchronization between the real-time position corresponding to the bronchoscope and the target navigation path.

Images