A bronchoscopic navigation method and system
Through the feature node extraction network and timing analysis model, the virtual bronchial model is analyzed and predicted in time, and a real-time virtual bronchial three-dimensional model is constructed, which solves the navigation accuracy problem under the influence of respiratory movement and achieves more efficient bronchoscopic navigation.
Patent Information
- Application Number
- CN202510262222.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-06
- Publication Date
- 2025-04-01
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing bronchoscopic navigation technology has caused the model to not match the actual anatomical structure due to the influence of respiratory movement on the virtual bronchial model, which affects the accuracy of navigation.
The virtual bronchial model is analyzed in timing through the feature node extraction network, and the bronchial characteristic nodes that are most susceptible to respiratory movements are obtained, and timing prediction is carried out based on the bronchial characteristic node timing analysis model to construct a real-time virtual bronchial three-dimensional model for navigation.
By compensating the impact of respiratory motion on virtual models in real time, the accuracy and efficiency of bronchoscopic navigation are improved.
Smart Images
Figure CN119732747A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of medical technology, and in particular to a bronchoscope navigation method and system. Background Art
[0002] Bronchoscopy is an important means of diagnosing and treating respiratory diseases. In order to improve the accuracy and efficiency of bronchoscopy, various bronchoscopic navigation technologies have emerged in recent years. These navigation technologies usually construct a three-dimensional virtual bronchial model based on the patient's chest CT scan image, and use the model to guide the bronchoscope to the target lesion.
[0003] However, existing bronchoscopic navigation technology still has some limitations. One of the main challenges is that respiratory motion can have a significant impact on the accuracy of the virtual bronchial model. During CT scanning, the patient's respiratory state is not completely still, which leads to differences between the bronchial structure in the scanned image and the bronchial structure in the patient's actual respiratory state. This difference will be magnified during the construction of the virtual bronchial model, making the model not completely match the patient's actual anatomical structure. Summary of the invention
[0004] The present invention first performs a time series analysis on the patient's virtual bronchial model through a feature node extraction network to obtain bronchial feature nodes that can reflect the patient's bronchi that are most susceptible to respiratory motion, and then performs time series prediction on the changes in bronchial feature nodes through changes in respiratory motion parameters based on the bronchial feature node time series analysis model, and uses the obtained predicted bronchial feature node set to construct a real-time virtual bronchial three-dimensional model of the next monitoring time point, so that during bronchoscopic navigation, each reference is made to the real-time virtual bronchial three-dimensional model that has been compensated for respiratory motion, and the bronchoscope navigation operation can be completed more accurately.
[0005] A bronchoscopic navigation method, comprising: Acquire N frames of virtual bronchial three-dimensional models of patients arranged in time, and form the virtual bronchial three-dimensional models of patients arranged in time into a virtual bronchial three-dimensional model time series analysis set, wherein the virtual bronchial three-dimensional model includes a number of point cloud data; then send the virtual bronchial three-dimensional model time series analysis set to a feature node extraction network for processing to output a bronchial feature node set, wherein the bronchial feature node set includes a number of bronchial feature nodes; the feature node extraction network includes a feature enhancement layer, a PointNet layer and a bronchial feature node set output layer, wherein the feature enhancement layer is used to analyze the time series changes of the bronchial feature nodes, and perform feature enhancement on the virtual bronchial three-dimensional model based on the time series changes of the bronchial feature nodes; the PointNet layer is used to perform feature extraction operations; the bronchial feature node set output layer is used to output the bronchial feature node set; In the process of bronchoscopic navigation, respiratory motion parameters at the monitoring time point are obtained, and the respiratory motion parameters at the current monitoring time point and the point cloud data corresponding to the bronchial feature nodes in the bronchial feature node set are spliced to construct bronchial feature node analysis data. The bronchial feature node analysis data at the current monitoring time point and the bronchial feature node analysis data at the previous N-1 monitoring time points are then combined into a bronchial feature node analysis time series data set, and the bronchial feature node analysis time series data set is sent to the bronchial feature node time series analysis model for processing to output a predicted bronchial feature point cloud data set at the next monitoring time point. The predicted bronchial feature point cloud data set includes a number of predicted bronchial feature point cloud data. At the next monitoring time point, the predicted bronchial feature point cloud data set is sent to the three-dimensional model reconstruction model for processing to obtain a real-time virtual bronchial three-dimensional model at the next monitoring time point, and the real-time bronchial image obtained through the bronchoscope at the next monitoring time point is image-aligned with the real-time virtual bronchial three-dimensional model at the next monitoring time point, and the moving direction of the bronchoscope is indicated to realize bronchoscope navigation.
[0006] Preferably, the virtual bronchial three-dimensional model time series analysis set is sent to a feature node extraction network for processing to output a bronchial feature node set, which specifically includes the following steps: Step S1: Send all virtual bronchial 3D models to the PointNet layer for processing respectively to output the corresponding bronchial feature node probability set, then perform a weighted average operation on all bronchial feature node probability sets to obtain a comprehensive bronchial feature node probability set, and send the comprehensive bronchial feature node probability set to the bronchial feature node set output layer for processing to output the bronchial feature node set; Step S2: In the feature enhancement layer, for each virtual bronchial three-dimensional model, the point cloud data corresponding to the bronchial feature nodes in the bronchial feature node set are recorded as feature point cloud data, and all feature point cloud data corresponding to the virtual bronchial three-dimensional model are combined into a feature point cloud data set; a corresponding feature point cloud adjacency matrix is constructed for the feature point cloud data set, and the feature point cloud adjacency matrix stores the normalized similarity between any two feature point cloud data in the feature point cloud data set; all feature point cloud data sets are combined into a feature point cloud time series data set in chronological order, and all feature point cloud adjacency matrices are combined into a feature point cloud adjacency matrix time series analysis set in chronological order, and then the feature point cloud time series data set and the feature point cloud adjacency matrix time series analysis set are respectively sent to the feature point cloud time series analysis network and the feature point cloud adjacency matrix time series analysis set. The feature point cloud adjacency matrix is processed by the time series analysis network to construct the corresponding feature point cloud time series feature vector and feature point cloud adjacency matrix time series feature vector; for each virtual bronchial 3D model, the virtual bronchial 3D model is updated by the following formula: H=σ(AWR), where H is the updated virtual bronchial 3D model, σ() is a nonlinear activation operation, A is the feature point cloud adjacency matrix time series feature vector, W is the graph convolution weight matrix, and R is the virtual bronchial 3D model. Based on the updated virtual bronchial 3D model and the feature point cloud time series feature vector, a self-attention mechanism is executed to enhance the features of each virtual bronchial 3D model to obtain an enhanced virtual bronchial 3D model, and the virtual bronchial 3D model is replaced by the enhanced virtual bronchial 3D model; Step S3: Send all virtual bronchial 3D models to the PointNet layer for processing respectively to output the corresponding bronchial feature node probability set, then perform a weighted average operation on all bronchial feature node probability sets to obtain a comprehensive bronchial feature node probability set, and send the comprehensive bronchial feature node probability set to the bronchial feature node set output layer for processing to output the bronchial feature node set; Step S4: sending all point cloud data corresponding to the bronchial feature node set to the three-dimensional model reconstruction model for processing, outputting the target virtual bronchial three-dimensional model, and taking the mean square error of the distance between all virtual bronchial three-dimensional models and the target virtual bronchial three-dimensional model as the real-time loss value, updating the feature node extraction network by the gradient descent method in the direction of minimizing the real-time loss value, and replacing the feature node extraction network; Step S5: Repeat steps S2 to S4 until the real-time training conditions are met, and output the bronchial feature node set in the last step S3.
[0007] Preferably, a self-attention mechanism is executed based on the updated virtual bronchial three-dimensional model and the time-series feature vector of the feature point cloud to obtain an enhanced virtual bronchial three-dimensional model, which specifically includes the following steps: The updated virtual bronchial 3D model is matrix multiplied with the value weight matrix and the key weight matrix to construct the corresponding value feature vector V and key feature vector K. The feature point cloud time series feature vector is matrix multiplied with the query weight matrix to construct the query feature vector Q. The attention weight matrix ATT=softmax(QK T / (D) 0.5 ), where D is the dimension of the key feature vector K, and then the attention weight matrix ATT is multiplied with the value feature vector V to obtain the enhanced virtual bronchial three-dimensional model.
[0008] Preferably, training the feature node extraction network specifically includes the following steps: A number of feature node extraction training samples are obtained, wherein the feature node extraction training samples include N frames of virtual bronchial three-dimensional models of patients arranged in time, and the feature node extraction training samples are labeled through the bronchial feature node set, and all labeled feature node extraction training samples are composed of a feature node extraction training set, and the feature node extraction network is trained through the feature node extraction training set. During the training, the bronchial feature node set is used as the target output, the feature node loss value is calculated, and the feature node loss value and the real-time loss value are combined into a comprehensive loss value, so as to minimize the direction of the comprehensive loss value, and the feature node extraction network is trained through the gradient descent method until the training conditions are met, and the trained feature node extraction network is output.
[0009] Preferably, training is performed on the three-dimensional model reconstruction model, specifically comprising the following steps: A number of 3D model reconstruction training samples are obtained, and the 3D model reconstruction training samples include a feature point cloud data set and a corresponding virtual bronchial 3D model. All the 3D model reconstruction training samples are combined into a 3D model reconstruction training set. The 3D model reconstruction model is trained by the 3D model reconstruction training set. During the training, the corresponding virtual bronchial 3D model is used as the target output, and the 3D model reconstruction loss value is calculated. The 3D model reconstruction model is trained by the gradient descent method in the direction of minimizing the 3D model reconstruction loss value until the training conditions are met, and the trained 3D model reconstruction model is output.
[0010] Preferably, the bronchial characteristic node time series analysis model is established based on the LSTM model.
[0011] A bronchoscopic navigation system, comprising: A virtual bronchial 3D model acquisition module is used to acquire N frames of virtual bronchial 3D models of patients arranged in time, and to form a virtual bronchial 3D model time series analysis set of the virtual bronchial 3D models of patients arranged in time, wherein the virtual bronchial 3D models include a plurality of point cloud data; The feature node extraction module is used to send the virtual bronchial three-dimensional model time series analysis set to the feature node extraction network for processing to output a bronchial feature node set, wherein the bronchial feature node set includes a number of bronchial feature nodes; the feature node extraction network includes a feature enhancement layer, a PointNet layer and a bronchial feature node set output layer, wherein the feature enhancement layer is used to analyze the time series changes of the bronchial feature nodes, and based on the time series changes of the bronchial feature nodes, the virtual bronchial three-dimensional model is feature enhanced; the PointNet layer is used to perform feature extraction operations; the bronchial feature node set output layer is used to output the bronchial feature node set; The real-time virtual bronchial three-dimensional model construction module is used to obtain the respiratory motion parameters at the monitoring time point during the bronchoscopic navigation process, splice the respiratory motion parameters at the current monitoring time point with the point cloud data corresponding to the bronchial feature nodes in the bronchial feature node set, construct the bronchial feature node analysis data, and then combine the bronchial feature node analysis data at the current monitoring time point and the bronchial feature node analysis data at the previous N-1 monitoring time points to form a bronchial feature node analysis time series data set, and send the bronchial feature node analysis time series data set to the bronchial feature node time series analysis model. Processing is performed to output a predicted bronchial feature point cloud data set for the next monitoring time point, wherein the predicted bronchial feature point cloud data set includes several predicted bronchial feature point cloud data. At the next monitoring time point, the predicted bronchial feature point cloud data set is sent to the three-dimensional model reconstruction model for processing to obtain a real-time virtual bronchial three-dimensional model of the next monitoring time point. The real-time bronchial image obtained through the bronchoscope at the next monitoring time point is image aligned with the real-time virtual bronchial three-dimensional model of the next monitoring time point, and the moving direction of the bronchoscope is indicated to realize the navigation of the bronchoscope.
[0012] The present invention has the following advantages: The present invention first performs a time series analysis on the patient's virtual bronchial model through a feature node extraction network to obtain bronchial feature nodes that can reflect the patient's bronchi that are most susceptible to respiratory motion, and then performs time series prediction on the changes in bronchial feature nodes through changes in respiratory motion parameters based on the bronchial feature node time series analysis model, and uses the obtained predicted bronchial feature node set to construct a real-time virtual bronchial three-dimensional model of the next monitoring time point, so that during bronchoscopic navigation, each reference is made to the real-time virtual bronchial three-dimensional model that has been compensated for respiratory motion, and the bronchoscope navigation operation can be completed more accurately. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 It is a schematic diagram of the structure of the bronchoscopic navigation system adopted in an embodiment of the present invention. DETAILED DESCRIPTION
[0014] In order to enable persons skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0015] Embodiment 1, a bronchoscopic navigation method, comprising: Obtain N frames of virtual bronchial 3D models of patients arranged in time, where N is generally 10, and compose the virtual bronchial 3D models of patients arranged in time into a virtual bronchial 3D model time series analysis set. It should be noted that before performing bronchoscopic navigation operation on the patient, a CT operation will be performed on the bronchial part of the patient to obtain the corresponding bronchial CT image, and the bronchial CT image corresponding to each time step will be constructed into a corresponding virtual bronchial 3D model through 3D reconstruction software (such as 3DSlicer, OsiriX and Mimics, etc.), and the virtual bronchial 3D model here is stored in the form of point cloud data, that is, the virtual bronchial 3D model includes several point cloud data ; Then, the virtual bronchial three-dimensional model timing analysis set is sent to the feature node extraction network for processing to output the bronchial feature node set, which includes a number of bronchial feature nodes, and these bronchial feature nodes can reflect the part of the patient's bronchi that is most susceptible to respiratory movement; the feature node extraction network includes a feature enhancement layer, a PointNet layer and a bronchial feature node set output layer, wherein the feature enhancement layer is used to analyze the timing changes of the bronchial feature nodes, and based on the timing changes of the bronchial feature nodes, the virtual bronchial three-dimensional model is feature enhanced; the PointNet layer is used to perform feature extraction operations; the bronchial feature node set output layer is used to output the bronchial feature node set; During the bronchoscopic navigation process, the respiratory motion parameters at the monitoring time point are obtained. Here, the respiratory motion parameters can be data such as the patient's respiratory rate and minute ventilation volume, which are used to reflect the patient's respiratory motion. The respiratory motion parameters at the current monitoring time point and the point cloud data corresponding to the bronchial feature nodes in the bronchial feature node set are spliced to construct bronchial feature node analysis data. The bronchial feature node analysis data at the current monitoring time point and the bronchial feature node analysis data at the previous N-1 monitoring time points are combined into a bronchial feature node analysis time series data set. The bronchial feature node analysis time series data set is then sent to the bronchial feature node time series analysis model for processing to output the predicted value at the next monitoring time point. The bronchial feature point cloud data set is measured. The predicted bronchial feature point cloud data set includes several predicted bronchial feature point cloud data. These predicted bronchial feature point cloud data can characterize the changes of bronchial feature nodes after being affected by respiratory movement. At the next monitoring time point, the predicted bronchial feature point cloud data set is sent to the three-dimensional model reconstruction model for processing to obtain a real-time virtual bronchial three-dimensional model at the next monitoring time point. The real-time bronchial image obtained by the bronchoscope at the next monitoring time point is image-aligned with the real-time virtual bronchial three-dimensional model at the next monitoring time point, and the moving direction of the bronchoscope is indicated to realize the navigation of the bronchoscope. The bronchial feature node time series analysis model is established based on the LSTM model. The present application first performs a time series analysis on the patient's virtual bronchial model through a feature node extraction network to obtain the bronchial feature nodes that can reflect the patient's bronchi that are most susceptible to respiratory motion, and then performs a time series prediction on the changes in bronchial feature nodes through changes in respiratory motion parameters based on the bronchial feature node time series analysis model, and uses the obtained predicted bronchial feature node set to construct a real-time virtual bronchial three-dimensional model of the next monitoring time point, so that during bronchoscopic navigation, each reference is made to the real-time virtual bronchial three-dimensional model that has been compensated for respiratory motion, so that the bronchoscope navigation operation can be completed more accurately.
[0016] The virtual bronchial 3D model time series analysis set is sent to the feature node extraction network for processing to output the bronchial feature node set, which specifically includes the following steps: Step S1: All virtual bronchial three-dimensional models are sent to the PointNet layer for processing respectively to output the corresponding bronchial feature node probability set, the bronchial feature node probability set includes the probability value of each node in the corresponding virtual bronchial three-dimensional model being a feature node, and then all bronchial feature node probability sets are subjected to a weighted average operation to obtain a comprehensive bronchial feature node probability set, which can be regarded as a voting operation of each virtual bronchial three-dimensional model on the feature node, and the comprehensive bronchial feature node probability set is sent to the bronchial feature node set output layer for processing. The specific processing operation is to select the nodes in the comprehensive bronchial feature node probability set whose probability values are greater than the feature node threshold as bronchial feature nodes, where the feature node threshold is artificially set, to output the bronchial feature node set; Step S2: In the feature enhancement layer, for each virtual bronchial three-dimensional model, the point cloud data corresponding to the bronchial feature nodes in the bronchial feature node set are recorded as feature point cloud data, and all feature point cloud data corresponding to the virtual bronchial three-dimensional model are combined into a feature point cloud data set; a corresponding feature point cloud adjacency matrix is constructed for the feature point cloud data set, and the feature point cloud adjacency matrix stores the normalized similarity between any two feature point cloud data in the feature point cloud data set. It should be noted that the normalization here can use the softmax function, and the similarity calculation uses the cosine similarity algorithm; all feature point cloud data sets are combined into a feature point cloud time series data set in chronological order, and all feature point cloud adjacency matrices are combined into a feature point cloud adjacency matrix time series analysis set in chronological order, and then the feature point cloud time series data set and the feature point cloud adjacency matrix time series analysis set are respectively sent to the feature point cloud time series analysis network and the feature point cloud adjacency matrix time series analysis network for processing, and the corresponding feature point cloud time series feature vector and feature point are constructed. The cloud adjacency matrix temporal feature vector, the feature point cloud temporal analysis network and the feature point cloud adjacency matrix temporal analysis network are all established based on the LSTM model. The feature point cloud temporal feature vector and the feature point cloud adjacency matrix temporal feature vector respectively characterize the changes of feature nodes and the connections between feature nodes with respiratory movement; for each virtual bronchial 3D model, the virtual bronchial 3D model is updated by the following formula: H=σ(AWR), where H is the updated virtual bronchial 3D model, σ() is a nonlinear activation operation, generally a ReLU function, A is the feature point cloud adjacency matrix temporal feature vector, W is the graph convolution weight matrix, and the training is performed following the feature node extraction network. R is the virtual bronchial 3D model. Based on the updated virtual bronchial 3D model and the feature point cloud temporal feature vector, a self-attention mechanism is performed to enhance the features of each virtual bronchial 3D model to obtain an enhanced virtual bronchial 3D model, and the virtual bronchial 3D model is replaced by the enhanced virtual bronchial 3D model. Step S3: Send all virtual bronchial 3D models to the PointNet layer for processing respectively to output the corresponding bronchial feature node probability set, then perform a weighted average operation on all bronchial feature node probability sets to obtain a comprehensive bronchial feature node probability set, and send the comprehensive bronchial feature node probability set to the bronchial feature node set output layer for processing to output the bronchial feature node set; Step S4: sending all point cloud data corresponding to the bronchial feature node set to the three-dimensional model reconstruction model for processing, outputting the target virtual bronchial three-dimensional model, and taking the mean square error of the distance between all virtual bronchial three-dimensional models and the target virtual bronchial three-dimensional model as the real-time loss value, updating the feature node extraction network by the gradient descent method in the direction of minimizing the real-time loss value, and replacing the feature node extraction network; Step S5: Repeat steps S2 to S4 until the real-time training conditions are met. The real-time training conditions here may be that the real-time loss value is lower than a certain value or the number of repetitions reaches a preset maximum number of real-time training times, and output the bronchial feature node set in the last step S3.
[0017] Based on the updated virtual bronchial 3D model and the time series feature vector of the feature point cloud, a self-attention mechanism is executed to obtain an enhanced virtual bronchial 3D model, which specifically includes the following steps: The updated virtual bronchial 3D model is matrix multiplied with the value weight matrix and the key weight matrix to construct the corresponding value feature vector V and key feature vector K. The feature point cloud time series feature vector is matrix multiplied with the query weight matrix to construct the query feature vector Q. The attention weight matrix ATT=softmax(QK T / (D) 0.5 ), where D is the dimension of the key feature vector K, and then the attention weight matrix ATT is multiplied with the value feature vector V to obtain the enhanced virtual bronchial three-dimensional model.
[0018] Training the feature node extraction network includes the following steps: A number of feature node extraction training samples are obtained, the feature node extraction training samples include N frames of virtual bronchial three-dimensional models of patients arranged in time, and these virtual bronchial three-dimensional models are constructed based on the actually collected CT images, and the feature node extraction training samples are annotated by a bronchial feature node set, wherein the bronchial feature node set here is generally annotated manually, and all annotated feature node extraction training samples are composed of a feature node extraction training set, and a feature node extraction network is trained by the feature node extraction training set. During the training, the bronchial feature node set is used as the target output, the feature node loss value is calculated, and the feature node loss value and the real-time loss value are composed of a comprehensive loss value, which can be specifically composed of a weighted summation, and the weights are manually set during weighting, in the direction of minimizing the comprehensive loss value, and the feature node extraction network is trained by the gradient descent method until the training conditions are met, and the training conditions can be that the comprehensive loss value is lower than a certain value or the number of training times reaches a preset maximum number of training times, and the trained feature node extraction network is output.
[0019] Training the 3D model reconstruction model includes the following steps: A number of three-dimensional model reconstruction training samples are obtained, and the three-dimensional model reconstruction training samples include a feature point cloud data set and a corresponding virtual bronchial three-dimensional model. It should be noted that the feature point cloud data set in the three-dimensional model reconstruction training samples here can be obtained by processing the virtual bronchial three-dimensional model through a trained feature node extraction network to obtain bronchial feature nodes, and all three-dimensional model reconstruction training samples are composed of a three-dimensional model reconstruction training set, and the three-dimensional model reconstruction model is trained through the three-dimensional model reconstruction training set. During the training, the corresponding virtual bronchial three-dimensional model is used as the target output, and the three-dimensional model reconstruction loss value is calculated. The three-dimensional model reconstruction model is trained by the gradient descent method in the direction of minimizing the three-dimensional model reconstruction loss value until the training conditions are met. The training conditions can be that the three-dimensional model reconstruction loss value is lower than a certain value or the number of training times reaches a preset maximum number of training times, and the trained three-dimensional model reconstruction model is output; it should be noted that the three-dimensional model reconstruction model here is designed based on a multi-layer perceptron.
[0020] Embodiment 2, a bronchoscopic navigation system, such as Figure 1 As shown, including: The virtual bronchial 3D model acquisition module is used to acquire N frames of virtual bronchial 3D models of patients arranged in time, where N is generally 10, and the virtual bronchial 3D models of patients arranged in time are composed into a virtual bronchial 3D model time series analysis set. It should be noted that before the bronchoscopic navigation operation is performed on the patient, a CT operation is performed on the bronchial part of the patient to obtain the corresponding bronchial CT image, and the bronchial CT image corresponding to each time step is constructed into a corresponding virtual bronchial 3D model through 3D reconstruction software (such as 3DSlicer, OsiriX and Mimics, etc.), and the virtual bronchial 3D model here is stored in the form of point cloud data, that is, the virtual bronchial 3D model includes a number of point cloud data; A feature node extraction module is used to send the virtual bronchial three-dimensional model time series analysis set to the feature node extraction network for processing to output a bronchial feature node set, wherein the bronchial feature node set includes a number of bronchial feature nodes, and these bronchial feature nodes can reflect the part of the patient's bronchus that is most susceptible to respiratory movement; the feature node extraction network includes a feature enhancement layer, a PointNet layer and a bronchial feature node set output layer, wherein the feature enhancement layer is used to analyze the time series changes of the bronchial feature nodes, and based on the time series changes of the bronchial feature nodes, the virtual bronchial three-dimensional model is feature enhanced; the PointNet layer is used to perform feature extraction operations; the bronchial feature node set output layer is used to output the bronchial feature node set; The real-time virtual bronchial three-dimensional model construction module is used to obtain the respiratory motion parameters at the monitoring time point during the bronchoscopic navigation process. The respiratory motion parameters here can be data such as the patient's respiratory rate and minute ventilation volume, which are used to reflect the patient's respiratory motion. The respiratory motion parameters at the current monitoring time point and the point cloud data corresponding to the bronchial feature nodes in the bronchial feature node set are spliced to construct bronchial feature node analysis data, and then the bronchial feature node analysis data at the current monitoring time point and the bronchial feature node analysis data at the previous N-1 monitoring time points are combined into a bronchial feature node analysis time series data set, and the bronchial feature node analysis time series data set is sent to the bronchial feature node time series analysis The predicted bronchial feature point cloud data set is processed in the model to output a predicted bronchial feature point cloud data set at the next monitoring time point. The predicted bronchial feature point cloud data set includes a number of predicted bronchial feature point cloud data. These predicted bronchial feature point cloud data can characterize the changes in bronchial feature nodes after being affected by respiratory movement. At the next monitoring time point, the predicted bronchial feature point cloud data set is sent to the three-dimensional model reconstruction model for processing to obtain a real-time virtual bronchial three-dimensional model at the next monitoring time point. The real-time bronchial image obtained through the bronchoscope at the next monitoring time point is image-aligned with the real-time virtual bronchial three-dimensional model at the next monitoring time point, and the moving direction of the bronchoscope is indicated to realize the navigation of the bronchoscope.
[0021] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention. Parts not described in detail in this specification belong to the prior art known to those skilled in the art.
Claims
1. A bronchoscopic navigation method, characterized in that: include: Acquire N frames of virtual bronchial three-dimensional models of patients arranged in time, and form the virtual bronchial three-dimensional models of patients arranged in time into a virtual bronchial three-dimensional model time series analysis set, wherein the virtual bronchial three-dimensional model includes a plurality of point cloud data; then send the virtual bronchial three-dimensional model time series analysis set to a feature node extraction network for processing to output a bronchial feature node set, wherein the bronchial feature node set includes a plurality of bronchial feature nodes; the feature node extraction network includes a feature enhancement layer, a PointNet layer and a bronchial feature node set output layer, wherein the feature enhancement layer is used to analyze the time series changes of the bronchial feature nodes, and perform feature enhancement on the virtual bronchial three-dimensional model based on the time series changes of the bronchial feature nodes; the PointNet layer is used to perform feature extraction operations; The bronchial feature node set output layer is used to output the bronchial feature node set; In the process of bronchoscopic navigation, respiratory motion parameters at the monitoring time point are obtained, and the respiratory motion parameters at the current monitoring time point and the point cloud data corresponding to the bronchial feature nodes in the bronchial feature node set are spliced to construct bronchial feature node analysis data. The bronchial feature node analysis data at the current monitoring time point and the bronchial feature node analysis data at the previous N-1 monitoring time points are then combined into a bronchial feature node analysis time series data set, and the bronchial feature node analysis time series data set is sent to the bronchial feature node time series analysis model for processing to output a predicted bronchial feature point cloud data set at the next monitoring time point. The predicted bronchial feature point cloud data set includes a number of predicted bronchial feature point cloud data. At the next monitoring time point, the predicted bronchial feature point cloud data set is sent to the three-dimensional model reconstruction model for processing to obtain a real-time virtual bronchial three-dimensional model at the next monitoring time point, and the real-time bronchial image obtained through the bronchoscope at the next monitoring time point is image-aligned with the real-time virtual bronchial three-dimensional model at the next monitoring time point, and the moving direction of the bronchoscope is indicated to realize bronchoscope navigation.
2. A bronchoscopic navigation method according to claim 1, characterized in that: The virtual bronchial 3D model time series analysis set is sent to the feature node extraction network for processing to output the bronchial feature node set, which specifically includes the following steps: Step S1: Send all virtual bronchial 3D models to the PointNet layer for processing respectively to output the corresponding bronchial feature node probability set, then perform a weighted average operation on all bronchial feature node probability sets to obtain a comprehensive bronchial feature node probability set, and send the comprehensive bronchial feature node probability set to the bronchial feature node set output layer for processing to output the bronchial feature node set; Step S2: In the feature enhancement layer, for each virtual bronchial three-dimensional model, the point cloud data corresponding to the bronchial feature nodes in the bronchial feature node set are recorded as feature point cloud data, and all feature point cloud data corresponding to the virtual bronchial three-dimensional model are combined into a feature point cloud data set; Construct a corresponding feature point cloud adjacency matrix for the feature point cloud data set. The feature point cloud adjacency matrix stores the normalized similarity between any two feature point cloud data in the feature point cloud data set. Construct all feature point cloud data sets in chronological order to form a feature point cloud time series data set. Construct all feature point cloud adjacency matrices in chronological order to form a feature point cloud adjacency matrix time series analysis set. Then, send the feature point cloud time series data set and the feature point cloud adjacency matrix time series analysis set to the feature point cloud time series analysis network and the feature point cloud adjacency matrix time series analysis network for processing, respectively, to construct the corresponding feature point cloud time series feature vector and feature point cloud adjacency matrix time series feature. vector; for each virtual bronchial 3D model, the virtual bronchial 3D model is updated by the following formula: H = σ (AWR), where H is the updated virtual bronchial 3D model, σ () is a nonlinear activation operation, A is the temporal feature vector of the feature point cloud adjacency matrix, W is the graph convolution weight matrix, and R is the virtual bronchial 3D model. Based on the updated virtual bronchial 3D model and the temporal feature vector of the feature point cloud, a self-attention mechanism is performed to enhance the features of each virtual bronchial 3D model to obtain an enhanced virtual bronchial 3D model, and the virtual bronchial 3D model is replaced by the enhanced virtual bronchial 3D model; Step S3: Send all virtual bronchial 3D models to the PointNet layer for processing respectively to output the corresponding bronchial feature node probability set, then perform a weighted average operation on all bronchial feature node probability sets to obtain a comprehensive bronchial feature node probability set, and send the comprehensive bronchial feature node probability set to the bronchial feature node set output layer for processing to output the bronchial feature node set; Step S4: sending all point cloud data corresponding to the bronchial feature node set to the three-dimensional model reconstruction model for processing, outputting the target virtual bronchial three-dimensional model, and taking the mean square error of the distance between all virtual bronchial three-dimensional models and the target virtual bronchial three-dimensional model as the real-time loss value, updating the feature node extraction network by the gradient descent method in the direction of minimizing the real-time loss value, and replacing the feature node extraction network; Step S5: Repeat steps S2 to S4 until the real-time training conditions are met, and output the bronchial feature node set in the last step S3.
3. A bronchoscopic navigation method according to claim 2, characterized in that: Based on the updated virtual bronchial 3D model and the time series feature vector of the feature point cloud, a self-attention mechanism is executed to obtain an enhanced virtual bronchial 3D model, which specifically includes the following steps: The updated virtual bronchial 3D model is matrix multiplied with the value weight matrix and the key weight matrix to construct the corresponding value feature vector V and key feature vector K. The feature point cloud time series feature vector is matrix multiplied with the query weight matrix to construct the query feature vector Q. The attention weight matrix ATT=softmax(QK T / (D) 0.5 ), where D is the dimension of the key feature vector K, and then the attention weight matrix ATT is multiplied with the value feature vector V to obtain the enhanced virtual bronchial three-dimensional model.
4. A bronchoscopic navigation method according to claim 3, characterized in that: Training the feature node extraction network includes the following steps: A number of feature node extraction training samples are obtained, wherein the feature node extraction training samples include N frames of virtual bronchial three-dimensional models of patients arranged in time, and the feature node extraction training samples are labeled through the bronchial feature node set, and all labeled feature node extraction training samples are composed of a feature node extraction training set, and the feature node extraction network is trained through the feature node extraction training set. During the training, the bronchial feature node set is used as the target output, the feature node loss value is calculated, and the feature node loss value and the real-time loss value are combined into a comprehensive loss value, so as to minimize the direction of the comprehensive loss value, and the feature node extraction network is trained through the gradient descent method until the training conditions are met, and the trained feature node extraction network is output.
5. A bronchoscopic navigation method according to claim 4, characterized in that: Training the 3D model reconstruction model includes the following steps: A number of 3D model reconstruction training samples are obtained, and the 3D model reconstruction training samples include a feature point cloud data set and a corresponding virtual bronchial 3D model. All the 3D model reconstruction training samples are combined into a 3D model reconstruction training set. The 3D model reconstruction model is trained by the 3D model reconstruction training set. During the training, the corresponding virtual bronchial 3D model is used as the target output, and the 3D model reconstruction loss value is calculated. The 3D model reconstruction model is trained by the gradient descent method in the direction of minimizing the 3D model reconstruction loss value until the training conditions are met, and the trained 3D model reconstruction model is output.
6. A bronchoscopic navigation method according to claim 5, characterized in that: The bronchial characteristic node time series analysis model is established based on the LSTM model.
7. A bronchoscopic navigation system, characterized in that: The system uses a bronchoscopic navigation method according to any one of claims 1 to 6, comprising: A virtual bronchial 3D model acquisition module is used to acquire N frames of virtual bronchial 3D models of patients arranged in time, and to form a virtual bronchial 3D model time series analysis set of the virtual bronchial 3D models of patients arranged in time, wherein the virtual bronchial 3D models include a plurality of point cloud data; The feature node extraction module is used to send the virtual bronchial three-dimensional model time series analysis set to the feature node extraction network for processing to output a bronchial feature node set, wherein the bronchial feature node set includes a number of bronchial feature nodes; the feature node extraction network includes a feature enhancement layer, a PointNet layer and a bronchial feature node set output layer, wherein the feature enhancement layer is used to analyze the time series changes of the bronchial feature nodes, and based on the time series changes of the bronchial feature nodes, the virtual bronchial three-dimensional model is feature enhanced; the PointNet layer is used to perform feature extraction operations; the bronchial feature node set output layer is used to output the bronchial feature node set; The real-time virtual bronchial three-dimensional model construction module is used to obtain the respiratory motion parameters at the monitoring time point during the bronchoscopic navigation process, splice the respiratory motion parameters at the current monitoring time point with the point cloud data corresponding to the bronchial feature nodes in the bronchial feature node set, construct the bronchial feature node analysis data, and then combine the bronchial feature node analysis data at the current monitoring time point and the bronchial feature node analysis data at the previous N-1 monitoring time points to form a bronchial feature node analysis time series data set, and send the bronchial feature node analysis time series data set to the bronchial feature node time series analysis model. Processing is performed to output a predicted bronchial feature point cloud data set for the next monitoring time point, wherein the predicted bronchial feature point cloud data set includes several predicted bronchial feature point cloud data. At the next monitoring time point, the predicted bronchial feature point cloud data set is sent to the three-dimensional model reconstruction model for processing to obtain a real-time virtual bronchial three-dimensional model of the next monitoring time point. The real-time bronchial image obtained through the bronchoscope at the next monitoring time point is image aligned with the real-time virtual bronchial three-dimensional model of the next monitoring time point, and the moving direction of the bronchoscope is indicated to realize the navigation of the bronchoscope.