A pulmonary artery segmentation method, system, device and medium based on RNN

Abstract

The present invention discloses a pulmonary artery segmentation method, system, device and medium based on RNN, which involves using artificial intelligence technology to perform pulmonary artery segmentation, and its purpose is to solve the technical problem in the prior art that the label data is incomplete due to the difficulty in labeling the peripheral branches of the pulmonary artery, which affects the accuracy of pulmonary artery segmentation. Its samples include labeled and unlabeled lung CT sample images; the pulmonary artery segmentation model includes a U-Net network and two Mamba networks with selective state spaces. The intermediate high-dimensional representation stage of the U-Net network introduces a nmODE module. One Mamba network is used to construct a scoring function that predicts the probability of each voxel being a foreground, and the other Mamba network is used to construct a labeling tendency function that estimates the possibility of the voxel being manually labeled as a foreground; the image is input into the U-Net network, and after the downsampling stage, the original features are extracted and then converted into feature representations after dynamic mapping by the nmODE module. After the upsampling stage, they are respectively input into the two Mamba networks and output the real structure and labeling status.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to pulmonary artery segmentation, and in particular to a pulmonary artery segmentation method, system, device and medium based on RNN. Background Art

[0002] The pulmonary artery is the main blood vessel that carries blood from the right ventricle to the lungs. Its tree-like structure is of great anatomical significance. Accurate segmentation of the pulmonary artery is not only crucial for the diagnosis and treatment of diseases such as pulmonary arterial hypertension (PAH), but also plays a key role in assessing pulmonary circulation function, pathological changes, and surgical planning.

[0003] With the rapid development of artificial intelligence in recent years, a large number of deep learning-based medical image segmentation methods have been proposed. The most classic is the U-Net architecture, which consists of an encoder, a decoder, and skip connections. The encoder captures the features and context of the input image through a series of convolutions and downsampling operations, gradually reducing the size of the feature map. The decoder, in contrast, gradually enlarges the feature map to its original size through a series of convolutions and upsampling operations, restoring spatial information. Skip connections directly connect the decoder and encoder layers, allowing the network to retain high-resolution information from the encoder feature map, helping to preserve details during the segmentation process. The U-Net architecture remains the mainstream architecture for medical image segmentation models, including in pulmonary artery segmentation.

[0004] The invention patent application with application number 202311359989.0 discloses a CT image pulmonary artery segmentation method and system based on contextual attention and feature fusion, which includes the following steps: in the training stage, the original CT image is obtained and image preprocessing is performed, image blocks are extracted, and a 3D U-Net network based on contextual attention and feature fusion is constructed and trained; in the inference stage, the CT image to be inferred is obtained and image preprocessing is performed, image blocks are extracted, the CT image to which the block belongs and its position in the CT image are recorded, and a pulmonary artery segmentation prediction value is obtained based on the trained 3D U-Net network; the pulmonary artery segmentation prediction values of all image blocks belonging to the same CT are combined to obtain the overall pulmonary artery segmentation prediction value of the CT image; the maximum connected component of the pulmonary artery is extracted after hard segmentation processing to obtain the segmentation result; wherein, the 3D The U-Net network includes: an encoder, a decoder, and a skip connection structure, wherein the encoder is connected to the decoder; the encoder includes a contextual attention convolution block and a downsampling operation module, which extracts abstract features from the input image block through convolution and downsampling operations, reduces the size of the feature map, increases the number of feature map channels, and obtains a feature map; the decoder includes a contextual attention convolution block and an upsampling operation module, which restores the feature map to the same size as the input image block through convolution and upsampling; the skip connection structure connects the low-level feature maps in the encoder with the high-level feature maps in the decoder.

[0005] As in the above-mentioned invention patent application, although the existing technology can achieve pulmonary artery segmentation through the U-Net network. However, it still faces some problems in the segmentation of the pulmonary artery tree structure, especially when segmenting small peripheral branches. The peripheral branches of the pulmonary artery are usually sparsely distributed and slightly changed, which makes them appear low in contrast in the image, and the accuracy of the labeling of small branches is often difficult to guarantee. Due to the difficulty in labeling the peripheral branches of the pulmonary artery in the existing technology, the labeling data of the pulmonary artery tree structure in medical images is mostly incomplete. Therefore, how to solve the problem of incomplete label data in the segmentation of the pulmonary artery tree structure and solve the problem of low segmentation accuracy of the pulmonary artery (especially small peripheral branches) caused by incomplete label data has become a major challenge in the current pulmonary artery medical image analysis and needs to be solved urgently. Summary of the Invention

[0006] The purpose of the present invention is to solve the technical problem in the prior art that the accuracy of pulmonary artery segmentation is affected by incomplete label data due to the difficulty in labeling the peripheral branches of the pulmonary artery. The present invention provides a pulmonary artery segmentation method, system, device and medium based on RNN, which uses a small amount of reliable annotations to segment the pulmonary artery tree structure, effectively improving the segmentation accuracy of small peripheral branches and reducing the workload of the annotator.

[0007] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0008] A pulmonary artery segmentation method based on RNN includes the following steps:

[0009] Step S1, obtaining sample data;

[0010] Obtain lung CT sample images covering different slices and branch structures, and annotate the pulmonary artery trunk and some branches in some lung CT sample images to obtain labeled lung CT sample images and unlabeled lung CT sample images;

[0011] Step S2, constructing a pulmonary artery segmentation model;

[0012] The pulmonary artery segmentation model includes a U-Net network and two Mamba networks with selective state spaces. The nmODE module is introduced in the intermediate high-dimensional representation stage of the U-Net network. One Mamba network is used to construct a scoring function to predict the probability of each voxel being a foreground, and the other Mamba network is used to construct a labeling propensity function to estimate the likelihood that the voxel has been manually labeled as a foreground.

[0013] Lung CT images are fed into the U-Net network. After the downsampling phase, the original features are extracted. The original features are dynamically mapped by the nmODE module and converted into feature representations with long-term memory. After the upsampling phase, they are fed into two Mamba networks. The Mamba networks output the predicted true structure and annotation status.

[0014] Step S3, training a pulmonary artery segmentation model;

[0015] The pulmonary artery segmentation model constructed in step S2 is trained using the labeled lung CT sample images and the unlabeled lung CT sample images in step S1;

[0016] Step S4, real-time segmentation;

[0017] A lung CT image to be segmented is obtained and input into the pulmonary artery segmentation model trained in step S3, and the pulmonary artery segmentation model outputs a pulmonary artery segmentation result.

[0018] Furthermore, data enhancement is performed on the lung CT sample image obtained in step S1. The specific method is as follows:

[0019] For labeled lung CT sample images, 50% of the lung CT sample images are horizontally flipped, and all or part of the remaining lung CT sample images are randomly rotated, with the rotation angle ranging from -15° to 15°;

[0020] For unlabeled lung CT sample images, different regions of the image are randomly selected for cropping, the local structure in the image is changed through nonlinear transformation, and the image is partially randomly blocked.

[0021] Furthermore, in step S2, the U-Net network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer, a first nmODE module, and a second nmODE module. The image is input to the first convolutional layer, the output of the first convolutional layer is input to the second convolutional layer after downsampling, the output of the second convolutional layer is input to the third convolutional layer after downsampling, the output of the third convolutional layer is input to the fourth convolutional layer after downsampling, and the output of the fourth convolutional layer is input to the fifth convolutional layer after downsampling; the fifth convolutional layer is combined with the first nmODE module, and the original features output by the fifth convolutional layer are converted into feature representations with long-term memory after dynamic mapping by the first nmODE module. ;

[0022] Feature Representation After upsampling, the sixth convolutional layer is input, and the sixth convolutional layer is combined with the second nmODE module. The features output by the sixth convolutional layer are dynamically mapped by the second nmODE module and converted into feature representations with long-term memory. ; Feature representation After upsampling, it is input into the seventh convolutional layer. The output of the seventh convolutional layer is input into the eighth convolutional layer after upsampling. The eighth convolutional layer is input into the ninth convolutional layer after upsampling. The output of the ninth convolutional layer is used as the input of the two Mamba networks.

[0023] Furthermore, the ordinary differential equations of the first nmODE module and the second nmODE module are:

[0024] ;

[0025] ;

[0026] in, represents a function related to time t, Represents the output function The derivative of represents a constant used to control the influence of external input on the stability of model mapping; Indicates external input; Represents a learnable weight parameter, which is used to adjust the impact of input features on the final output; Represents input features; Represents the bias term, which is used to adjust the input features In calculating external input The impact of time.

[0027] Further, in step S2, the Mamba network includes a first linear projection layer, a tenth convolutional layer, a second linear projection layer, a convolutional layer, an activation function, a selective SSM layer, and a third linear projection layer;

[0028] The output of the U-Net network is used as the input of the Mamba network and is input to the first linear projection layer and the second linear projection layer respectively. The output of the first linear projection layer is sequentially passed through the tenth convolutional layer, the activation function and the selective SSM layer, and then concatenated with the output of the second linear projection layer as the input of the third linear projection layer. The third linear projection layer outputs the predicted real structure. and marking status .

[0029] Furthermore, in step S3, when training the pulmonary artery segmentation model, the total loss function Expressed as:

[0030] ;

[0031] ;

[0032] ;

[0033] in, 、 Both represent weights, represents the scoring loss, represents the tendency loss, Indicates the The predicted labels of samples, represents the scoring function, represents the labeling tendency function, Indicates the The input features of samples, Indicates the The labeling status of the samples, 、 Represents the optimized parameters of the two loss functions, Indicates the sample size.

[0034] Furthermore, in step S3, when training the pulmonary artery segmentation model, the two Mamba networks are iteratively calculated using the EM algorithm. The E step is responsible for inferring the posterior probability of the label, and the M step updates the parameters by maximizing the likelihood function, specifically:

[0035] In the E step, the calculation formula for the posterior probability is:

[0036] ;

[0037] in, Represents a given feature When the label is The conditional probability of Representation characteristics The probability of the labeled state; Represents the conditional probability of the labeled state, indicating that the feature Whether it is marked as positive;

[0038] In the M step, according to the posterior probability obtained in the E step , the model parameters are updated by maximizing the log-likelihood function, with the goal of maximizing the joint probability of the label and the probability of the labeled state; the formula of the maximization process is expressed as:

[0039]

[0040] ;

[0041] in, represents the log-likelihood function of the parametric model, represents the number of samples, Indicates the The true labels of the samples The estimated probability of Indicates the The true labels of the samples The expected value of represents the conditional probability of the labeled state, Represents a given input feature and parameters For the The true labels of the samples The predicted probability of Represents a given input feature , true label and parameters The following model is for The labeling status of samples The predicted probability of .

[0042] A pulmonary artery segmentation system based on RNN, comprising:

[0043] A sample data acquisition module is used to acquire lung CT sample images covering different slices and branch structures, and to annotate the pulmonary artery trunk and some branches in some lung CT sample images to obtain labeled lung CT sample images and unlabeled lung CT sample images;

[0044] The pulmonary artery segmentation model construction module includes a U-Net network and two Mamba networks with selective state spaces. The nmODE module is introduced in the intermediate high-dimensional representation stage of the U-Net network. One Mamba network is used to construct a scoring function to predict the probability of each voxel being a foreground, and the other Mamba network is used to construct a labeling propensity function to estimate the likelihood that the voxel has been manually labeled as a foreground.

[0045] Lung CT images are fed into the U-Net network. After the downsampling phase, the original features are extracted. The original features are dynamically mapped by the nmODE module and converted into feature representations with long-term memory. After the upsampling phase, they are fed into two Mamba networks. The Mamba networks output the predicted true structure and annotation status.

[0046] a pulmonary artery segmentation model training module, configured to train the pulmonary artery segmentation model constructed by the pulmonary artery segmentation model construction module using the labeled pulmonary CT sample images and the unlabeled pulmonary CT sample images in the sample data acquisition module;

[0047] The real-time segmentation module is used to obtain the lung CT image to be segmented and input the pulmonary artery segmentation model trained by the pulmonary artery segmentation model training module, and the pulmonary artery segmentation model outputs the pulmonary artery segmentation result.

[0048] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0049] A computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the steps of the above method.

[0050] The beneficial effects of the present invention are as follows:

[0051] 1. In the present invention, the pulmonary artery segmentation method is based on the memory-enhanced multi-head attention mechanism (Mamba) and the neural ordinary differential equation (nmODE), which can effectively handle the complex topological relationship of the pulmonary artery tree structure, especially in the segmentation task of small peripheral branches and distal areas, significantly improving the segmentation accuracy and reducing the false negative rate; in addition, the sample images include labeled samples and unlabeled samples, and are used as sample data when training the pulmonary artery segmentation model. The dual-branch structure adopted by the model can be combined with the EM algorithm to infer potential labels during the training process, so that the model can use a small amount of reliable annotations to segment the pulmonary artery tree structure, effectively improving the segmentation accuracy of small peripheral branches, reducing the workload of the annotator, and effectively solving the problem of low segmentation accuracy of the pulmonary artery (especially small peripheral branches) due to incomplete label data.

[0052] 2. In this invention, by introducing a weakly supervised learning method, the dependence on complete labeled data is reduced, allowing efficient training through simple annotations (such as main branch annotations), and the model can self-infer small branch areas, thereby greatly reducing the labeling workload and improving labeling efficiency.

[0053] 3. In this invention, the advantages of Mamba and nmODE are combined, which can simultaneously capture global information and local details, and use the memory enhancement mechanism to maintain a stable feature representation, further enhancing the robustness of the model to complex structures, improving the resistance to image noise and interference, and ensuring the accurate segmentation of pulmonary artery branches.

[0054] 4. In the present invention, the pulmonary artery segmentation method has lower computational cost and higher efficiency, and is suitable for real-time or near real-time clinical applications. It can help doctors accurately diagnose pulmonary artery-related diseases, especially in the early detection, diagnosis and treatment of pulmonary arterial hypertension (PAH), and provide strong support for the formulation of personalized treatment plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic flow diagram of the present invention;

[0056] Figure 2 It is a schematic diagram of the structure of the U-Net network in the present invention;

[0057] Figure 3 It is a schematic diagram of the structure of the Mamba network in the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0059] Therefore, based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0060] Example 1

[0061] This embodiment provides a pulmonary artery segmentation method based on RNN, which is used to segment the pulmonary artery in the lung CT image. Figure 1 As shown, it includes the following steps:

[0062] Step S1, obtaining sample data;

[0063] Lung CT sample images covering different slices and branch structures are obtained, and the main trunk and some branches of the pulmonary artery in some lung CT sample images are annotated to obtain labeled lung CT sample images and unlabeled lung CT sample images.

[0064] For labeled sample data, the inclusion criteria are: 1) patients undergo pulmonary artery-related imaging examinations before undergoing a thin-slice chest CT scan (0.75–1.5 mm); 2) clear pulmonary artery annotation data, especially annotations of the main pulmonary artery trunk and branches; 3) the annotation data is provided by experienced medical experts, and each case contains complete pulmonary artery regional annotation information. The exclusion criteria are: 1) lack of key clinical data (such as age, gender, medical history, etc.); 2) no CT scan or the pulmonary artery region in the CT image is difficult to distinguish and annotate, such as pulmonary artery adhesion to the hilum or severe atelectasis; 3) poor data quality, resulting in the annotation not accurately reflecting the actual situation of the pulmonary artery; 4) low image quality of the CT scan, which cannot clearly show the structure of the pulmonary artery, especially the small branches.

[0065] For unlabeled sample data, the inclusion criteria were: 1) patients underwent pulmonary artery imaging before undergoing a thin-slice chest CT scan (0.75–1.5 mm); 2) included unlabeled pulmonary artery image data from the same clinical population, and the image quality met the standards. Exclusion criteria were: 1) lack of key clinical data (such as age, gender, and medical history); 2) no CT scan or the pulmonary artery region in the CT image was difficult to distinguish and label, such as pulmonary artery adhesion to the hilum or severe atelectasis; 3) poor data quality, resulting in the inability to clearly extract and analyze pulmonary artery image information.

[0066] To meet the inclusion criteria for the sample data, a total of 203 lung CT images of patients diagnosed with pulmonary nodules were collected from four medical centers and two device manufacturers. The images were standardized to a resolution of 512 x 512 pixels, with a pixel pitch of 0.5 to 0.95 mm and a slice thickness of 1 mm. All data were anonymized to ensure patient privacy. The dataset was divided into training, validation, and test sets, consisting of 100, 30, and 73 images, respectively. Specific parameters of the dataset, such as age, number of slices, and pixel pitch, were known, and detailed information such as minimum, maximum, and mean values were provided. During the annotation process, each CT image in the dataset was manually labeled by experienced radiologists and semi-automatically annotated using the region growing method using MIMICS software. First, the experts adjusted the window width and window level to ensure clear visualization of the pulmonary artery structures. Next, the experts iteratively and manually selected seed points to obtain a rough mask of the pulmonary artery structure. Subsequently, all experts fine-tuned the resulting mask of the pulmonary artery structure and mutually reviewed the annotations. Finally, the annotations from the five experts were combined using a voting method. All annotation processes are rigorously reviewed to ensure accuracy and consistency.

[0067] In order to increase the diversity of sample data and improve the generalization ability of the model, different data augmentation strategies are used for labeled and unlabeled images. The specific data augmentation methods are:

[0068] For labeled lung CT sample images, 50% of the lung CT sample images are horizontally flipped to simulate pulmonary artery images from different perspectives; all or part of the remaining lung CT sample images are randomly rotated with a rotation angle ranging from -15° to 15°. This allows the model to learn the characteristics of the pulmonary artery at different angles and avoid dependence on a specific direction.

[0069] For unlabeled lung CT sample images, different areas of the image are randomly selected for cropping to simulate local detail changes in clinical images; the local structure in the image is changed by nonlinear transformation, and the image is partially randomly occluded to simulate the common missing information problem in clinical data, thereby improving the model's adaptability to data incompleteness.

[0070] Through the combined use of the above enhancement technologies, the model is exposed to a wider range of pulmonary artery structural variations, noise interference and incomplete data during training, which significantly enhances the robustness of the model; it also enables the model to better adapt to the different quality, perspectives, missing information and noise conditions in clinical images, thereby improving the accuracy and generalization ability of pulmonary artery segmentation.

[0071] Step S2, constructing a pulmonary artery segmentation model;

[0072] The pulmonary artery segmentation model includes a U-Net network and two Mamba networks with selective state spaces. The nmODE module is introduced in the intermediate high-dimensional representation stage of the U-Net network. A Mamba network is used to construct a scoring function to predict the probability of each voxel being a foreground. , another Mamba network is used to construct a labeling tendency function that estimates the likelihood that the voxel is manually labeled as foreground , its structure is as follows Figure 2 shown.

[0073] The lung CT image is input into the U-Net network, and the original features are extracted after the downsampling stage. The original features are dynamically mapped by the nmODE module and converted into feature representations with long-term memory. , and then after the upsampling stage, they are input into two Mamba networks respectively, and the Mamba network outputs the predicted real structure and marking status .

[0074] like Figure 2 As shown, the U-Net network includes the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, the fifth convolutional layer, the sixth convolutional layer, the seventh convolutional layer, the eighth convolutional layer, the ninth convolutional layer, the first nmODE module, and the second nmODE module. The image is input to the first convolutional layer, the output of the first convolutional layer is input to the second convolutional layer after downsampling, the output of the second convolutional layer is input to the third convolutional layer after downsampling, the output of the third convolutional layer is input to the fourth convolutional layer after downsampling, and the output of the fourth convolutional layer is input to the fifth convolutional layer after downsampling; the fifth convolutional layer is combined with the first nmODE module, and the original features output by the fifth convolutional layer are converted into feature representations with long-term memory after dynamic mapping by the first nmODE module. ;

[0075] Feature Representation After upsampling, the sixth convolutional layer is input, and the sixth convolutional layer is combined with the second nmODE module. The features output by the sixth convolutional layer are dynamically mapped by the second nmODE module and converted into feature representations with long-term memory. ; Feature representation After upsampling, it is input into the seventh convolutional layer. The output of the seventh convolutional layer is input into the eighth convolutional layer after upsampling. The eighth convolutional layer is input into the ninth convolutional layer after upsampling. The output of the ninth convolutional layer is used as the input of the two Mamba networks.

[0076] like Figure 3 As shown, the Mamba network includes the first linear projection layer, the tenth convolutional layer, the second linear projection layer, the convolutional layer, the activation function, the selective SSM layer, and the third linear projection layer;

[0077] The output of the U-Net network is used as the input of the Mamba network and is input to the first linear projection layer and the second linear projection layer respectively. The output of the first linear projection layer is sequentially passed through the tenth convolutional layer, the activation function and the selective SSM layer, and then concatenated with the output of the second linear projection layer as the input of the third linear projection layer. The third linear projection layer outputs the predicted real structure. and marking status .

[0078] The U-Net network is used as a feature extraction module. It is built on the U-Net network and utilizes the differentiable mapping of nmODE. The U-Net network consists of four downsampling stages, each of which includes a 3*3*3 convolution layer, followed by instance normalization and Leaky-ReLU activation function. The nmODE module is a continuous dynamic system, and a stable mapping can be established between its data space and representation space through a global attractor. Since images are sparse in low-dimensional space, the nmODE module is applied to high-dimensional space to give full play to its powerful mapping capabilities. The nmODE module is applied to high-dimensional data after U-Net network feature extraction. Among them, the ordinary differential equations of the first nmODE module and the second nmODE module are:

[0079] ;

[0080] ;

[0081] in, represents a function related to time t, Represents the output function The derivative of represents a constant used to control the influence of external input on the stability of model mapping; Indicates external input; Represents a learnable weight parameter, which is used to adjust the impact of input features on the final output; Represents input features; Represents the bias term, which is used to adjust the input features In calculating external input The impact of time.

[0082] is a nonlinear term, which is obtained by adding external input Ability to dynamically adjust features according to different input conditions; given input features and initial value When the nmODE module gives an output that changes with time t .along with , the output will converge to an attractor, which represents a stable state in the feature space.

[0083] Step S3, training a pulmonary artery segmentation model;

[0084] The pulmonary artery segmentation model constructed in step S2 is trained using the labeled lung CT sample images and the unlabeled lung CT sample images in step S1.

[0085] When training the pulmonary artery segmentation model, a weakly supervised learning method is used for training, specifically:

[0086] Assume that each sample is a voxel instance in the feature space, and the annotation is indistinguishable, so there is a label corresponds to the true label of the voxel. In addition, we define is a random variable indicating whether the voxel is labeled as a positive class (i.e. indicates that it is labeled as positive class).

[0087] During the annotation process, annotation bias is introduced, where the annotation status Features of the instance and tags Specifically, the labeling process can be modeled by the following formula:

[0088] ;

[0089] in, Represents a given feature Time Tags The conditional probability of Represents a given label and features Time mark status The probability of Represents the conditional probability of the labeled state.

[0090] During the marking process, The probability of To model, the function describes the relationship between the annotation state and the features and labels, and the formula is expressed as:

[0091] ;

[0092] in, Represents the parameters of the function, which is used for feature learning in the labeling process.

[0093] According to the above definition, the goal of weakly supervised learning is to find the labeled state through inference and optimization and the true label During the training process, the EM algorithm is used to infer the potential labels. , and use the labeled data to optimize the model parameters. Specifically:

[0094] Given features and annotation status Under observable conditions, the true label is potential. Using the conditional joint distribution given by the following formula, the goal of the model is to infer the label , to build a segmentation model. The specific conditional joint distribution can be expressed as follows:

[0095] ;

[0096] This formula shows how to infer potential labels based on sample features and annotation information given a training set. and annotation status The joint distribution of .

[0097] On the basis of the above weakly supervised learning, the EM algorithm is used to enhance the label inference ability of the model and improve the segmentation accuracy. In the E step, the EM algorithm calculates the posterior probability of each instance. , inferred labels ; Then optimize the model parameters in the M step; By alternating these two steps, the model is gradually optimized and infers the most likely label, thereby improving the accuracy of pulmonary artery segmentation.

[0098] Conditional probability , which is a given input feature The posterior probability when , is given by the following scoring function To decide, there are the following facts:

[0099] ;

[0100] in, is the input feature The probability of being labeled as positive. In order to estimate the parameter , we need to solve the following maximization problem:

[0101] ;

[0102] By taking the logarithm of the right side of the above formula, the maximum likelihood function is equivalent to:

[0103] ;

[0104] Then, through alternating iterations of the E-step and the M-step, the EM algorithm can effectively train the segmentation model by inferring the label y and optimizing the model parameters in the case of incomplete labeling. In this process, the E-step is responsible for inferring the posterior probability of the label, and the M-step updates the model parameters by maximizing the likelihood function, as follows:

[0105] In the E step, the calculation formula for the posterior probability is:

[0106] ;

[0107] in, Represents a given feature When the label is The conditional probability of Representation characteristics The probability of the labeled state; Represents the conditional probability of the labeled state, indicating that the feature Whether it is marked as positive;

[0108] In the M step, according to the posterior probability obtained in the E step , the model parameters are updated by maximizing the log-likelihood function, with the goal of maximizing the joint probability of the label and the probability of the labeled state; the formula of the maximization process is expressed as:

[0109]

[0110] ;

[0111] in, represents the log-likelihood function of the parametric model, represents the number of samples, Indicates the The true labels of the samples The estimated probability of Indicates the The true labels of the samples The expected value of represents the conditional probability of the labeled state, Represents a given input feature and parameters For the The true labels of the samples The predicted probability of Represents a given input feature , true label and parameters The following model is for The labeling status of samples The predicted probability of .

[0112] This step updates the model by optimizing parameters, corresponding to the model for assigning labels and the model for annotating states.

[0113] Features extracted by the U-Net network when training the pulmonary artery segmentation model is passed as input to the Mamba network to construct the scoring function and labeled propensity function , and the Mamba network helps the model pay more attention to small branches and complex structures through memory enhancement and multi-head attention mechanism. According to the results of the EM algorithm, two parameters need to be updated, corresponding to two loss functions. Therefore, the total loss function Expressed as:

[0114] ;

[0115] ;

[0116] ;

[0117] in, 、 Both represent weights, represents the scoring loss, represents the tendency loss, Indicates the The predicted labels of samples, represents the scoring function, represents the labeling tendency function, Indicates the The input features of samples, Indicates the The labeling status of the samples, 、 Represents the optimized parameters of the two loss functions, Indicates the sample size.

[0118] Step S4, real-time segmentation;

[0119] A lung CT image to be segmented is obtained and input into the pulmonary artery segmentation model trained in step S3, and the pulmonary artery segmentation model outputs a pulmonary artery segmentation result.

[0120] Example 2

[0121] This embodiment provides a RNN-based pulmonary artery segmentation system, which includes:

[0122] A sample data acquisition module is used to acquire lung CT sample images covering different slices and branch structures, and to annotate the pulmonary artery trunk and some branches in some lung CT sample images to obtain labeled lung CT sample images and unlabeled lung CT sample images;

[0123] The pulmonary artery segmentation model construction module includes a U-Net network and two Mamba networks with selective state spaces. The nmODE module is introduced in the intermediate high-dimensional representation stage of the U-Net network. One Mamba network is used to construct a scoring function to predict the probability of each voxel being a foreground, and the other Mamba network is used to construct a labeling propensity function to estimate the likelihood that the voxel has been manually labeled as a foreground.

[0124] Lung CT images are fed into the U-Net network. After the downsampling phase, the original features are extracted. The original features are dynamically mapped by the nmODE module and converted into feature representations with long-term memory. After the upsampling phase, they are fed into two Mamba networks. The Mamba networks output the predicted true structure and annotation status.

[0125] a pulmonary artery segmentation model training module, configured to train the pulmonary artery segmentation model constructed by the pulmonary artery segmentation model construction module using the labeled pulmonary CT sample images and the unlabeled pulmonary CT sample images in the sample data acquisition module;

[0126] The real-time segmentation module is used to obtain the lung CT image to be segmented and input the pulmonary artery segmentation model trained by the pulmonary artery segmentation model training module, and the pulmonary artery segmentation model outputs the pulmonary artery segmentation result.

[0127] For more specific details of each module, please refer to the description in Example 1.

[0128] Example 3

[0129] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a RNN-based pulmonary artery segmentation method.

[0130] The computer device may be a desktop computer, a notebook computer, a PDA, a cloud server, etc. The computer device may interact with the user via a keyboard, a mouse, a remote control, a touchpad, or a voice control device.

[0131] The memory includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or D interface display memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory may be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. In other embodiments, the memory may also be an external storage device of the computer device, such as a plug-in hard disk equipped with the computer device, a smart memory card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is often used to store the operating system and various application software installed on the computer device, such as the program code of the RNN-based pulmonary artery segmentation method. In addition, the memory may also be used to temporarily store various types of data that have been output or are about to be output.

[0132] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is typically used to control the overall operation of the computer device. In this embodiment, the processor is used to execute program code stored in the memory or process data, such as executing the program code of the RNN-based pulmonary artery segmentation method.

[0133] Example 4

[0134] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor performs the steps of a RNN-based pulmonary artery segmentation method.

[0135] The computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor to enable the at least one processor to perform the steps of the RNN-based pulmonary artery segmentation method as described above.

[0136] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment method can be implemented by means of software plus the necessary general hardware platform, of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions for enabling a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the RNN-based pulmonary artery segmentation method described in the embodiment of the present application.

Claims

1. A pulmonary artery segmentation method based on RNN, characterized in that: The following steps are involved: Step S1, obtaining sample data; Obtain lung CT sample images covering different slices and branch structures, and annotate the pulmonary artery trunk and some branches in some lung CT sample images to obtain labeled lung CT sample images and unlabeled lung CT sample images; Step S2, constructing a pulmonary artery segmentation model; The pulmonary artery segmentation model includes a U-Net network and two Mamba networks with selective state spaces. The nmODE module is introduced in the intermediate high-dimensional representation stage of the U-Net network. One Mamba network is used to construct a scoring function to predict the probability of each voxel being a foreground, and the other Mamba network is used to construct a labeling propensity function to estimate the likelihood that the voxel has been manually labeled as a foreground. Lung CT images are fed into the U-Net network. After the downsampling phase, the original features are extracted. The original features are dynamically mapped by the nmODE module and converted into feature representations with long-term memory. After the upsampling phase, they are fed into two Mamba networks. The Mamba networks output the predicted true structure and annotation status. Step S3, training a pulmonary artery segmentation model; The pulmonary artery segmentation model constructed in step S2 is trained using the labeled lung CT sample images and the unlabeled lung CT sample images in step S1; Step S4, real-time segmentation; A lung CT image to be segmented is obtained and input into the pulmonary artery segmentation model trained in step S3, and the pulmonary artery segmentation model outputs a pulmonary artery segmentation result.

2. The RNN-based pulmonary artery segmentation method according to claim 1, characterized in that: Data enhancement is performed on the lung CT sample image obtained in step S1. The specific method is as follows: For labeled lung CT sample images, 50% of the lung CT sample images are horizontally flipped, and all or part of the remaining lung CT sample images are randomly rotated, with the rotation angle ranging from -15° to 15°; For unlabeled lung CT sample images, different regions of the image are randomly selected for cropping, the local structure in the image is changed through nonlinear transformation, and the image is partially randomly blocked.

3. The RNN-based pulmonary artery segmentation method according to claim 1, characterized in that: In step S2, the U-Net network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, an eighth convolutional layer, a ninth convolutional layer, a first nmODE module, and a second nmODE module. The image is input to the first convolutional layer, the output of the first convolutional layer is input to the second convolutional layer after downsampling, the output of the second convolutional layer is input to the third convolutional layer after downsampling, the output of the third convolutional layer is input to the fourth convolutional layer after downsampling, and the output of the fourth convolutional layer is input to the fifth convolutional layer after downsampling; the fifth convolutional layer is combined with the first nmODE module, and the original features output by the fifth convolutional layer are converted into feature representations with long-term memory after dynamic mapping by the first nmODE module. ; Feature Representation After upsampling, the sixth convolutional layer is input, and the sixth convolutional layer is combined with the second nmODE module. The features output by the sixth convolutional layer are dynamically mapped by the second nmODE module and converted into feature representations with long-term memory. ; Feature representation After upsampling, it is input into the seventh convolutional layer. The output of the seventh convolutional layer is input into the eighth convolutional layer after upsampling. The eighth convolutional layer is input into the ninth convolutional layer after upsampling. The output of the ninth convolutional layer is used as the input of the two Mamba networks.

4. The RNN-based pulmonary artery segmentation method according to claim 3, characterized in that: The ordinary differential equations of the first nmODE module and the second nmODE module are: ； ； in, represents a function related to time t, Represents the output function The derivative of represents a constant used to control the influence of external input on the stability of model mapping; Indicates external input; Represents a learnable weight parameter, which is used to adjust the impact of input features on the final output; Represents input features; Represents the bias term, which is used to adjust the input features In calculating external input The impact of time.

5. The RNN-based pulmonary artery segmentation method according to claim 3, characterized in that: In step S2, the Mamba network includes a first linear projection layer, a tenth convolutional layer, a second linear projection layer, a convolutional layer, an activation function, a selective SSM layer, and a third linear projection layer; The output of the U-Net network is used as the input of the Mamba network and is input to the first linear projection layer and the second linear projection layer respectively. The output of the first linear projection layer is sequentially passed through the tenth convolutional layer, the activation function and the selective SSM layer, and then concatenated with the output of the second linear projection layer as the input of the third linear projection layer. The third linear projection layer outputs the predicted real structure. and marking status .

6. The RNN-based pulmonary artery segmentation method according to claim 1, characterized in that: In step S3, when training the pulmonary artery segmentation model, the total loss function Expressed as: ； ； ； in, 、 Both represent weights, represents the scoring loss, represents the tendency loss, Indicates the The predicted labels of samples, represents the scoring function, represents the labeling tendency function, Indicates the The input features of samples, Indicates the The labeling status of the samples, 、 Represents the optimized parameters of the two loss functions, Indicates the sample size.

7. The RNN-based pulmonary artery segmentation method according to claim 6, characterized in that: In step S3, when training the pulmonary artery segmentation model, the two Mamba networks are iteratively calculated using the EM algorithm. The E step is responsible for inferring the posterior probability of the label, and the M step updates the parameters by maximizing the likelihood function. Specifically, In the E step, the calculation formula for the posterior probability is: ； in, Represents a given feature When the label is The conditional probability of Representation characteristics The probability of the labeled state; Represents the conditional probability of the labeled state, indicating that the feature Whether it is marked as positive; In the M step, according to the posterior probability obtained in the E step , the model parameters are updated by maximizing the log-likelihood function, with the goal of maximizing the joint probability of the label and the probability of the labeled state; the formula of the maximization process is expressed as: ； in, represents the log-likelihood function of the parametric model, represents the number of samples, Indicates the The true labels of the samples The estimated probability of Indicates the The true labels of the samples The expected value of represents the conditional probability of the labeled state, Represents a given input feature and parameters For the The true labels of the samples The predicted probability of Represents a given input feature , true label and parameters The following model is for The labeling status of samples The predicted probability of .

8. A pulmonary artery segmentation system based on RNN, characterized in that: include: A sample data acquisition module is used to acquire lung CT sample images covering different slices and branch structures, and to annotate the pulmonary artery trunk and some branches in some lung CT sample images to obtain labeled lung CT sample images and unlabeled lung CT sample images; The pulmonary artery segmentation model construction module includes a U-Net network and two Mamba networks with selective state spaces. The nmODE module is introduced in the intermediate high-dimensional representation stage of the U-Net network. One Mamba network is used to construct a scoring function to predict the probability of each voxel being a foreground, and the other Mamba network is used to construct a labeling propensity function to estimate the likelihood that the voxel has been manually labeled as a foreground. Lung CT images are fed into the U-Net network. After the downsampling phase, the original features are extracted. The original features are dynamically mapped by the nmODE module and converted into feature representations with long-term memory. After the upsampling phase, they are fed into two Mamba networks. The Mamba networks output the predicted true structure and annotation status. a pulmonary artery segmentation model training module, configured to train the pulmonary artery segmentation model constructed by the pulmonary artery segmentation model construction module using the labeled pulmonary CT sample images and the unlabeled pulmonary CT sample images in the sample data acquisition module; The real-time segmentation module is used to obtain the lung CT image to be segmented and input the pulmonary artery segmentation model trained by the pulmonary artery segmentation model training module, and the pulmonary artery segmentation model outputs the pulmonary artery segmentation result.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the RNN-based pulmonary artery segmentation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor is caused to perform the steps of a RNN-based pulmonary artery segmentation method according to any one of claims 1 to 7.

Images