Heart image semi-supervised segmentation method based on global-local information fusion LPV-Net and 3D-EDA

Abstract

The invention discloses a heart image semi-supervised segmentation method based on global-local information fusion LPV-Net and 3D-EDA, relates to the technical field of medical image segmentation, and combines global-local information fusion LPV-Net branches and 3D-EDA branches. The framework utilizes two different attention mechanisms to capture different dependencies in medical images to improve model performance. A 3D-EDA discriminator model is adopted, and the performance of the model is enhanced by using the idea of adversarial learning, especially the processing of unlabeled data. The overall loss function is composed of supervised loss and unsupervised loss, training is carried out through back propagation and an optimizer to gradually adjust model parameters so as to minimize the total loss, good performance is achieved, and accurate segmentation of the left atrium can be achieved. And the accuracy and efficiency of heart image segmentation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a cardiac image semi-supervised segmentation method based on global-local information fusion LPV-Net and 3D-EDA. Background Art

[0002] Heart segmentation can provide accurate information about the structure of the heart, assisting doctors in making diagnoses, formulating treatment plans, and conducting surgical evaluations, thereby significantly improving clinical treatment outcomes and reducing the occurrence of complications. However, heart segmentation technology still faces many challenges in practical applications. Traditional manual segmentation methods are not only time-consuming and labor-intensive, but also highly subjective. Although fully supervised heart segmentation methods have achieved certain results, they rely on a large amount of labeled data. Existing semi-supervised segmentation methods still perform less than ideally when dealing with complex heart structures and pathological changes, making them difficult to apply stably in actual clinical environments. Summary of the invention

[0003] In order to overcome the shortcomings of the above technologies, the present invention provides a method for semi-supervised segmentation of cardiac images based on global-local information fusion LPV-Net and 3D-EDA to improve the accuracy and efficiency of cardiac image segmentation. The technical solution adopted by the present invention to overcome the technical problems is:

[0004] A semi-supervised segmentation method for cardiac images based on global-local information fusion LPV-Net and 3D-EDA, comprising:

[0005] a) Obtain i labeled or unlabeled cardiac MRI images to form a dataset X, where X = {X l ,X u}, X l is a labeled cardiac MRI image dataset, X u is an unlabeled cardiac MRI image dataset. is the jth labeled cardiac MRI image, j∈{1,...,p}, is the jth unlabeled cardiac MRI image, j∈{1,...,q}, p is the number of labeled cardiac MRI images, q is the number of unlabeled cardiac MRI images, p+q=i;

[0006] b) Preprocess the data set X to obtain the preprocessed data set X′, X′={X l′ ,X u′}, X l′ is the preprocessed labeled cardiac MRI image dataset, X u′ It is a preprocessed unlabeled cardiac MRI image dataset;

[0007] c) Divide the preprocessed data set X′ into a training set X L , test set X U ;

[0008] d) Establish a GLB-Net model. The GLB-Net model consists of an LPV-Net branch and a 3D-EDA branch. The LPV-Net branch consists of an encoder, a LinformerPerformerBlock, and a decoder.

[0009] e) The training set X L The image in is input into the encoder of the LPV-Net branch, and the output is the feature map Fm 5-j ;

[0010] f) The feature map Fm 5-j Input to the LinformerPerformerBlock of the LPV-Net branch, and output the feature map Fm″′ 5-j ;

[0011] g) The feature map Fm″′ 5-j Input into the decoder of the LPV-Net branch and output the segmentation map

[0012] h) The prediction graph Input into the 3D-EDA branch to obtain the unsupervised loss function and the supervised loss function; i) Calculate the total loss L based on the unsupervised loss function and the supervised loss function total , using the Adam optimizer with a total loss L total Train the GLB-Net model to obtain the optimized GLB-Net model;

[0013] j) The test set X U The image in is input into the LPV-Net branch of the optimized GLB-Net model, and the segmentation map is output. Furthermore, in step a), i labeled or unlabeled left atrium cardiac MRI images are extracted from the 2018 left atrium segmentation challenge dataset.

[0014] Further, step b) comprises the following steps:

[0015] b-1) Take the jth labeled cardiac MRI image The area to be segmented is marked as 1, and the rest of the area is marked as 0. Use the np.nonzero function to find the jth labeled cardiac MRI image. The coordinates of all non-zero pixels in the label are calculated by using the np.min function to obtain the minimum value of the coordinates of all non-zero pixels, and the maximum value of the coordinates of all non-zero pixels is calculated by using the np.max function. The minimum rectangular frame containing all cardiac structures is obtained based on the minimum and maximum values ​​of the coordinates.

[0016] b-2) Use standard Numpy slicing operations on the jth labeled cardiac MRI image according to the minimum rectangular box Perform cropping to obtain the cropped image

[0017] b-3) The cropped image The image is transformed by the RandomCrop function Image Perform normalization operation to obtain the preprocessed image Preprocessed labeled cardiac MRI image dataset

[0018] b-4) Take the jth unlabeled cardiac MRI image The image is randomly cropped by RandomCrop to obtain the cropped image

[0019] b-5) The cropped image The preprocessed image is obtained by randomly rotating 0, 90, 180 or 270 degrees Preprocessed unlabeled cardiac MRI image dataset

[0020] Further, from the preprocessed labeled cardiac MRI image data X l′ Select M preprocessed images and the preprocessed unlabeled cardiac MRI image data X u′ Select O preprocessed images from the training set X L , From the preprocessed unlabeled cardiac MRI image dataset X u′ Select N preprocessed images to get the training set X U ,

[0021] Further, step e) comprises the following steps:

[0022] e-1) The encoder of the LPV-Net branch consists of the first ConvBlock block, the first DownsamplingConvBlock block, the second ConvBlock block, the second DownsamplingConvBlock block, the third ConvBlock block, the fourth DownsamplingConvBlock block, and the fifth ConvBlock block;

[0023] e-2) The first ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function. L The jth preprocessed image in Input into the first ConvBlock block and output the feature map Fm 1-j The first DownsamplingConvBlock block of the encoder is composed of a three-dimensional convolutional layer and a ReLU activation function. 1-j Input to the first DownsamplingConvBlock block, and output the feature map Fm′ 1-j ;

[0024] e-3) The second ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function. 1-j Input into the second ConvBlock block and output the feature map Fm 2-j The second DownsamplingConvBlock block of the encoder is composed of a three-dimensional convolutional layer and a ReLU activation function. 2-j Input to the second DownsamplingConvBlock block, and output the feature map Fm′ 2-j ;

[0025] e-4) The third ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function in sequence, and the feature map Fm′ is transformed into 2-j Input into the third ConvBlock block and output the feature map Fm 3-j The third DownsamplingConvBlock block of the encoder is composed of a three-dimensional convolutional layer and a ReLU activation function. 3-j Input to the third DownsamplingConvBlock block, and output the feature map Fm′ 3-j ;

[0026] e-5) The fourth ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function in sequence, and the feature map Fm′ is transformed into3-j Input to the fourth ConvBlock block and output the feature map Fm 4-j The fourth DownsamplingConvBlock block of the encoder is composed of a three-dimensional convolutional layer and a ReLU activation function, which transforms the feature map Fm 4-j Input to the fourth DownsamplingConvBlock block, and output the feature map Fm′ 4-j ;

[0027] e-6) The fifth ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function in sequence, and the feature map Fm′ is transformed into 4-j Input into the fifth ConvBlock block and output the feature map Fm 5-j .

[0028] Further, step f) comprises the following steps:

[0029] f-1) The LinformerPerformerBlock of the LPV-Net branch consists of the Linformer attention mechanism, the Performer attention mechanism, and the LayerNorm layer;

[0030] f-2) The feature map Fm 5-j Input into the Linformer attention mechanism of LinformerPerformerBlock, and output the feature map Fm″ 5-j ;

[0031] f-3) The feature map Fm″ 5-j Input to the Performer attention mechanism of LinformerPerformerBlock, and output the feature map Fm″′ 5-j ;

[0032] f-4) The feature map Fm″′ 5-j Input to the LayerNorm layer of LinformerPerformerBlock, and output the feature map Fm″′ 5-j .

[0033] Further, step g) comprises the following steps:

[0034] g-1) The decoder of the LPV-Net branch is composed of a first UpsamplingDeconvBlock block, a first ConvBlock block, a second UpsamplingDeconvBlock block, a second ConvBlock block, a third UpsamplingDeconvBlock block, a third ConvBlock block, a fourth UpsamplingDeconvBlock block, and a fourth ConvBlock block;

[0035] g-2) The first UpsamplingDeconvBlock of the decoder is composed of a three-dimensional convolutional layer and a ReLU activation function, and the feature map Fm″′ 5-j Input to the first UpsamplingDeconvBlock block, and output the feature map Fm′ 5-j ;

[0036] g-3) The first ConvBlock block of the decoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function. 5-j Input into the first ConvBlock block and output the feature map Fm 6-j The second UpsamplingDeconvBlock of the decoder is composed of a three-dimensional convolutional layer and a ReLU activation function. 6-j Input to the second UpsamplingDeconvBlock block, and output the feature map Fm′ 6-j ;

[0037] g-4) The second ConvBlock block of the decoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function. 6-j Input into the second ConvBlock block and output the feature map Fm 7-j The third UpsamplingDeconvBlock of the decoder is composed of a three-dimensional convolutional layer and a ReLU activation function. 7-j Input to the third UpsamplingDeconvBlock block, and output the feature map Fm′ 7-j ;

[0038] g-5) The third ConvBlock block of the decoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function in sequence, and the feature map Fm′ is transformed into 7-j Input into the third ConvBlock block and output the feature map Fm 8-jThe fourth UpsamplingDeconvBlock of the decoder is composed of a three-dimensional convolutional layer and a ReLU activation function. 8-j Input to the fourth UpsamplingDeconvBlock block, and output the feature map Fm′ 8-j ;

[0039] g-6) The fourth ConvBlock block of the decoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function, and the feature map Fm′ is transformed into 8-j Input to the fourth ConvBlock block, and output the feature map Fm 9-j , the feature map Fm 9-j Use the tanh function to get the horizontal distance graph The horizontal distance map Input into the softmax function and output the segmentation map Further, step h) comprises the following steps:

[0040] h-1) The 3D-EDA branch of the GLB-Net model consists of the first convolutional layer, the second convolutional layer, the third convolutional layer, the first LeakyReLU activation function, the first dropout layer, the fourth convolutional layer, the second LeakyReLU activation function, the second dropout layer, the fifth convolutional layer, the third LeakyReLU activation function, the third dropout layer, the CARELayer module, the pooling layer, and the fully connected layer;

[0041] h-2) The segmentation map Input into the first convolutional layer of the 3D-EDA branch, and output the feature map Em0. L The jth preprocessed image in Input into the second convolutional layer of the 3D-EDA branch, output the feature map Em1, add the feature map Em0 and the feature map Em1 to get the feature map Em2;

[0042] h-3) Input the feature map Em2 into the third convolution layer, the first LeakyReLU activation function, and the first dropout layer of the 3D-EDA branch in sequence, and output the feature map Em3;

[0043] h-4) The feature map Em3 is sequentially input into the fourth convolutional layer, the second LeakyReLU activation function, and the second dropout layer of the 3D-EDA branch, and the feature map Em4 is output;

[0044] h-5) The feature map Em4 is sequentially input into the fifth convolutional layer, the third LeakyReLU activation function, and the third dropout layer of the 3D-EDA branch, and the feature map Em5 is output;

[0045] h-6) The CARELLayer module of the 3D-EDA branch is composed of an average pooling layer, a first fully connected layer, a second fully connected layer, and an attention mechanism in sequence. The feature map Em5 is input into the average pooling layer of the CARELLayer module, and the output is a vector z. The vector z is input into the first fully connected layer of the CARELLayer module, and the output is a weight vector z1. The weight vector z1 is input into the second fully connected layer of the CARELLayer module, and the output is a weight vector z2. The weight vector z2 is input into the attention mechanism of the CARELLayer module, and the output is an enhanced feature map Em6.

[0046] h-7) The feature map Em6 is input into the pooling layer and the fully connected layer of the 3D-EDA branch in sequence, and the binary classification result is output.

[0047] Furthermore, in step i), by formula L total =L sup +L con ×L adv Calculate the total loss L total , where L sup is the unsupervised loss function, L sup =L dice +βL mse , L dice is the dice loss function, β is the adjustment factor, L mse is the MSE loss function, L con is the consistency loss function, L adv is the unsupervised loss function calculated by the binary classification results.

[0048] Preferably, when training the GLB-Net model, the initial learning rate is set to 0.01, the batch_size is set to 4, the epoch is set to 16000, and the lr is set to 0.001.

[0049] The beneficial effects of the present invention are: combining the global-local information fusion LPV-Net branch and the 3D-EDA branch. The framework uses two different attention mechanisms to capture different dependencies in medical images to improve model performance. The discriminator model of 3D-EDA is adopted, and the idea of ​​adversarial learning is used to enhance the performance of the model, especially for the processing of unlabeled data. The overall loss function consists of supervised loss and unsupervised loss, and is trained through back propagation and optimizer to gradually adjust the model parameters to minimize the total loss. It has good performance and can achieve accurate segmentation of the left atrium. The accuracy and efficiency of cardiac image segmentation are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a structural diagram of the GLB-Net model of the present invention;

[0051] Figure 2 It is a structural diagram of the LPV-Net branch of the present invention. DETAILED DESCRIPTION

[0052] The following is combined with Figure 1 , Attachment Figure 2 The present invention is further described.

[0053] A semi-supervised segmentation method for cardiac images based on global-local information fusion LPV-Net and 3D-EDA, comprising:

[0054] a) Obtain i labeled or unlabeled cardiac MRI images to form a dataset X, where X = {X l ,X u}, X l is a labeled cardiac MRI image dataset, X u is an unlabeled cardiac MRI image dataset. is the jth labeled cardiac MRI image, j∈{1,...,p}, is the j-th unlabeled cardiac MRI image, j∈{1,...,q}, p is the number of labeled cardiac MRI images, q is the number of unlabeled cardiac MRI images, and p+q=i.

[0055] b) Preprocess the data set X to obtain the preprocessed data set X′, X′={X l′ ,X u′}, X l′ is the preprocessed labeled cardiac MRI image dataset, X u′ It is a preprocessed unlabeled cardiac MRI image dataset.

[0056] c) Divide the preprocessed data set X′ into a training set X L , test set X U .

[0057] d) Establish the GLB-Net model. The GLB-Net model consists of the LPV-Net branch and the 3D-EDA branch. The LPV-Net branch consists of the encoder, LinformerPerformerBlock, and decoder.

[0058] e) The training set X L The image in is input into the encoder of the LPV-Net branch, and the output is the feature map Fm 5-j .

[0059] f) The feature map Fm 5-j Input to the LinformerPerformerBlock of the LPV-Net branch, and output the feature map Fm″′ 5-j .

[0060] g) The feature map Fm″′ 5-j Input into the decoder of the LPV-Net branch and output the segmentation map

[0061] h) The prediction graph Input into the 3D-EDA branch to obtain the unsupervised loss function and the supervised loss function. i) Calculate the total loss L based on the unsupervised loss function and the supervised loss function total , using the Adam optimizer with a total loss L total Train the GLB-Net model and obtain the optimized GLB-Net model.

[0062] j) The test set X U The image in is input into the LPV-Net branch of the optimized GLB-Net model, and the segmentation map is output. Table 1. Comparison of experimental results

[0063]

[0064] As shown in Table 1, in order to verify the reliability of this patented method, a series of experiments were conducted, in which different models were implemented for the Dice coefficient, Jaccard coefficient, 95HD coefficient, and ASD coefficient in the LA dataset. In our experimental method, we compared this model with seven state-of-the-art semi-supervised methods, including consistency-based (MC-Net, DTC), uncertainty-based (UA-MT, DUWM), constraint-based (SASSNet), low-entropy-based (SS-Net), and multi-scale-based (URPC), using the fully supervised model training results, 10% labeled data, and 20% labeled data, respectively. The results show that it is worth noting that, only in the case of labeled training data, the average Dice coefficient of the V-Net model reached 89.01%, and the Jaccard coefficient reached 82.13%. The proposed GLB-Net framework is superior to the upper limit of other methods in terms of Dice of 90.39%, Jaccard of 82.54%, and ASD of 1.77 mm in the left atrium dataset with 20% labeled data, but the 95HD effect is lower than that of V-Net.

[0065] In one embodiment of the present invention, in step a), i labeled or unlabeled left atrium cardiac MRI images are extracted from the 2018 Left Atrium Segmentation Challenge dataset.

[0066] In one embodiment of the present invention, step b) comprises the following steps:

[0067] b-1) Take the jth labeled cardiac MRI image The area to be segmented is marked as 1, and the rest of the area is marked as 0. Use the np.nonzero function to find the jth labeled cardiac MRI image. The coordinates of all non-zero pixels in the label are calculated, the minimum value of the coordinates of all non-zero pixels is calculated by the np.min function, the maximum value of the coordinates of all non-zero pixels is calculated by the np.max function, and the minimum rectangular frame containing all cardiac structures is obtained according to the minimum value and the maximum value of the coordinates.

[0068] b-2) Use standard Numpy slicing operations on the jth labeled cardiac MRI image according to the minimum rectangular box Perform cropping to obtain the cropped image

[0069] b-3) The cropped image The image is transformed by the RandomCrop function Image Perform normalization operation to obtain the preprocessed image Preprocessed labeled cardiac MRI image dataset

[0070] b-4) Take the jth unlabeled cardiac MRI image The image is randomly cropped by RandomCrop to obtain the cropped image

[0071] b-5) The cropped image The preprocessed image is obtained by randomly rotating 0, 90, 180 or 270 degrees Preprocessed unlabeled cardiac MRI image dataset

[0072] In one embodiment of the present invention, from the pre-processed labeled cardiac MRI image data X l′ Select M preprocessed images and the preprocessed unlabeled cardiac MRI image data X u′ Select O preprocessed images from the training set X L , From the preprocessed unlabeled cardiac MRI image dataset X u′ Select N preprocessed images to get the training set X U , In one embodiment of the present invention, step e) comprises the following steps:

[0073] e-1) The encoder of the LPV-Net branch consists of the first ConvBlock block, the first DownsamplingConvBlock block, the second ConvBlock block, the second DownsamplingConvBlock block, the third ConvBlock block, the fourth DownsamplingConvBlock block, and the fifth ConvBlock block.

[0074] e-2) The first ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function. L The jth preprocessed image in Input into the first ConvBlock block and output the feature map Fm 1-j , the ReLU activation function undergoes nonlinear changes. The first DownsamplingConvBlock block of the encoder is composed of a three-dimensional convolutional layer and a ReLU activation function in sequence. 1-j Input to the first DownsamplingConvBlock block, and output the feature map Fm′ with reduced spatial resolution 1-j .

[0075] e-3) The second ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function. 1-j Input into the second ConvBlock block and output the feature map Fm 2-j , the ReLU activation function undergoes nonlinear changes. The second DownsamplingConvBlock block of the encoder is composed of a three-dimensional convolutional layer and a ReLU activation function in sequence. 2-j Input to the second DownsamplingConvBlock block, and output the feature map Fm′ with reduced spatial resolution 2-j .

[0076] e-4) The third ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function in sequence, and the feature map Fm′ is transformed into 2-j Input into the third ConvBlock block and output the feature map Fm 3-j , the ReLU activation function undergoes nonlinear changes. The third DownsamplingConvBlock block of the encoder is composed of a three-dimensional convolutional layer and a ReLU activation function in sequence. 3-j Input to the third DownsamplingConvBlock block, and output the feature map Fm′ with reduced spatial resolution 3-j .

[0077] e-5) The fourth ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function in sequence, and the feature map Fm′ is transformed into 3-j Input to the fourth ConvBlock block, and output the feature map Fm 4-j , the ReLU activation function undergoes nonlinear changes. The fourth DownsamplingConvBlock block of the encoder is composed of a three-dimensional convolutional layer and a ReLU activation function in sequence. 4-j Input to the fourth DownsamplingConvBlock block, and output the feature map Fm′ with reduced spatial resolution 4-j .

[0078] e-6) The fifth ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function in sequence, and the feature map Fm′ is transformed into 4-j Input into the fifth ConvBlock block and output the feature map Fm 5-j , the ReLU activation function performs nonlinear changes.

[0079] In one embodiment of the present invention, step f) comprises the following steps:

[0080] The LinformerPerformerBlock of the LPV-Net branch in f-1) consists of the Linformer attention mechanism, the Performer attention mechanism, and the LayerNorm layer.

[0081] f-2) The feature map Fm 5-j Input to the Linformer attention mechanism of LinformerPerformerBlock, perform low-rank decomposition on the length of the input sequence to reduce complexity, and output the feature map Fm″5″ j f-3) The feature map Fm″ 5-j Input into the Performer attention mechanism of LinformerPerformerBlock to reduce the complexity of self-attention calculation, and output the feature map Fm″′ 5-j .

[0082] f-4) The feature map Fm″′ 5-j Input to the LayerNorm layer of LinformerPerformerBlock to accelerate convergence and output the feature map Fm″′ 5-j .

[0083] In one embodiment of the present invention, step g) comprises the following steps:

[0084] g-1) The decoder of the LPV-Net branch is composed of a first UpsamplingDeconvBlock block, a first ConvBlock block, a second UpsamplingDeconvBlock block, a second ConvBlock block, a third UpsamplingDeconvBlock block, a third ConvBlock block, a fourth UpsamplingDeconvBlock block, and a fourth ConvBlock block.

[0085] g-2) The first UpsamplingDeconvBlock of the decoder is composed of a three-dimensional convolutional layer and a ReLU activation function, and the feature map Fm″′ 5-j Input to the first UpsamplingDeconvBlock block, and output the feature map Fm′ with restored spatial resolution 5-j .

[0086] g-3) The first ConvBlock block of the decoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function. 5-jInput into the first ConvBlock block and output the feature map Fm 6-j , the purpose of the ReLU activation function is to perform linear transformation. The second UpsamplingDeconvBlock block of the decoder is composed of a three-dimensional convolutional layer and a ReLU activation function in sequence. 6-j Input to the second UpsamplingDeconvBlock block, and output the feature map Fm′ with reduced spatial resolution 6-j .

[0087] g-4) The second ConvBlock block of the decoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function. 6-j Input into the second ConvBlock block and output the feature map Fm 7-j , the purpose of the ReLU activation function is to perform linear transformation. The third UpsamplingDeconvBlock block of the decoder is composed of a three-dimensional convolutional layer and a ReLU activation function in sequence. 7-j Input to the third UpsamplingDeconvBlock block, and output the feature map Fm7′ with reduced spatial resolution -j g-5) The third ConvBlock block of the decoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function, which transforms the feature map Fm′ 7-j Input into the third ConvBlock block and output the feature map Fm 8-j , the purpose of the ReLU activation function is to perform linear transformation. The fourth UpsamplingDeconvBlock block of the decoder is composed of a three-dimensional convolutional layer and a ReLU activation function in sequence. 8-j Input to the fourth UpsamplingDeconvBlock block, and output the feature map Fm′ with reduced spatial resolution 8-j .

[0088] g-6) The fourth ConvBlock block of the decoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function, and the feature map Fm′ is transformed into 8-j Input to the fourth ConvBlock block, and output the feature map Fm 9-j , the feature map Fm 9-j Use the tanh function to get the horizontal distance graph The horizontal distance map Input into the softmax function and output the segmentation map In one embodiment of the present invention, step h) comprises the following steps:

[0089] h-1) The 3D-EDA branch of the GLB-Net model consists of the first convolutional layer, the second convolutional layer, the third convolutional layer, the first LeakyReLU activation function, the first dropout layer, the fourth convolutional layer, the second LeakyReLU activation function, the second dropout layer, the fifth convolutional layer, the third LeakyReLU activation function, the third dropout layer, the CARELayer module, the pooling layer, and the fully connected layer.

[0090] h-2) The segmentation map Input into the first convolutional layer of the 3D-EDA branch, and output the feature map Em0. L The jth preprocessed image in The feature map Em1 is output after inputting into the second convolutional layer of the 3D-EDA branch. The feature map Em0 is added to the feature map Em1 to obtain the feature map Em2.

[0091] h-3) Input the feature map Em2 into the third convolutional layer, the first LeakyReLU activation function, and the first dropout layer of the 3D-EDA branch in sequence, and output the feature map Em3.

[0092] h-4) The feature map Em3 is sequentially input into the fourth convolutional layer, the second LeakyReLU activation function, and the second dropout layer of the 3D-EDA branch, and the feature map Em4 is output.

[0093] h-5) The feature map Em4 is sequentially input into the fifth convolutional layer, the third LeakyReLU activation function, and the third dropout layer of the 3D-EDA branch, and the feature map Em5 is output.

[0094] h-6) The CARELLayer module of the 3D-EDA branch is composed of an average pooling layer, a first fully connected layer, a second fully connected layer, and an attention mechanism in sequence. The feature map Em5 is input into the average pooling layer of the CARELLayer module, and the output is a vector z. The vector z is input into the first fully connected layer of the CARELLayer module, and the output is a weight vector z1. The weight vector z1 is input into the second fully connected layer of the CARELLayer module, and the output is a weight vector z2. The weight vector z2 is input into the attention mechanism of the CARELLayer module, and the output is an enhanced feature map Em6.

[0095] h-7) The feature map Em6 is input into the pooling layer and the fully connected layer of the 3D-EDA branch in sequence, and the binary classification result is output.

[0096] In one embodiment of the present invention, in step i), the formula L total =Lsup +L con ×L adv Calculate the total loss L total , where L sup is the unsupervised loss function, L sup =L dice +βL mse , L dice is the dice loss function, β is the adjustment factor, which is used to adjust the weight of segmentation loss and shape loss in the total loss, L mse is the MSE loss function, L con is the consistency loss function, which is used to measure the consistency of the output of the model in different situations. adv is the unsupervised loss function calculated by the binary classification result, which is used for the difference between the output of the beam 3D-EDA and the real data. In this embodiment, when training the GLB-Net model, the initial learning rate is set to 0.01, the batch_size is set to 4, the epoch is set to 16000, and the lr is set to 0.001.

[0097] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A semi-supervised segmentation method for cardiac images based on global-local information fusion LPV-Net and 3D-EDA, characterized in that: include: a) Obtain i labeled or unlabeled cardiac MRI images to form a dataset X, where X = {X l ,X u }, X l is a labeled cardiac MRI image dataset, X u is an unlabeled cardiac MRI image dataset. is the jth labeled cardiac MRI image, j∈{1,...,p}, is the jth unlabeled cardiac MRI image, j∈{1,...,q}, p is the number of labeled cardiac MRI images, q is the number of unlabeled cardiac MRI images, p+q=i; b) Preprocess the data set X to obtain the preprocessed data set X′, X′={X l′ ,X u′ }, X l′ is the preprocessed labeled cardiac MRI image dataset, X u ′ is the preprocessed unlabeled cardiac MRI image dataset; c) Divide the preprocessed data set X′ into a training set X L , test set X U ; d) Establish a GLB-Net model. The GLB-Net model consists of an LPV-Net branch and a 3D-EDA branch. The LPV-Net branch consists of an encoder, a LinformerPerformerBlock, and a decoder. e) The training set X L The image in is input into the encoder of the LPV-Net branch, and the output is the feature map Fm 5-j ; f) The feature map Fm 5-j Input to the LinformerPerformerBlock of the LPV-Net branch, and output the feature map Fm″″ 5-j ; g) The feature map Fm″″ 5-j Input into the decoder of the LPV-Net branch and output the segmentation map h) The prediction graph Input into the 3D-EDA branch to obtain the unsupervised loss function and the supervised loss function; i) Calculate the total loss L based on the unsupervised loss function and the supervised loss function total , using the Adam optimizer with a total loss L total Train the GLB-Net model to obtain the optimized GLB-Net model; j) The test set X U The image in is input into the LPV-Net branch of the optimized GLB-Net model, and the segmentation map is output.

2. The method for semi-supervised segmentation of cardiac images based on global-local information fusion LPV-Net and 3D-EDA according to claim 1, characterized in that: In step a), i labeled or unlabeled left atrium cardiac MRI images are extracted from the 2018 Left Atrium Segmentation Challenge dataset.

3. The method for semi-supervised segmentation of cardiac images based on global-local information fusion LPV-Net and 3D-EDA according to claim 1, characterized in that: Step b) comprises the following steps: b-1) Take the jth labeled cardiac MRI image The area to be segmented is marked as 1, and the rest of the area is marked as 0. Use the np.nonzero function to find the jth labeled cardiac MRI image. The coordinates of all non-zero pixels in the label are calculated by using the np.min function to obtain the minimum value of the coordinates of all non-zero pixels, and the maximum value of the coordinates of all non-zero pixels is calculated by using the np.max function. The minimum rectangular frame containing all cardiac structures is obtained based on the minimum and maximum values ​​of the coordinates. b-2) Use standard Numpy slicing operations on the jth labeled cardiac MRI image according to the minimum rectangular box Perform cropping to obtain the cropped image b-3) The cropped image The image is transformed by the RandomCrop function Image Perform normalization operation to obtain the preprocessed image Preprocessed labeled cardiac MRI image dataset b-4) Take the jth unlabeled cardiac MRI image The image is randomly cropped by RandomCrop to obtain the cropped image b-5) The cropped image The preprocessed image is obtained by randomly rotating 0, 90, 180 or 270 degrees Preprocessed unlabeled cardiac MRI image dataset 4. The method for semi-supervised segmentation of cardiac images based on global-local information fusion LPV-Net and 3D-EDA according to claim 1, characterized in that: From the preprocessed labeled cardiac MRI image data X l ′ and select M preprocessed images and unlabeled cardiac MRI image data X from the preprocessed u ′, select O preprocessed images to obtain the training set X L , From the preprocessed unlabeled cardiac MRI image dataset X u ′, select N preprocessed images to obtain the training set X U , 5. The method for semi-supervised segmentation of cardiac images based on global-local information fusion LPV-Net and 3D-EDA according to claim 4, characterized in that: Step e) comprises the following steps: e-1) The encoder of the LPV-Net branch consists of the first ConvBlock block, the first DownsamplingConvBlock block, the second ConvBlock block, the second DownsamplingConvBlock block, the third ConvBlock block, the fourth DownsamplingConvBlock block, and the fifth ConvBlock block; e-2) The first ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function. L The jth preprocessed image in Input into the first ConvBlock block and output the feature map Fm 1-j The first DownsamplingConvBlock block of the encoder is composed of a three-dimensional convolutional layer and a ReLU activation function. 1-j Input to the first DownsamplingConvBlock block, and output the feature map Fm1′ -j ; e-3) The second ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function, which transforms the feature map Fm1′ -j Input into the second ConvBlock block and output the feature map Fm 2-j The second DownsamplingConvBlock block of the encoder is composed of a three-dimensional convolutional layer and a ReLU activation function. 2-j Input to the second DownsamplingConvBlock block, and output the feature map Fm2′ -j ; e-4) The third ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function in sequence, and the feature map Fm2′ -j Input into the third ConvBlock block and output the feature map Fm 3-j The third DownsamplingConvBlock block of the encoder is composed of a three-dimensional convolutional layer and a ReLU activation function. 3-j Input to the third DownsamplingConvBlock block, and output the feature map Fm3′ -j ; e-5) The fourth ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function in sequence, and the feature map Fm3′ -j Input to the fourth ConvBlock block, and output the feature map Fm 4-j The fourth DownsamplingConvBlock block of the encoder is composed of a three-dimensional convolutional layer and a ReLU activation function, which transforms the feature map Fm 4-j Input to the fourth DownsamplingConvBlock block, and output the feature map Fm′ 4-j ; e-6) The fifth ConvBlock block of the encoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function in sequence, and the feature map Fm′ is transformed into 4-j Input into the fifth ConvBlock block and output the feature map Fm 5-j .

6. The method for semi-supervised segmentation of cardiac images based on global-local information fusion LPV-Net and 3D-EDA according to claim 1, characterized in that: Step f) comprises the following steps: f-1) The LinformerPerformerBlock of the LPV-Net branch consists of the Linformer attention mechanism, the Performer attention mechanism, and the LayerNorm layer; f-2) The feature map Fm 5-j Input into the Linformer attention mechanism of LinformerPerformerBlock, and output the feature map Fm″ 5-j ; f-3) The feature map Fm″ 5-j Input to the Performer attention mechanism of LinformerPerformerBlock, and output the feature map Fm″′ 5-j ; f-4) The feature map Fm″′ 5-j Input to the LayerNorm layer of LinformerPerformerBlock, and output the feature map Fm″″ 5-j .

7. The method for semi-supervised segmentation of cardiac images based on global-local information fusion LPV-Net and 3D-EDA according to claim 1, characterized in that: Step g) comprises the following steps: g-1) The decoder of the LPV-Net branch is composed of a first UpsamplingDeconvBlock block, a first ConvBlock block, a second UpsamplingDeconvBlock block, a second ConvBlock block, a third UpsamplingDeconvBlock block, a third ConvBlock block, a fourth UpsamplingDeconvBlock block, and a fourth ConvBlock block; g-2) The first UpsamplingDeconvBlock of the decoder is composed of a three-dimensional convolutional layer and a ReLU activation function, and the feature map Fm″″ 5-j Input to the first UpsamplingDeconvBlock block, and output the feature map Fm5′ -j ; g-3) The first ConvBlock block of the decoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function in sequence, and the feature map Fm5′ -j Input into the first ConvBlock block and output the feature map Fm 6-j The second UpsamplingDeconvBlock of the decoder is composed of a three-dimensional convolutional layer and a ReLU activation function. 6-j Input to the second UpsamplingDeconvBlock block, and output the feature map Fm′ 6-j ; g-4) The second ConvBlock block of the decoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function. 6-j Input into the second ConvBlock block and output the feature map Fm 7-j The third UpsamplingDeconvBlock of the decoder is composed of a three-dimensional convolutional layer and a ReLU activation function. 7-j Input to the third UpsamplingDeconvBlock block, and output the feature map Fm7′ -j ; g-5) The third ConvBlock block of the decoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function in sequence, and the feature map Fm7′ -j Input into the third ConvBlock block and output the feature map Fm 8-j The fourth UpsamplingDeconvBlock of the decoder is composed of a three-dimensional convolutional layer and a ReLU activation function. 8-j Input to the fourth UpsamplingDeconvBlock block, and output the feature map Fm8′ -j ; g-6) The fourth ConvBlock block of the decoder is composed of a three-dimensional convolutional layer, a BN layer, and a ReLU activation function in sequence, and the feature map Fm8′ -j Input to the fourth ConvBlock block, and output the feature map Fm 9-j , the feature map Fm 9-j Use the tanh function to get the horizontal distance graph The horizontal distance map Input into the softmax function and output the segmentation map 8. The method for semi-supervised segmentation of cardiac images based on global-local information fusion LPV-Net and 3D-EDA according to claim 1, characterized in that: Step h) comprises the following steps: h-1) The 3D-EDA branch of the GLB-Net model consists of the first convolutional layer, the second convolutional layer, the third convolutional layer, the first LeakyReLU activation function, the first dropout layer, the fourth convolutional layer, the second LeakyReLU activation function, the second dropout layer, the fifth convolutional layer, the third LeakyReLU activation function, the third dropout layer, the CARELayer module, the pooling layer, and the fully connected layer; h-2) The segmentation map Input into the first convolutional layer of the 3D-EDA branch, and output the feature map Em0. L The jth preprocessed image in Input into the second convolutional layer of the 3D-EDA branch, output the feature map Em1, add the feature map Em0 and the feature map Em1 to get the feature map Em2; h-3) Input the feature map Em2 into the third convolution layer, the first LeakyReLU activation function, and the first dropout layer of the 3D-EDA branch in sequence, and output the feature map Em3; h-4) The feature map Em3 is sequentially input into the fourth convolutional layer, the second LeakyReLU activation function, and the second dropout layer of the 3D-EDA branch, and the feature map Em4 is output; h-5) The feature map Em4 is sequentially input into the fifth convolutional layer, the third LeakyReLU activation function, and the third dropout layer of the 3D-EDA branch, and the feature map Em5 is output; h-6) The CARELLayer module of the 3D-EDA branch is composed of an average pooling layer, a first fully connected layer, a second fully connected layer, and an attention mechanism in sequence. The feature map Em5 is input into the average pooling layer of the CARELLayer module, and the output is a vector z. The vector z is input into the first fully connected layer of the CARELLayer module, and the output is a weight vector z1. The weight vector z1 is input into the second fully connected layer of the CARELLayer module, and the output is a weight vector z2. The weight vector z2 is input into the attention mechanism of the CARELLayer module, and the output is an enhanced feature map Em6. h-7) The feature map Em6 is input into the pooling layer and the fully connected layer of the 3D-EDA branch in sequence, and the binary classification result is output.

9. The method for semi-supervised segmentation of cardiac images based on global-local information fusion LPV-Net and 3D-EDA according to claim 8, characterized in that: In step i), the formula L total =L sup +L con ×L adv Calculate the total loss L total , where L sup is the unsupervised loss function, L sup =L dice +βL mse , L dice is the dice loss function, β is the adjustment factor, L mse is the MSE loss function, L con is the consistency loss function, L adv is the unsupervised loss function calculated by the binary classification results.

10. The method for semi-supervised segmentation of cardiac images based on global-local information fusion LPV-Net and 3D-EDA according to claim 8, characterized in that: When training the GLB-Net model, the initial learning rate is set to 0.01, batch_size is set to 4, epoch is set to 16000, and lr is set to 0.001.

Images