3D heart image segmentation system based on two visual angles and semi-supervised attention model

Abstract

The invention discloses a 3D heart image segmentation system based on two visual angles and a semi-supervised attention model, and belongs to the technical field of medical image segmentation. The problem that a traditional 3D medical image segmentation method is poor in segmentation performance is solved. According to the method, a data enhancement mechanism is adopted, that is, a data set is expanded by using a mixed region in the image so as to enhance the performance in a 3D cardiac medical image segmentation task, the 3D cardiac image is segmented from two view angles of a coronal plane and a cross section, and complementary segmentation information of the two view angles can be obtained; the segmentation network designed by the invention more pays more attention to segmentation areas and edge parts, and meanwhile, by adopting a semi-supervised learning mode, the image segmentation performance is ensured in combination with complementary segmentation information of two visual angles under the condition that all 3D heart images do not need to be labeled. The method can be applied to segmentation of the 3D heart image.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image segmentation, and in particular relates to a 3D cardiac image segmentation system based on two perspectives and a semi-supervised attention model. Background Art

[0002] Medical image segmentation has become one of the most important tasks in computer vision. Three-dimensional (3D) medical image segmentation has become increasingly important because it can provide more information than two-dimensional medical images. Accurate 3D medical image segmentation can enhance clinical diagnosis and decision-making. In recent years, with the rapid development of deep learning segmentation, good results have been achieved in many scenarios. However, traditional 3D medical image segmentation methods often rely heavily on doctors' experience annotation. And usually only images from one direction are trained, which is not conducive to learning complementary information for 3D medical image segmentation. Semi-supervised learning is a machine learning method that uses both labeled and unlabeled data for training. In semi-supervised learning, only a small part of the dataset is manually labeled, while most of the data is unlabeled. Compared with fully supervised learning, this method can make more efficient use of data, especially when labeled data is difficult to obtain or costly.

[0003] In summary, since traditional 3D medical image segmentation methods only train images from one direction, it is not conducive to learning the complementary information of 3D medical image segmentation. In addition, the collection and annotation of medical image data sets are difficult, and a small amount of data cannot train a more ideal result. Therefore, the segmentation performance of traditional 3D medical image segmentation methods is still poor. It is very necessary to propose a new 3D medical image segmentation method to solve the above problems. Summary of the invention

[0004] The purpose of the present invention is to solve the problem of poor segmentation performance of traditional 3D medical image segmentation methods, and propose a 3D cardiac image segmentation system based on two-view and semi-supervised attention model.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a 3D cardiac image segmentation system based on two perspectives and a semi-supervised attention model, the system includes a 3D cardiac image acquisition module, a 3D cardiac image preprocessing module, an image mixing module and a semi-supervised attention model, wherein:

[0006] The 3D heart image acquisition module is used to acquire a 3D heart image and annotate a segmentation label on a portion of the acquired 3D heart image;

[0007] The 3D cardiac image preprocessing module is used to preprocess the acquired 3D cardiac image to obtain various image patches;

[0008] The image mixing module is used to process the image patches to generate a mixed image;

[0009] The semi-supervised attention model is trained based on labeled 3D heart images, unlabeled 3D heart images, and mixed images;

[0010] The trained semi-supervised attention model is used to segment the 3D heart image to be segmented.

[0011] Furthermore, the 3D heart image acquired by the 3D heart image acquisition module comes from the LA dataset and the CETUS dataset.

[0012] Furthermore, the working process of the 3D cardiac image preprocessing module is as follows:

[0013] After random rotation, random flipping and scaling, the 3D cardiac images were center cropped and randomly cropped. The 3D cardiac images in the LA dataset were preprocessed to an image patch size of 112×112×80, and the 3D cardiac images in the CETUS dataset were preprocessed to an image patch size of 192×192×64.

[0014] Furthermore, the working process of the image mixing module is as follows:

[0015] For any two labeled image patches:

[0016] Randomly select the position and size of the cropping box, and then cut out the image in the cropping box from the two labeled image patches according to the position and size of the cropping box. The images cut out from the two labeled image patches are recorded as x A and x B ;

[0017] The generated fused image for: Where M is the mask matrix;

[0018] According to x A The label and x B Generate mixed images with labels Tags Among them, y A is x A The label, y B is x B The label of , λ is the weight;

[0019] Using the generated fused image Replace the position of the cropped image in the original image patch to obtain a mixed image;

[0020] For any two unlabeled image patches:

[0021] Randomly select the position and size of the cropping box, and then cut out the image in the cropping box from the two labeled image patches according to the position and size of the cropping box. The images cut out from the two labeled image patches are recorded as x′ A and x′ B ;

[0022] The generated fused image for: Where M is the mask matrix;

[0023] Using the generated fused image Replace the position of the cropped image in the original image patch to obtain a mixed image.

[0024] Furthermore, the semi-supervised attention model includes a supervision module and a segmentation network, and the segmentation network includes a first segmentation module and a second segmentation module;

[0025] The supervision module is used to obtain weight maps in two directions, the coronal plane and the cross-sectional plane;

[0026] The first segmentation module is trained based on the labeled 3D cardiac image, the unlabeled 3D cardiac image, the mixed image, and the weight map of the coronal plane direction;

[0027] The second segmentation module is trained based on labeled 3D cardiac images, unlabeled 3D cardiac images, mixed images, and weight maps in the cross-sectional direction.

[0028] Furthermore, the working process of the supervision module is:

[0029] For the coronal direction

[0030] Step 1: Select the middle slice from the coronal plane of the three-dimensional medical image, and record the weight of the middle slice as weight;

[0031] Step 2: For any other slice in the coronal plane direction, calculate the weight of the slice according to the distance d between the slice and the middle slice. d ;

[0032] Step 3: generating a weight map of the three-dimensional medical image in the coronal direction according to the weights of each slice in the coronal direction;

[0033] Similarly, a weight map of the three-dimensional medical image in the cross-sectional direction is generated.

[0034] Furthermore, the first segmentation module includes an encoder part and a decoder part, and the encoder part includes four downsampling units, and the decoder part includes four upsampling units;

[0035] The first downsampling unit includes an adaptive pooling subunit, a first convolutional layer, a CBAM subunit and a second convolutional layer; and the working process of the first downsampling unit is:

[0036] The input of the first segmentation module is used as the input of the first downsampling unit of the encoder part. In the first downsampling unit, the input first passes through the adaptive pooling subunit, the first convolutional layer and the CBAM subunit in sequence, and then the input of the first downsampling unit and the output of the CBAM subunit are element-by-element summed to obtain an element-by-element summation result A. Finally, the element-by-element summation result A passes through the second convolutional layer, and the output of the second convolutional layer is used as the output of the first downsampling unit;

[0037] The second downsampling unit includes a first convolutional layer, a second convolutional layer, a CBAM subunit and a third convolutional layer; and the working process of the second downsampling unit is:

[0038] The output of the first down-sampling unit is used as the input of the second down-sampling unit. In the second down-sampling unit, the input first passes through the first convolution layer, the second convolution layer and the CBAM sub-unit in sequence, and then the input of the second down-sampling unit and the output of the CBAM sub-unit are element-by-element summed to obtain an element-by-element summation result B. Finally, the element-by-element summation result B passes through the third convolution layer, and the output of the third convolution layer is used as the output of the second down-sampling unit;

[0039] The third down-sampling unit includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a CBAM subunit and a fourth convolutional layer; and the working process of the third down-sampling unit is:

[0040] The output of the second downsampling unit is used as the input of the third downsampling unit. In the third downsampling unit, the input first passes through the first convolution layer, the second convolution layer, the third convolution layer and the CBAM subunit in sequence, and then the input of the third downsampling unit and the output of the CBAM subunit are element-by-element summed to obtain an element-by-element summation result C. Finally, the element-by-element summation result C passes through the fourth convolution layer, and the output of the fourth convolution layer is used as the output of the third downsampling unit;

[0041] The fourth down-sampling unit includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a CBAM subunit and a fourth convolutional layer; and the working process of the fourth down-sampling unit is:

[0042] The output of the third down-sampling unit is used as the input of the fourth down-sampling unit. In the fourth down-sampling unit, the input first passes through the first convolution layer, the second convolution layer, the third convolution layer and the CBAM sub-unit in sequence, and then the input of the fourth down-sampling unit and the output of the CBAM sub-unit are element-by-element summed to obtain an element-by-element summation result D. Finally, the element-by-element summation result D passes through the fourth convolution layer, and the output of the fourth convolution layer is used as the output of the fourth down-sampling unit;

[0043] The element-by-element summation result D is then passed through three convolutional layers in sequence to obtain the output E, and then E and D are element-by-element summed to obtain F, which is then passed through a deconvolution layer to obtain the output F' of the deconvolution layer;

[0044] The first upsampling unit includes a first convolution layer, a second convolution layer, a third convolution layer and a deconvolution layer; and the working process of the first upsampling unit is:

[0045] After feature connection is performed on the element-by-element summation result D and the output F' of the deconvolution layer, the feature connection result is recorded as D'. In the first upsampling unit, the feature connection result D' passes through the first convolution layer, the second convolution layer and the third convolution layer in sequence, and then the output of the third convolution layer and F' are element-by-element summed, and then the element-by-element summation result passes through the deconvolution layer, and the output of the deconvolution layer is used as the output of the first upsampling unit;

[0046] The second upsampling unit includes a first convolution layer, a second convolution layer, a third convolution layer and a deconvolution layer; and the working process of the second upsampling unit is:

[0047] After feature connection is performed on the element-by-element summation result C and the output of the first upsampling unit, the feature connection result is sequentially passed through the first convolution layer, the second convolution layer, and the third convolution layer, and then the output of the third convolution layer is element-wise summed with the output of the first upsampling unit, and then the element-by-element summation result is passed through the deconvolution layer, and the output of the deconvolution layer is used as the output of the second upsampling unit;

[0048] The third upsampling unit includes a first convolution layer, a second convolution layer and a deconvolution layer; and the working process of the third upsampling unit is:

[0049] After feature connection is performed on the element-by-element summation result B and the output of the second upsampling unit, the feature connection result is sequentially passed through the first convolution layer and the second convolution layer, and then the output of the second convolution layer and the output of the second upsampling unit are element-wise summed, and then the element-by-element summation result is passed through the deconvolution layer, and the output of the deconvolution layer is used as the output of the third upsampling unit;

[0050] The fourth upsampling unit includes the first convolutional layer; and the working process of the fourth upsampling unit is:

[0051] After feature connection is performed on the element-by-element summation result A and the output of the third upsampling unit, the feature connection result passes through the first convolution layer, and then the output of the first convolution layer and the output of the third upsampling unit are element-by-element summed, and the element-by-element sum is used as the output of the fourth upsampling unit, that is, the output of the first segmentation module is obtained;

[0052] The network structure of the second segmentation module is the same as that of the first segmentation module.

[0053] Furthermore, the working process of the adaptive pooling subunit is:

[0054] The input of the adaptive pooling subunit is recorded as x, and x is passed through the adaptive average pooling layer, the sigmoid activation function layer, and the expansion layer in sequence. The output of the expansion layer is multiplied element-by-element by x, and the element-by-element multiplication result is used as the output of the adaptive pooling subunit;

[0055] The working process of the CBAM subunit is:

[0056] The input of the CBAM sub-unit is denoted as y. First, y passes through the convolution layer, and the output of the convolution layer is denoted as y'. y' passes through the channel attention layer, and then the output of the channel attention layer is fused with y'. The fusion result passes through the convolution layer again to obtain the output y" of the convolution layer. y" passes through the spatial attention layer, and the output of the spatial attention layer is fused with y", and the fusion result is used as the output of the CBAM sub-unit.

[0057] Furthermore, the first segmentation module and the second segmentation module are jointly trained; the specific process of the joint training is:

[0058] Step 1: Initialize training data batch t=1;

[0059] Step 2: Select the tth batch of training data from the coronal data set;

[0060] Step 3: Use the training data selected in step 2 to train the first segmentation module. The loss function L used in training is total for:

[0061] L total =L sup +L semi +λ·L uncertainty

[0062] Among them, L sup is the segmentation loss of the labeled image dataset, L semi is the segmentation loss of the unlabeled image dataset, L uncertainty is the uncertainty correction loss, λ is the weight coefficient;

[0063] Segmentation loss L sup and L semi Both include cross loss L ce and Dice loss; among them:

[0064] For unlabeled images, the cross loss L ce The calculation method is:

[0065]

[0066] Among them, w i is the i-th voxel of the weight map W in the coronal plane, H, W and D are the length, width and height of the image, p i represents the probability that the i-th voxel belongs to the foreground, y i represents the pseudo label of the i-th voxel;

[0067] For unlabeled images, the Dice loss is calculated as:

[0068]

[0069] Similarly, calculate the cross loss and Dice loss of labeled images;

[0070] Uncertainty Correction Loss L uncertainty for:

[0071]

[0072] in, is the predicted value output by the ith voxel after the sth upsampling unit, is the average value of the predicted value output by all S upsampling units for the i-th voxel, represents the correction value of the i-th voxel after the s-th upsampling unit, express The uncertainty of ||·|| 2 represents the 2-norm;

[0073]

[0074] in, yes The value at the jth channel, yes The value at the jth channel, C is the number of channels;

[0075] Step 4: Using the average exponential parameter transfer method, the parameters of the first segmentation module trained in step 3 are transferred to the second segmentation module, and the second segmentation module updates its own parameters according to the received parameters;

[0076] Step 5: Select the tth batch of training data from the cross-sectional data set;

[0077] Step 6: Use the training data selected in step 5 to train the second segmentation module after parameter update;

[0078] Step 7: Using the average exponential parameter transfer method, the parameters of the second segmentation module trained in step 6 are transferred to the first segmentation module, and the first segmentation module updates its own parameters according to the received parameters;

[0079] Step 8: Set the batch number t=t+1, and then return to step 2 until the training is completed.

[0080] Furthermore, the trained semi-supervised attention model is used to segment the 3D heart image to be segmented, specifically:

[0081] The 3D heart image to be segmented is used as the input of the trained first segmentation module and the second segmentation module respectively, to obtain the segmentation result output by the first segmentation module and the segmentation result output by the second segmentation module;

[0082] Then, the best segmentation result is selected from the segmentation results output by the first segmentation module and the segmentation results output by the second segmentation module according to the evaluation index.

[0083] The beneficial effects of the present invention are:

[0084] The present invention adopts a data enhancement mechanism, that is, using the mixed area in the image to expand the data set to enhance the performance in the 3D cardiac medical image segmentation task. The 3D cardiac image is segmented from the coronal and cross-sectional perspectives to obtain complementary segmentation information of the two perspectives. The segmentation network designed by the present invention pays more attention to the segmentation area and the edge part. At the same time, by adopting a semi-supervised learning method, the image segmentation performance is guaranteed by combining the complementary segmentation information of the two perspectives without the need to annotate the entire 3D cardiac image. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 It is a structural schematic diagram of a 3D cardiac image segmentation system based on two-view and semi-supervised attention model of the present invention;

[0086] Figure 2 It is the structural diagram of the segmentation module;

[0087] Adaptive Channel Attention Block represents an adaptive pooling subunit, input represents input, output represents output, Adaptive average pooling represents an adaptive average pooling layer, Element-wiseMultiplication represents element-wise multiplication, Channel attention represents a channel attention layer, and Spatitalattention represents a spatial attention layer. DETAILED DESCRIPTION

[0088] Specific implementation method 1: Combination Figure 1 The present embodiment is described in a 3D cardiac image segmentation system based on two perspectives and a semi-supervised attention model, the system comprising a 3D cardiac image acquisition module, a 3D cardiac image preprocessing module, an image mixing module and a semi-supervised attention model, wherein:

[0089] The 3D cardiac image acquisition module is used to acquire 3D cardiac images and annotate the acquired part of the 3D cardiac images with segmentation labels (only a small part of the images need to be labeled, and other images do not need to be labeled);

[0090] The 3D cardiac image preprocessing module is used to preprocess the acquired 3D cardiac image to obtain various image patches;

[0091] The image mixing module is used to process the image patches to generate a mixed image;

[0092] The semi-supervised attention model is trained based on labeled 3D heart images, unlabeled 3D heart images and mixed images (the training stop condition is to reach a preset number of training times);

[0093] The trained semi-supervised attention model is used to segment the 3D heart image to be segmented.

[0094] After partially annotating the obtained three-dimensional cardiac image dataset, the present invention divides the dataset into a training set and a test set in a ratio of 0.8:0.2; and uses the training set and the test set to train and test the semi-supervised attention network.

[0095] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that the 3D heart image acquired by the 3D heart image acquisition module comes from the LA dataset and the CETUS dataset.

[0096] The other steps and parameters are the same as those in the first embodiment.

[0097] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that the working process of the 3D cardiac image preprocessing module is as follows:

[0098] After random rotation, random flipping and scaling, the 3D cardiac images were center cropped and randomly cropped. The 3D cardiac images in the LA dataset were preprocessed to an image patch size of 112×112×80, and the 3D cardiac images in the CETUS dataset were preprocessed to an image patch size of 192×192×64.

[0099] The other steps and parameters are the same as those in the first or second embodiment.

[0100] Specific implementation method 4: This implementation method is different from any one of the specific implementation methods 1 to 3 in that the working process of the image mixing module is as follows:

[0101] For any two labeled image patches:

[0102] Randomly select the position and size of the cropping box, and then cut out the image in the cropping box from the two labeled image patches according to the position and size of the cropping box. The images cut out from the two labeled image patches are recorded as x A and x B ;

[0103] The generated fused image for: Where M is the mask matrix;

[0104] According to x A The label and x B Generate mixed images with labels Tags Among them, y A is x A The label, y B is x B The label, λ is the weight, the value is 0.5;

[0105] Using the generated fused image Replace the position of the cropped image in the original image patch to obtain a mixed image;

[0106] For any two unlabeled image patches:

[0107] Randomly select the position and size of the cropping box, and then cut out the image in the cropping box from the two labeled image patches according to the position and size of the cropping box. The images cut out from the two labeled image patches are recorded as x′ A and x′ B ;

[0108] The generated fused image for: Where M is the mask matrix;

[0109] Using the generated fused image Replace the position of the cropped image in the original image patch to obtain a mixed image.

[0110] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0111] Specific implementation five: This implementation differs from any one of specific implementations one to four in that the semi-supervised attention model includes a supervision module and a segmentation network, and the segmentation network includes a first segmentation module and a second segmentation module;

[0112] The supervision module is used to obtain weight maps in two directions, the coronal plane and the cross-sectional plane;

[0113] The first segmentation module is trained based on the labeled 3D cardiac images, the unlabeled 3D cardiac images, the mixed images, and the weight map in the coronal plane direction;

[0114] The second segmentation module is trained based on labeled 3D cardiac images, unlabeled 3D cardiac images, mixed images, and weight maps in the cross-sectional direction.

[0115] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0116] Specific implementation method 6: This implementation method is different from the specific implementation methods 1 to 5 in that the working process of the supervision module is:

[0117] For the coronal direction

[0118] Step 1: Select the middle slice from the coronal plane of the three-dimensional medical image, and record the weight of the middle slice as weight;

[0119] Step 2: For any other slice in the coronal plane direction, calculate the weight of the slice according to the distance d between the slice and the middle slice. d ;

[0120] Step 3: generating a weight map of the three-dimensional medical image in the coronal direction according to the weights of each slice in the coronal direction;

[0121] Similarly, a weight map of the three-dimensional medical image in the cross-sectional direction is generated.

[0122] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0123] Specific implementation method seven: Combination Figure 2 This embodiment is described. This embodiment is different from any one of the specific embodiments 1 to 6 in that the first segmentation module includes an encoder part and a decoder part, and the encoder part includes four downsampling units, and the decoder part includes four upsampling units;

[0124] The first downsampling unit includes an adaptive pooling subunit, a first convolutional layer (the convolution kernel size is 5×5×5), a CBAM subunit, and a second convolutional layer (the convolution kernel size is 2×2×2); and the working process of the first downsampling unit is:

[0125] The input of the first segmentation module is used as the input of the first downsampling unit of the encoder part. In the first downsampling unit, the input first passes through the adaptive pooling subunit, the first convolutional layer and the CBAM subunit in sequence, and then the input of the first downsampling unit and the output of the CBAM subunit are element-by-element summed to obtain an element-by-element summation result A. Finally, the element-by-element summation result A passes through the second convolutional layer, and the output of the second convolutional layer is used as the output of the first downsampling unit;

[0126] The second downsampling unit includes the first convolution layer (the convolution kernel size is 5×5×5), the second convolution layer (the convolution kernel size is 5×5×5), the CBAM subunit and the third convolution layer (the convolution kernel size is 2×2×2); and the working process of the second downsampling unit is:

[0127] The output of the first down-sampling unit is used as the input of the second down-sampling unit. In the second down-sampling unit, the input first passes through the first convolution layer, the second convolution layer and the CBAM sub-unit in sequence, and then the input of the second down-sampling unit and the output of the CBAM sub-unit are element-by-element summed to obtain an element-by-element summation result B. Finally, the element-by-element summation result B passes through the third convolution layer, and the output of the third convolution layer is used as the output of the second down-sampling unit;

[0128] The third downsampling unit includes the first convolution layer (the convolution kernel size is 5×5×5), the second convolution layer (the convolution kernel size is 5×5×5), the third convolution layer (the convolution kernel size is 5×5×5), the CBAM subunit and the fourth convolution layer (the convolution kernel size is 2×2×2); and the working process of the third downsampling unit is:

[0129] The output of the second downsampling unit is used as the input of the third downsampling unit. In the third downsampling unit, the input first passes through the first convolution layer, the second convolution layer, the third convolution layer and the CBAM subunit in sequence, and then the input of the third downsampling unit and the output of the CBAM subunit are element-by-element summed to obtain an element-by-element summation result C. Finally, the element-by-element summation result C passes through the fourth convolution layer, and the output of the fourth convolution layer is used as the output of the third downsampling unit;

[0130] The fourth downsampling unit includes the first convolution layer (the convolution kernel size is 5×5×5), the second convolution layer (the convolution kernel size is 5×5×5), the third convolution layer (the convolution kernel size is 5×5×5), the CBAM subunit and the fourth convolution layer (the convolution kernel size is 2×2×2); and the working process of the fourth downsampling unit is:

[0131] The output of the third down-sampling unit is used as the input of the fourth down-sampling unit. In the fourth down-sampling unit, the input first passes through the first convolution layer, the second convolution layer, the third convolution layer and the CBAM sub-unit in sequence, and then the input of the fourth down-sampling unit and the output of the CBAM sub-unit are element-by-element summed to obtain an element-by-element summation result D. Finally, the element-by-element summation result D passes through the fourth convolution layer, and the output of the fourth convolution layer is used as the output of the fourth down-sampling unit;

[0132] The element-by-element summation result D is then passed through three convolutional layers (the convolution kernel size is 5×5×5) to obtain the output E, and then E and D are element-by-element summed to obtain F, which is then passed through a deconvolution layer to obtain the output F' of the deconvolution layer;

[0133] The first upsampling unit includes the first convolution layer (the convolution kernel size is 5×5×5), the second convolution layer (the convolution kernel size is 2×2×2), the third convolution layer (the convolution kernel size is 5×5×5) and the deconvolution layer; and the working process of the first upsampling unit is:

[0134] After feature connection is performed on the element-by-element summation result D and the output F' of the deconvolution layer, the feature connection result is recorded as D'. In the first upsampling unit, the feature connection result D' passes through the first convolution layer, the second convolution layer and the third convolution layer in sequence, and then the output of the third convolution layer and F' are element-by-element summed, and then the element-by-element summation result passes through the deconvolution layer, and the output of the deconvolution layer is used as the output of the first upsampling unit;

[0135] The second upsampling unit includes the first convolution layer (the convolution kernel size is 5×5×5), the second convolution layer (the convolution kernel size is 2×2×2), the third convolution layer (the convolution kernel size is 5×5×5) and the deconvolution layer; and the working process of the second upsampling unit is:

[0136] After feature connection is performed on the element-by-element summation result C and the output of the first upsampling unit, the feature connection result is sequentially passed through the first convolution layer, the second convolution layer, and the third convolution layer, and then the output of the third convolution layer is element-wise summed with the output of the first upsampling unit, and then the element-by-element summation result is passed through the deconvolution layer, and the output of the deconvolution layer is used as the output of the second upsampling unit;

[0137] The third upsampling unit includes a first convolution layer (the convolution kernel size is 5×5×5), a second convolution layer (the convolution kernel size is 5×5×5) and a deconvolution layer; and the working process of the third upsampling unit is:

[0138] After feature connection is performed on the element-by-element summation result B and the output of the second upsampling unit, the feature connection result is sequentially passed through the first convolution layer and the second convolution layer, and then the output of the second convolution layer and the output of the second upsampling unit are element-wise summed, and then the element-by-element summation result is passed through the deconvolution layer, and the output of the deconvolution layer is used as the output of the third upsampling unit;

[0139] The fourth upsampling unit includes the first convolution layer (the convolution kernel size is 5×5×5); and the working process of the fourth upsampling unit is:

[0140] After feature connection is performed on the element-by-element summation result A and the output of the third upsampling unit, the feature connection result passes through the first convolution layer, and then the output of the first convolution layer and the output of the third upsampling unit are element-by-element summed, and the element-by-element sum is used as the output of the fourth upsampling unit, that is, the output of the first segmentation module is obtained;

[0141] The network structure of the second segmentation module is the same as that of the first segmentation module.

[0142] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0143] Specific implementation eight: This implementation differs from any one of specific implementations one to seven in that the working process of the adaptive pooling subunit is as follows:

[0144] The input of the adaptive pooling subunit is recorded as x, and x is passed through the adaptive average pooling layer, the sigmoid activation function layer, and the expansion layer (expand) in sequence. The output of the expansion layer is multiplied element-by-element by x, and the result of the element-by-element multiplication is used as the output of the adaptive pooling subunit;

[0145] The working process of the CBAM subunit is:

[0146] The input of the CBAM sub-unit is denoted as y. First, y passes through the convolution layer, and the output of the convolution layer is denoted as y'. y' passes through the channel attention layer, and then the output of the channel attention layer is fused with y'. The fusion result passes through the convolution layer again to obtain the output y" of the convolution layer. y" passes through the spatial attention layer, and the output of the spatial attention layer is fused with y", and the fusion result is used as the output of the CBAM sub-unit.

[0147] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0148] Specific implementation method 9: This implementation method is different from any one of specific implementation methods 1 to 8 in that the first segmentation module and the second segmentation module are jointly trained; the specific process of the joint training is:

[0149] Step 1: Initialize training data batch t=1;

[0150] Step 2: Select the tth batch of training data from the coronal data set;

[0151] Step 3: Use the training data selected in step 2 to train the first segmentation module. The loss function L used in training is total for:

[0152] L total =L sup +L semi +λ·L uncertainty

[0153] Among them, L sup is the segmentation loss of the labeled image dataset, L semi is the segmentation loss of the unlabeled image dataset, L uncertainty is the uncertainty correction loss, λ is the weight coefficient (set by the Gaussian warm-up function, usually set to 0.5);

[0154] Segmentation loss L sup and L semi Both include cross loss L ce and Dice loss; among them:

[0155] For unlabeled images, the cross loss L ce The calculation method is:

[0156]

[0157] Among them, w i is the i-th voxel of the weight map W in the coronal plane, H, W and D are the length, width and height of the image, p i represents the probability that the i-th voxel belongs to the foreground, y i represents the pseudo label of the i-th voxel;

[0158] For unlabeled images, the Dice loss is calculated as:

[0159]

[0160] Similarly, calculate the cross loss and Dice loss of labeled images;

[0161] Uncertainty Correction Loss L uncertainty for:

[0162]

[0163] in, is the predicted value output by the ith voxel after the sth upsampling unit, is the average value (also called uncertainty value) of the predicted value output by all S upsampling units for the i-th voxel, represents the correction value of the i-th voxel after the s-th upsampling unit, express The uncertainty of ||·|| 2 represents the 2-norm;

[0164]

[0165] in, yes The value at the jth channel, yes The value at the jth channel, C is the number of channels;

[0166] Step 4: Use the average exponential parameter transfer method to pass the parameters of the first segmentation module trained in step 3 to the second segmentation module, and the second segmentation module updates its own parameters according to the received parameters (the update method is the same as that of the teacher model and the student model);

[0167] Step 5: Select the tth batch of training data from the cross-sectional data set;

[0168] Step 6: Use the training data selected in step 5 to train the second segmentation module after parameter update;

[0169] The loss function calculation method for the second segmentation module during training is the same as that for the first segmentation module;

[0170] Step 7: Using the average exponential parameter transfer method, the parameters of the second segmentation module trained in step 6 are transferred to the first segmentation module, and the first segmentation module updates its own parameters according to the received parameters;

[0171] Step 8: Set the batch number t=t+1, and then return to step 2 until the training is completed.

[0172] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0173] Specific embodiment 10: This embodiment is different from any one of specific embodiments 1 to 9 in that the trained semi-supervised attention model is used to segment the 3D heart image to be segmented, specifically:

[0174] The 3D heart image to be segmented is used as the input of the trained first segmentation module and the second segmentation module respectively, to obtain the segmentation result output by the first segmentation module and the segmentation result output by the second segmentation module;

[0175] Then, the best segmentation result is selected from the segmentation results output by the first segmentation module and the segmentation results output by the second segmentation module according to the evaluation index.

[0176] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0177] In this implementation, the segmentation result output by the first segmentation module is compared with the segmentation result output by the second segmentation module according to the evaluation indicators Dice, Jaccard, 95HD and ASD, and a better segmentation result is selected.

[0178] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A 3D cardiac image segmentation system based on two-view and semi-supervised attention model, characterized in that: The system includes a 3D cardiac image acquisition module, a 3D cardiac image preprocessing module, an image mixing module and a semi-supervised attention model, wherein: The 3D heart image acquisition module is used to acquire a 3D heart image and annotate a segmentation label on a portion of the acquired 3D heart image; The 3D cardiac image preprocessing module is used to preprocess the acquired 3D cardiac image to obtain various image patches; The image mixing module is used to process the image patches to generate a mixed image; The semi-supervised attention model is trained based on labeled 3D heart images, unlabeled 3D heart images, and mixed images; The trained semi-supervised attention model is used to segment the 3D heart image to be segmented.

2. A 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 1, characterized in that: The 3D heart image acquired by the 3D heart image acquisition module comes from the LA dataset and the CETUS dataset.

3. The 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 2, characterized in that: The working process of the 3D cardiac image preprocessing module is as follows: After random rotation, random flipping and scaling, the 3D cardiac images were center cropped and randomly cropped. The 3D cardiac images in the LA dataset were preprocessed to an image patch size of 112×112×80, and the 3D cardiac images in the CETUS dataset were preprocessed to an image patch size of 192×192×64.

4. The 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 1, characterized in that: The working process of the image mixing module is as follows: For any two labeled image patches: Randomly select the position and size of the cropping box, and then cut out the image in the cropping box from the two labeled image patches according to the position and size of the cropping box. The images cut out from the two labeled image patches are recorded as x A and x B ; The generated fused image for: Where M is the mask matrix; According to x A The label and x B Generate mixed images with labels Tags Among them, y A is x A The label, y B is x B The label of , λ is the weight; Using the generated fused image Replace the position of the cropped image in the original image patch to obtain a mixed image; For any two unlabeled image patches: Randomly select the position and size of the cropping box, and then cut out the image in the cropping box from the two labeled image patches according to the position and size of the cropping box. The images cut out from the two labeled image patches are recorded as x′ A and x′ B ; The generated fused image for: Where M is the mask matrix; Using the generated fused image Replace the position of the cropped image in the original image patch to obtain a mixed image.

5. A 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 4, characterized in that: The semi-supervised attention model includes a supervision module and a segmentation network, and the segmentation network includes a first segmentation module and a second segmentation module; The supervision module is used to obtain weight maps in two directions, the coronal plane and the cross-sectional plane; The first segmentation module is trained based on the labeled 3D cardiac images, the unlabeled 3D cardiac images, the mixed images, and the weight map in the coronal plane direction; The second segmentation module is trained based on labeled 3D cardiac images, unlabeled 3D cardiac images, mixed images, and weight maps in the cross-sectional direction.

6. A 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 5, characterized in that: The working process of the supervision module is as follows: For the coronal direction Step 1: Select the middle slice from the coronal plane of the three-dimensional medical image, and record the weight of the middle slice as weight; Step 2: For any other slice in the coronal plane direction, calculate the weight of the slice according to the distance d between the slice and the middle slice. d ; Step 3: generating a weight map of the three-dimensional medical image in the coronal direction according to the weights of each slice in the coronal direction; Similarly, a weight map of the three-dimensional medical image in the cross-sectional direction is generated.

7. The 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 6, characterized in that: The first segmentation module includes an encoder part and a decoder part, wherein the encoder part includes four down-sampling units, and the decoder part includes four up-sampling units; The first downsampling unit includes an adaptive pooling subunit, a first convolutional layer, a CBAM subunit and a second convolutional layer; and the working process of the first downsampling unit is: The input of the first segmentation module is used as the input of the first downsampling unit of the encoder part. In the first downsampling unit, the input first passes through the adaptive pooling subunit, the first convolutional layer and the CBAM subunit in sequence, and then the input of the first downsampling unit and the output of the CBAM subunit are element-by-element summed to obtain an element-by-element summation result A. Finally, the element-by-element summation result A passes through the second convolutional layer, and the output of the second convolutional layer is used as the output of the first downsampling unit; The second downsampling unit includes a first convolutional layer, a second convolutional layer, a CBAM subunit and a third convolutional layer; and the working process of the second downsampling unit is: The output of the first down-sampling unit is used as the input of the second down-sampling unit. In the second down-sampling unit, the input first passes through the first convolution layer, the second convolution layer and the CBAM sub-unit in sequence, and then the input of the second down-sampling unit and the output of the CBAM sub-unit are element-by-element summed to obtain an element-by-element summation result B. Finally, the element-by-element summation result B passes through the third convolution layer, and the output of the third convolution layer is used as the output of the second down-sampling unit; The third down-sampling unit includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a CBAM subunit and a fourth convolutional layer; and the working process of the third down-sampling unit is: The output of the second downsampling unit is used as the input of the third downsampling unit. In the third downsampling unit, the input first passes through the first convolution layer, the second convolution layer, the third convolution layer and the CBAM subunit in sequence, and then the input of the third downsampling unit and the output of the CBAM subunit are element-by-element summed to obtain an element-by-element summation result C. Finally, the element-by-element summation result C passes through the fourth convolution layer, and the output of the fourth convolution layer is used as the output of the third downsampling unit; The fourth down-sampling unit includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a CBAM subunit and a fourth convolutional layer; and the working process of the fourth down-sampling unit is: The output of the third down-sampling unit is used as the input of the fourth down-sampling unit. In the fourth down-sampling unit, the input first passes through the first convolution layer, the second convolution layer, the third convolution layer and the CBAM sub-unit in sequence, and then the input of the fourth down-sampling unit and the output of the CBAM sub-unit are element-by-element summed to obtain an element-by-element summation result D. Finally, the element-by-element summation result D passes through the fourth convolution layer, and the output of the fourth convolution layer is used as the output of the fourth down-sampling unit; The element-by-element summation result D is then passed through three convolutional layers in sequence to obtain the output E, and then E and D are element-by-element summed to obtain F, which is then passed through a deconvolution layer to obtain the output F' of the deconvolution layer; The first upsampling unit includes a first convolution layer, a second convolution layer, a third convolution layer and a deconvolution layer; and the working process of the first upsampling unit is: After feature connection is performed on the element-by-element summation result D and the output F' of the deconvolution layer, the feature connection result is recorded as D'. In the first upsampling unit, the feature connection result D' passes through the first convolution layer, the second convolution layer and the third convolution layer in sequence, and then the output of the third convolution layer and F' are element-by-element summed, and then the element-by-element summation result passes through the deconvolution layer, and the output of the deconvolution layer is used as the output of the first upsampling unit; The second upsampling unit includes a first convolution layer, a second convolution layer, a third convolution layer and a deconvolution layer; and the working process of the second upsampling unit is: After feature connection is performed on the element-by-element summation result C and the output of the first upsampling unit, the feature connection result is sequentially passed through the first convolution layer, the second convolution layer, and the third convolution layer, and then the output of the third convolution layer is element-wise summed with the output of the first upsampling unit, and then the element-by-element summation result is passed through the deconvolution layer, and the output of the deconvolution layer is used as the output of the second upsampling unit; The third upsampling unit includes a first convolution layer, a second convolution layer and a deconvolution layer; and the working process of the third upsampling unit is: After feature connection is performed on the element-by-element summation result B and the output of the second upsampling unit, the feature connection result is sequentially passed through the first convolution layer and the second convolution layer, and then the output of the second convolution layer and the output of the second upsampling unit are element-wise summed, and then the element-by-element summation result is passed through the deconvolution layer, and the output of the deconvolution layer is used as the output of the third upsampling unit; The fourth upsampling unit includes the first convolutional layer; and the working process of the fourth upsampling unit is: After feature connection is performed on the element-by-element summation result A and the output of the third upsampling unit, the feature connection result passes through the first convolution layer, and then the output of the first convolution layer and the output of the third upsampling unit are element-by-element summed, and the element-by-element sum is used as the output of the fourth upsampling unit, that is, the output of the first segmentation module is obtained; The network structure of the second segmentation module is the same as that of the first segmentation module.

8. The 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 7, characterized in that: The working process of the adaptive pooling subunit is as follows: The input of the adaptive pooling subunit is recorded as x, and x is passed through the adaptive average pooling layer, the sigmoid activation function layer, and the expansion layer in sequence. The output of the expansion layer is multiplied element-by-element by x, and the element-by-element multiplication result is used as the output of the adaptive pooling subunit; The working process of the CBAM subunit is: The input of the CBAM sub-unit is denoted as y. First, y passes through the convolution layer, and the output of the convolution layer is denoted as y'. y' passes through the channel attention layer, and then the output of the channel attention layer is fused with y'. The fusion result passes through the convolution layer again to obtain the output y" of the convolution layer. y" passes through the spatial attention layer, and the output of the spatial attention layer is fused with y", and the fusion result is used as the output of the CBAM sub-unit.

9. The 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 8, characterized in that: The first segmentation module and the second segmentation module are jointly trained; the specific process of the joint training is: Step 1: Initialize training data batch t=1; Step 2: Select the tth batch of training data from the coronal data set; Step 3: Use the training data selected in step 2 to train the first segmentation module. The loss function L used in training is total for: L total =L sup +L semi +λ·L uncertainty Among them, L sup is the segmentation loss of the labeled image dataset, L semi is the segmentation loss of the unlabeled image dataset, L uncertainty is the uncertainty correction loss, λ is the weight coefficient; Segmentation loss L sup and L semi Both include cross loss L ce and Dice loss; among them: For unlabeled images, the cross loss l ce The calculation method is: Among them, w i is the i-th voxel of the weight map W in the coronal plane, H, W and D are the length, width and height of the image, p i represents the probability that the i-th voxel belongs to the foreground, y i represents the pseudo label of the i-th voxel; For unlabeled images, the Dice loss is calculated as: Similarly, calculate the cross loss and Dice loss of labeled images; Uncertainty Correction Loss L uncertainty for: in, is the predicted value output by the ith voxel after the sth upsampling unit, is the average value of the predicted value output by all S upsampling units for the i-th voxel, represents the correction value of the i-th voxel after the s-th upsampling unit, express The uncertainty of ||·|| 2 represents the 2-norm; in, yes The value at the jth channel, yes The value at the jth channel, C is the number of channels; Step 4: Using the average exponential parameter transfer method, the parameters of the first segmentation module trained in step 3 are transferred to the second segmentation module, and the second segmentation module updates its own parameters according to the received parameters; Step 5: Select the tth batch of training data from the cross-sectional data set; Step 6: Use the training data selected in step 5 to train the second segmentation module after parameter update; Step 7: Using the average exponential parameter transfer method, the parameters of the second segmentation module trained in step 6 are transferred to the first segmentation module, and the first segmentation module updates its own parameters according to the received parameters; Step 8: Set the batch number t=t+1, and then return to step 2 until the training is completed.

10. A 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 9, characterized in that: The trained semi-supervised attention model is used to segment the 3D heart image to be segmented, specifically: The 3D heart image to be segmented is used as the input of the trained first segmentation module and the second segmentation module respectively, to obtain the segmentation result output by the first segmentation module and the segmentation result output by the second segmentation module; Then, the best segmentation result is selected from the segmentation results output by the first segmentation module and the segmentation results output by the second segmentation module according to the evaluation index.

Images