A 3D cardiac image segmentation system based on two-view and semi-supervised attention model

Abstract

A 3D cardiac image segmentation system based on two perspectives and a semi-supervised attention model belongs to the field of medical image segmentation technology. The present invention solves the problem of poor segmentation performance of traditional 3D medical image segmentation methods. The present invention adopts a data enhancement mechanism, that is, using the mixed area in the image to expand the data set to enhance the performance in the 3D cardiac medical image segmentation task. By segmenting the 3D cardiac image from the coronal and cross-sectional perspectives, the complementary segmentation information of the two perspectives can be obtained. Moreover, the segmentation network designed by the present invention pays more attention to the segmentation area and the edge part. At the same time, by adopting a semi-supervised learning method, the image segmentation performance is guaranteed by combining the complementary segmentation information of the two perspectives without the need to label the entire 3D cardiac image. The method of the present invention can be applied to the segmentation of 3D cardiac images.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image segmentation, and in particular relates to a 3D cardiac image segmentation system based on two perspectives and a semi-supervised attention model. Background Art

[0002] Medical image segmentation has become a crucial task in computer vision. Three-dimensional (3D) medical image segmentation has become increasingly important because it provides more information than two-dimensional medical images. Accurate 3D medical image segmentation can enhance clinical diagnosis and decision-making. In recent years, deep learning has made rapid progress in segmentation, achieving excellent results in many scenarios. However, traditional 3D medical image segmentation methods often rely heavily on physicians' empirical annotations. Furthermore, they are typically trained on images from only one orientation, which is not conducive to learning complementary information for 3D medical image segmentation. Semi-supervised learning is a machine learning method that utilizes both labeled and unlabeled data for training. In semi-supervised learning, only a small portion of the dataset is manually labeled, while the majority of the data is unlabeled. Compared to fully supervised learning, this method can more efficiently utilize data, especially when labeled data is difficult or expensive to obtain.

[0003] In summary, since traditional 3D medical image segmentation methods only train images from one direction, it is not conducive to learning complementary information for 3D medical image segmentation. In addition, the collection and annotation of medical image datasets are difficult, and small amounts of data cannot produce ideal results. Therefore, the segmentation performance of traditional 3D medical image segmentation methods is still poor. It is very necessary to propose a new 3D medical image segmentation method to solve the above problems. Summary of the Invention

[0004] The purpose of the present invention is to solve the problem of poor segmentation performance of traditional 3D medical image segmentation methods and propose a 3D cardiac image segmentation system based on two-view and semi-supervised attention model.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a 3D cardiac image segmentation system based on two views and a semi-supervised attention model, the system includes a 3D cardiac image acquisition module, a 3D cardiac image preprocessing module, an image blending module and a semi-supervised attention model, wherein:

[0006] The 3D cardiac image acquisition module is used to acquire a 3D cardiac image and annotate a segmentation label on a portion of the acquired 3D cardiac image;

[0007] The 3D cardiac image preprocessing module is used to preprocess the acquired 3D cardiac image to obtain various image patches;

[0008] The image mixing module is used to process the image patches to generate a mixed image;

[0009] The semi-supervised attention model is trained based on labeled 3D heart images, unlabeled 3D heart images, and mixed images;

[0010] The trained semi-supervised attention model is used to segment the 3D heart image to be segmented.

[0011] Furthermore, the 3D cardiac image acquired by the 3D cardiac image acquisition module comes from the LA dataset and the CETUS dataset.

[0012] Furthermore, the working process of the 3D cardiac image preprocessing module is as follows:

[0013] After random rotation, random flipping and scaling, the 3D cardiac images were center cropped and randomly cropped. The 3D cardiac images in the LA dataset were preprocessed to an image patch size of 112×112×80, and the 3D cardiac images in the CETUS dataset were preprocessed to an image patch size of 192×192×64.

[0014] Furthermore, the working process of the image mixing module is as follows:

[0015] For any two labeled image patches:

[0016] Randomly select the position and size of the cropping box, and then cut out the image in the cropping box from the two labeled image patches according to the position and size of the cropping box. The images cut out from the two labeled image patches are respectively denoted as x A and x B ;

[0017] The generated fusion image for: Where M is the mask matrix;

[0018] According to x A The label and x B Generate mixed images with labels Tags Among them, y A is x A The label, y B is x B The label, λ is the weight;

[0019] Using the generated fused image Replace the position of the cropped image in the original image patch to obtain a mixed image;

[0020] For any two unlabeled image patches:

[0021] Randomly select the position and size of the cropping box, and then cut out the image in the cropping box from the two labeled image patches according to the position and size of the cropping box. The images cut out from the two labeled image patches are recorded as x′ A and x′ B ;

[0022] The generated fusion image for: Where M is the mask matrix;

[0023] Using the generated fused image Replace the position of the cropped image in the original image patch to obtain a mixed image.

[0024] Furthermore, the semi-supervised attention model includes a supervision module and a segmentation network, and the segmentation network includes a first segmentation module and a second segmentation module;

[0025] The supervision module is used to obtain weight maps in both coronal and transverse directions;

[0026] The first segmentation module is trained based on labeled 3D cardiac images, unlabeled 3D cardiac images, mixed images, and weight maps in the coronal plane direction;

[0027] The second segmentation module is trained based on labeled 3D cardiac images, unlabeled 3D cardiac images, mixed images, and weight maps in the cross-sectional direction.

[0028] Furthermore, the working process of the supervision module is as follows:

[0029] For the coronal direction

[0030] Step 1: Select the middle slice from the coronal plane of the 3D medical image, and record the weight of the middle slice as weight;

[0031] Step 2: For any other slice in the coronal direction, calculate the weight of the slice according to the distance d between the slice and the middle slice. d ;

[0032] Step 3: generating a weight map of the three-dimensional medical image in the coronal direction according to the weights of each slice in the coronal direction;

[0033] Similarly, a weight map of the three-dimensional medical image in the cross-sectional direction is generated.

[0034] Furthermore, the first segmentation module includes an encoder part and a decoder part, and the encoder part includes four downsampling units, and the decoder part includes four upsampling units;

[0035] The first downsampling unit includes an adaptive pooling subunit, a first convolutional layer, a CBAM subunit, and a second convolutional layer; and the working process of the first downsampling unit is as follows:

[0036] The input of the first segmentation module is used as the input of the first downsampling unit of the encoder part. In the first downsampling unit, the input first passes through the adaptive pooling subunit, the first convolutional layer, and the CBAM subunit in sequence. Then, the input of the first downsampling unit and the output of the CBAM subunit are element-wise summed to obtain the element-wise summation result A. Finally, the element-wise summation result A passes through the second convolutional layer, and the output of the second convolutional layer is used as the output of the first downsampling unit;

[0037] The second downsampling unit includes the first convolutional layer, the second convolutional layer, the CBAM subunit and the third convolutional layer; and the working process of the second downsampling unit is:

[0038] The output of the first downsampling unit is used as the input of the second downsampling unit. In the second downsampling unit, the input first passes through the first convolutional layer, the second convolutional layer, and the CBAM subunit in sequence. Then, the input of the second downsampling unit and the output of the CBAM subunit are element-wise summed to obtain the element-wise summation result B. Finally, the element-wise summation result B passes through the third convolutional layer, and the output of the third convolutional layer is used as the output of the second downsampling unit.

[0039] The third downsampling unit includes the first convolution layer, the second convolution layer, the third convolution layer, the CBAM subunit and the fourth convolution layer; and the working process of the third downsampling unit is:

[0040] The output of the second downsampling unit is used as the input of the third downsampling unit. In the third downsampling unit, the input first passes through the first convolutional layer, the second convolutional layer, the third convolutional layer and the CBAM subunit in sequence. Then, the input of the third downsampling unit and the output of the CBAM subunit are element-wise summed to obtain the element-wise summation result C. Finally, the element-wise summation result C passes through the fourth convolutional layer, and the output of the fourth convolutional layer is used as the output of the third downsampling unit.

[0041] The fourth downsampling unit includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a CBAM subunit, and a fourth convolutional layer; and the working process of the fourth downsampling unit is as follows:

[0042] The output of the third downsampling unit is used as the input of the fourth downsampling unit. In the fourth downsampling unit, the input first passes through the first convolutional layer, the second convolutional layer, the third convolutional layer and the CBAM subunit in sequence. Then, the input of the fourth downsampling unit and the output of the CBAM subunit are element-wise summed to obtain the element-wise summation result D. Finally, the element-wise summation result D passes through the fourth convolutional layer, and the output of the fourth convolutional layer is used as the output of the fourth downsampling unit.

[0043] The element-by-element summation result D then passes through three convolutional layers in sequence to obtain the output E, and then E and D are element-by-element summed to obtain F, which then passes through the deconvolution layer to obtain the output F' of the deconvolution layer;

[0044] The first upsampling unit includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a deconvolutional layer; and the working process of the first upsampling unit is:

[0045] After feature concatenation of the element-by-element summation result D and the output F' of the deconvolution layer, the feature concatenation result is recorded as D'. In the first upsampling unit, the feature concatenation result D' passes through the first convolution layer, the second convolution layer, and the third convolution layer in sequence. The output of the third convolution layer is then element-wise summed with F'. The element-by-element summation result then passes through the deconvolution layer, and the output of the deconvolution layer is used as the output of the first upsampling unit.

[0046] The second upsampling unit includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a deconvolutional layer; and the working process of the second upsampling unit is:

[0047] After feature concatenation of the element-by-element summation result C and the output of the first upsampling unit, the feature concatenation result is passed through the first convolutional layer, the second convolutional layer, and the third convolutional layer in sequence. The output of the third convolutional layer is then element-wise summed with the output of the first upsampling unit. The element-by-element summation result is then passed through the deconvolution layer, and the output of the deconvolution layer is used as the output of the second upsampling unit.

[0048] The third upsampling unit includes a first convolutional layer, a second convolutional layer, and a deconvolutional layer; and the working process of the third upsampling unit is:

[0049] After feature concatenation of the element-by-element summation result B and the output of the second upsampling unit, the feature concatenation result is sequentially passed through the first convolutional layer and the second convolutional layer. The output of the second convolutional layer and the output of the second upsampling unit are then element-wise summed. The element-by-element summation result is then passed through the deconvolution layer, and the output of the deconvolution layer is used as the output of the third upsampling unit.

[0050] The fourth upsampling unit includes the first convolutional layer; and the working process of the fourth upsampling unit is:

[0051] After feature concatenation of the element-by-element summation result A and the output of the third upsampling unit, the feature concatenation result is passed through the first convolutional layer. The output of the first convolutional layer is then element-wise summed with the output of the third upsampling unit. The element-by-element sum is used as the output of the fourth upsampling unit, thus obtaining the output of the first segmentation module.

[0052] The network structure of the second segmentation module is the same as that of the first segmentation module.

[0053] Furthermore, the working process of the adaptive pooling subunit is as follows:

[0054] The input of the adaptive pooling subunit is denoted as x. x passes through the adaptive average pooling layer, the sigmoid activation function layer, and the expansion layer in sequence. The output of the expansion layer is multiplied element-wise by x, and the result of the element-wise multiplication is used as the output of the adaptive pooling subunit.

[0055] The working process of the CBAM subunit is:

[0056] The input of the CBAM sub-unit is denoted as y. First, y is passed through the convolutional layer, and the output of the convolutional layer is denoted as y'. y' is passed through the channel attention layer, and the output of the channel attention layer is fused with y'. The fusion result is passed through the convolutional layer again to obtain the output y" of the convolutional layer. y" is passed through the spatial attention layer, and the output of the spatial attention layer is fused with y", and the fusion result is used as the output of the CBAM sub-unit.

[0057] Furthermore, the first segmentation module and the second segmentation module are jointly trained; the specific process of the joint training is:

[0058] Step 1: Initialize training data batch t=1;

[0059] Step 2: Select the tth batch of training data from the coronal plane data set;

[0060] Step 3: Use the training data selected in step 2 to train the first segmentation module. The loss function L used in training is total for:

[0061] L total =L sup +L semi +λ·L uncertainty

[0062] Among them, L sup is the segmentation loss of the labeled image dataset, L semi is the segmentation loss of the unlabeled image dataset, L uncertainty is the uncertainty correction loss, λ is the weight coefficient;

[0063] Segmentation loss L sup and L semi Both include cross loss L ce and Dice loss; among them:

[0064] For unlabeled images, the cross loss L ce The calculation method is:

[0065]

[0066] Among them, w i is the i-th voxel of the weight map W in the coronal plane, H, W and D are the length, width and height of the image, p i represents the probability that the i-th voxel belongs to the foreground, y i represents the pseudo label of the i-th voxel;

[0067] For unlabeled images, the Dice loss is calculated as:

[0068]

[0069] Similarly, calculate the cross loss and Dice loss of labeled images;

[0070] Uncertainty correction loss L uncertainty for:

[0071]

[0072] in, is the predicted value output by the s-th upsampling unit of the i-th voxel, is the average value of the predicted value output by all S upsampling units of the i-th voxel, Represents the correction value of the i-th voxel after the s-th upsampling unit, express uncertainty,||·|| 2 represents the 2-norm;

[0073]

[0074] in, yes The value at the jth channel, yes The value at the jth channel, C is the number of channels;

[0075] Step 4: Using the average exponential parameter transfer method, the parameters of the first segmentation module trained in step 3 are transferred to the second segmentation module, and the second segmentation module updates its own parameters according to the received parameters;

[0076] Step 5: Select the tth batch of training data from the cross-sectional data set;

[0077] Step 6: Use the training data selected in step 5 to train the second segmentation module after parameter update;

[0078] Step 7: Using the average exponential parameter transfer method, the parameters of the second segmentation module trained in step 6 are transferred to the first segmentation module, and the first segmentation module updates its own parameters according to the received parameters;

[0079] Step 8: Set the batch number t = t + 1, then return to step 2 and continue until the training is completed.

[0080] Furthermore, the trained semi-supervised attention model is used to segment the 3D heart image to be segmented, specifically:

[0081] The 3D heart image to be segmented is used as input of the trained first segmentation module and the second segmentation module, respectively, to obtain a segmentation result output by the first segmentation module and a segmentation result output by the second segmentation module;

[0082] Then, the best segmentation result is selected from the segmentation results output by the first segmentation module and the segmentation results output by the second segmentation module according to the evaluation index.

[0083] The beneficial effects of the present invention are:

[0084] The present invention adopts a data augmentation mechanism, namely, using mixed areas in the image to expand the data set to enhance the performance in the 3D cardiac medical image segmentation task. By segmenting the 3D cardiac image from both the coronal and cross-sectional perspectives, complementary segmentation information from the two perspectives can be obtained. In addition, the segmentation network designed by the present invention pays more attention to the segmented area and edge parts. At the same time, by adopting a semi-supervised learning approach, the image segmentation performance is guaranteed by combining the complementary segmentation information from the two perspectives without the need to annotate the entire 3D cardiac image. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to the present invention;

[0086] Figure 2 It is the structural diagram of the segmentation module;

[0087] Adaptive Channel Attention Block represents the adaptive pooling subunit, input represents input, output represents output, Adaptive average pooling represents the adaptive average pooling layer, Element-wise Multiplication represents element-wise multiplication, Channel attention represents the channel attention layer, and Spatitalattention represents the spatial attention layer. DETAILED DESCRIPTION

[0088] Specific implementation method 1: Combination Figure 1 This embodiment describes a 3D cardiac image segmentation system based on two views and a semi-supervised attention model, comprising a 3D cardiac image acquisition module, a 3D cardiac image preprocessing module, an image blending module, and a semi-supervised attention model, wherein:

[0089] The 3D cardiac image acquisition module is used to acquire 3D cardiac images and annotate the acquired 3D cardiac images with segmentation labels (only a small portion of the images needs to be labeled, and the other images do not need to be labeled);

[0090] The 3D cardiac image preprocessing module is used to preprocess the acquired 3D cardiac image to obtain various image patches;

[0091] The image mixing module is used to process the image patches to generate a mixed image;

[0092] The semi-supervised attention model is trained based on labeled 3D heart images, unlabeled 3D heart images, and mixed images (the training stop condition is to reach a preset number of training times);

[0093] The trained semi-supervised attention model is used to segment the 3D heart image to be segmented.

[0094] After partially annotating the obtained three-dimensional cardiac image dataset, the present invention divides the dataset into a training set and a test set in a ratio of 0.8:0.2; and uses the training set and the test set to train and test the semi-supervised attention network.

[0095] Specific embodiment 2: This embodiment differs from specific embodiment 1 in that the 3D heart image acquired by the 3D heart image acquisition module comes from the LA dataset and the CETUS dataset.

[0096] Other steps and parameters are the same as those in the first embodiment.

[0097] Specific embodiment three: This embodiment differs from specific embodiments one or two in that the working process of the 3D cardiac image preprocessing module is as follows:

[0098] After random rotation, random flipping and scaling, the 3D cardiac images were center cropped and randomly cropped. The 3D cardiac images in the LA dataset were preprocessed to an image patch size of 112×112×80, and the 3D cardiac images in the CETUS dataset were preprocessed to an image patch size of 192×192×64.

[0099] Other steps and parameters are the same as those in the first or second embodiment.

[0100] Specific embodiment 4: This embodiment differs from any one of specific embodiments 1 to 3 in that the working process of the image mixing module is as follows:

[0101] For any two labeled image patches:

[0102] Randomly select the position and size of the cropping box, and then cut out the image in the cropping box from the two labeled image patches according to the position and size of the cropping box. The images cut out from the two labeled image patches are respectively denoted as x A and x B ;

[0103] The generated fusion image for: Where M is the mask matrix;

[0104] According to x A The label and x B Generate mixed images with labels Tags Among them, y A is x A The label, y B is x B The label, λ is the weight, the value is 0.5;

[0105] Using the generated fused image Replace the position of the cropped image in the original image patch to obtain a mixed image;

[0106] For any two unlabeled image patches:

[0107] Randomly select the position and size of the cropping box, and then cut out the image in the cropping box from the two labeled image patches according to the position and size of the cropping box. The images cut out from the two labeled image patches are recorded as x′ A and x′ B ;

[0108] The generated fusion image for: Where M is the mask matrix;

[0109] Using the generated fused image Replace the position of the cropped image in the original image patch to obtain a mixed image.

[0110] The other steps and parameters are the same as those in the first to third embodiments.

[0111] Specific embodiment 5: This embodiment differs from any one of specific embodiments 1 to 4 in that the semi-supervised attention model includes a supervision module and a segmentation network, and the segmentation network includes a first segmentation module and a second segmentation module;

[0112] The supervision module is used to obtain weight maps in both coronal and transverse directions;

[0113] The first segmentation module is trained based on labeled 3D cardiac images, unlabeled 3D cardiac images, mixed images, and weight maps in the coronal plane direction;

[0114] The second segmentation module is trained based on labeled 3D cardiac images, unlabeled 3D cardiac images, mixed images, and weight maps in the cross-sectional direction.

[0115] The other steps and parameters are the same as those in the first to fourth embodiments.

[0116] Specific embodiment 6: This embodiment differs from specific embodiments 1 to 5 in that the working process of the supervision module is as follows:

[0117] For the coronal direction

[0118] Step 1: Select the middle slice from the coronal plane of the 3D medical image, and record the weight of the middle slice as weight;

[0119] Step 2: For any other slice in the coronal direction, calculate the weight of the slice according to the distance d between the slice and the middle slice. d ;

[0120] Step 3: generating a weight map of the three-dimensional medical image in the coronal direction according to the weights of each slice in the coronal direction;

[0121] Similarly, a weight map of the three-dimensional medical image in the cross-sectional direction is generated.

[0122] The other steps and parameters are the same as those in the first to fifth embodiments.

[0123] Specific implementation method seven: combination Figure 2 This embodiment is described. The difference between this embodiment and any one of the specific embodiments 1 to 6 is that the first segmentation module includes an encoder part and a decoder part, and the encoder part includes four downsampling units, and the decoder part includes four upsampling units;

[0124] The first downsampling unit includes an adaptive pooling subunit, a first convolutional layer (the convolution kernel size is 5×5×5), a CBAM subunit, and a second convolutional layer (the convolution kernel size is 2×2×2). The working process of the first downsampling unit is as follows:

[0125] The input of the first segmentation module is used as the input of the first downsampling unit of the encoder part. In the first downsampling unit, the input first passes through the adaptive pooling subunit, the first convolutional layer, and the CBAM subunit in sequence. Then, the input of the first downsampling unit and the output of the CBAM subunit are element-wise summed to obtain the element-wise summation result A. Finally, the element-wise summation result A passes through the second convolutional layer, and the output of the second convolutional layer is used as the output of the first downsampling unit;

[0126] The second downsampling unit includes the first convolution layer (convolution kernel size is 5×5×5), the second convolution layer (convolution kernel size is 5×5×5), the CBAM subunit and the third convolution layer (convolution kernel size is 2×2×2); and the working process of the second downsampling unit is:

[0127] The output of the first downsampling unit is used as the input of the second downsampling unit. In the second downsampling unit, the input first passes through the first convolutional layer, the second convolutional layer, and the CBAM subunit in sequence. Then, the input of the second downsampling unit and the output of the CBAM subunit are element-wise summed to obtain the element-wise summation result B. Finally, the element-wise summation result B passes through the third convolutional layer, and the output of the third convolutional layer is used as the output of the second downsampling unit.

[0128] The third downsampling unit includes the first convolution layer (convolution kernel size is 5×5×5), the second convolution layer (convolution kernel size is 5×5×5), the third convolution layer (convolution kernel size is 5×5×5), the CBAM subunit and the fourth convolution layer (convolution kernel size is 2×2×2); and the working process of the third downsampling unit is:

[0129] The output of the second downsampling unit is used as the input of the third downsampling unit. In the third downsampling unit, the input first passes through the first convolutional layer, the second convolutional layer, the third convolutional layer and the CBAM subunit in sequence. Then, the input of the third downsampling unit and the output of the CBAM subunit are element-wise summed to obtain the element-wise summation result C. Finally, the element-wise summation result C passes through the fourth convolutional layer, and the output of the fourth convolutional layer is used as the output of the third downsampling unit.

[0130] The fourth downsampling unit includes the first convolution layer (convolution kernel size is 5×5×5), the second convolution layer (convolution kernel size is 5×5×5), the third convolution layer (convolution kernel size is 5×5×5), the CBAM subunit and the fourth convolution layer (convolution kernel size is 2×2×2); and the working process of the fourth downsampling unit is:

[0131] The output of the third downsampling unit is used as the input of the fourth downsampling unit. In the fourth downsampling unit, the input first passes through the first convolutional layer, the second convolutional layer, the third convolutional layer and the CBAM subunit in sequence. Then, the input of the fourth downsampling unit and the output of the CBAM subunit are element-wise summed to obtain the element-wise summation result D. Finally, the element-wise summation result D passes through the fourth convolutional layer, and the output of the fourth convolutional layer is used as the output of the fourth downsampling unit.

[0132] The element-by-element summation result D is then passed through three convolutional layers (the convolution kernel size is 5×5×5) to obtain the output E. E and D are then element-by-element summed to obtain F. F is then passed through a deconvolution layer to obtain the output F' of the deconvolution layer.

[0133] The first upsampling unit includes the first convolution layer (convolution kernel size is 5×5×5), the second convolution layer (convolution kernel size is 2×2×2), the third convolution layer (convolution kernel size is 5×5×5) and the deconvolution layer; and the working process of the first upsampling unit is:

[0134] After feature concatenation of the element-by-element summation result D and the output F' of the deconvolution layer, the feature concatenation result is recorded as D'. In the first upsampling unit, the feature concatenation result D' passes through the first convolution layer, the second convolution layer, and the third convolution layer in sequence. The output of the third convolution layer is then element-wise summed with F'. The element-by-element summation result then passes through the deconvolution layer, and the output of the deconvolution layer is used as the output of the first upsampling unit.

[0135] The second upsampling unit includes the first convolution layer (convolution kernel size is 5×5×5), the second convolution layer (convolution kernel size is 2×2×2), the third convolution layer (convolution kernel size is 5×5×5) and the deconvolution layer; and the working process of the second upsampling unit is:

[0136] After feature concatenation of the element-by-element summation result C and the output of the first upsampling unit, the feature concatenation result is passed through the first convolutional layer, the second convolutional layer, and the third convolutional layer in sequence. The output of the third convolutional layer is then element-wise summed with the output of the first upsampling unit. The element-by-element summation result is then passed through the deconvolution layer, and the output of the deconvolution layer is used as the output of the second upsampling unit.

[0137] The third upsampling unit includes the first convolution layer (the convolution kernel size is 5×5×5), the second convolution layer (the convolution kernel size is 5×5×5), and the deconvolution layer; and the working process of the third upsampling unit is as follows:

[0138] After feature concatenation of the element-by-element summation result B and the output of the second upsampling unit, the feature concatenation result is sequentially passed through the first convolutional layer and the second convolutional layer. The output of the second convolutional layer and the output of the second upsampling unit are then element-wise summed. The element-by-element summation result is then passed through the deconvolution layer, and the output of the deconvolution layer is used as the output of the third upsampling unit.

[0139] The fourth upsampling unit includes the first convolution layer (the convolution kernel size is 5×5×5); and the working process of the fourth upsampling unit is:

[0140] After feature concatenation of the element-by-element summation result A and the output of the third upsampling unit, the feature concatenation result is passed through the first convolutional layer. The output of the first convolutional layer is then element-wise summed with the output of the third upsampling unit. The element-by-element sum is used as the output of the fourth upsampling unit, thus obtaining the output of the first segmentation module.

[0141] The network structure of the second segmentation module is the same as that of the first segmentation module.

[0142] The other steps and parameters are the same as those in the first to sixth embodiments.

[0143] Specific embodiment eight: This embodiment differs from any one of specific embodiments one to seven in that the working process of the adaptive pooling subunit is as follows:

[0144] The input of the adaptive pooling subunit is denoted as x. x is passed through the adaptive average pooling layer, the sigmoid activation function layer, and the expansion layer (expand) in sequence. The output of the expansion layer is multiplied element-wise by x, and the result of the element-wise multiplication is used as the output of the adaptive pooling subunit.

[0145] The working process of the CBAM subunit is:

[0146] The input of the CBAM sub-unit is denoted as y. First, y is passed through the convolutional layer, and the output of the convolutional layer is denoted as y'. y' is passed through the channel attention layer, and the output of the channel attention layer is fused with y'. The fusion result is passed through the convolutional layer again to obtain the output y" of the convolutional layer. y" is passed through the spatial attention layer, and the output of the spatial attention layer is fused with y", and the fusion result is used as the output of the CBAM sub-unit.

[0147] The other steps and parameters are the same as those in the first to seventh embodiments.

[0148] Specific embodiment 9: This embodiment differs from any one of specific embodiments 1 to 8 in that the first segmentation module and the second segmentation module are jointly trained; the specific process of the joint training is as follows:

[0149] Step 1: Initialize training data batch t=1;

[0150] Step 2: Select the tth batch of training data from the coronal plane data set;

[0151] Step 3: Use the training data selected in step 2 to train the first segmentation module. The loss function L used in training is total for:

[0152] L total =L sup +L semi +λ·L uncertainty

[0153] Among them, L sup is the segmentation loss of the labeled image dataset, L semi is the segmentation loss of the unlabeled image dataset, L uncertainty is the uncertainty correction loss, λ is the weight coefficient (set by the Gaussian warm-up function, usually set to 0.5);

[0154] Segmentation loss L sup and L semi Both include cross loss L ce and Dice loss; among them:

[0155] For unlabeled images, the cross loss L ce The calculation method is:

[0156]

[0157] Among them, w i is the i-th voxel of the weight map W in the coronal plane, H, W and D are the length, width and height of the image, p i represents the probability that the i-th voxel belongs to the foreground, y i represents the pseudo label of the i-th voxel;

[0158] For unlabeled images, the Dice loss is calculated as:

[0159]

[0160] Similarly, calculate the cross loss and Dice loss of labeled images;

[0161] Uncertainty correction loss L uncertainty for:

[0162]

[0163] in, is the predicted value output by the s-th upsampling unit of the i-th voxel, is the average value (also called uncertainty value) of the predicted value output by all S upsampling units for the i-th voxel, Represents the correction value of the i-th voxel after the s-th upsampling unit, express uncertainty,||·|| 2 represents the 2-norm;

[0164]

[0165] in, yes The value at the jth channel, yes The value at the jth channel, C is the number of channels;

[0166] Step 4: Use the average exponential parameter transfer method to pass the parameters of the first segmentation module trained in step 3 to the second segmentation module. The second segmentation module updates its own parameters based on the received parameters (the update method is the same as that of the teacher model and student model);

[0167] Step 5: Select the tth batch of training data from the cross-sectional data set;

[0168] Step 6: Use the training data selected in step 5 to train the second segmentation module after parameter update;

[0169] The loss function calculation method for the second segmentation module during training is the same as that for the first segmentation module;

[0170] Step 7: Using the average exponential parameter transfer method, the parameters of the second segmentation module trained in step 6 are transferred to the first segmentation module, and the first segmentation module updates its own parameters according to the received parameters;

[0171] Step 8: Set the batch number t = t + 1, then return to step 2 and continue until the training is completed.

[0172] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0173] Specific embodiment 10: This embodiment differs from any one of specific embodiments 1 to 9 in that the trained semi-supervised attention model is used to segment the 3D heart image to be segmented, specifically:

[0174] The 3D heart image to be segmented is used as input of the trained first segmentation module and the second segmentation module, respectively, to obtain a segmentation result output by the first segmentation module and a segmentation result output by the second segmentation module;

[0175] Then, the best segmentation result is selected from the segmentation results output by the first segmentation module and the segmentation results output by the second segmentation module according to the evaluation index.

[0176] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0177] In this embodiment, the segmentation result output by the first segmentation module is compared with the segmentation result output by the second segmentation module according to the evaluation indicators Dice, Jaccard, 95HD and ASD, and the better segmentation result is selected.

[0178] The above examples are merely illustrative of the calculation model and process of the present invention and are not intended to limit the embodiments of the present invention. Persons skilled in the art will readily appreciate that other variations or modifications based on the above description are possible. This list of embodiments is not exhaustive; however, any obvious variations or modifications derived from the technical solution of the present invention remain within the scope of protection of the present invention.

Claims

1. A 3D cardiac image segmentation system based on two-view and semi-supervised attention model, characterized by: The system includes a 3D cardiac image acquisition module, a 3D cardiac image preprocessing module, an image blending module, and a semi-supervised attention model, wherein: The 3D cardiac image acquisition module is used to acquire a 3D cardiac image and annotate a segmentation label on a portion of the acquired 3D cardiac image; The 3D cardiac image preprocessing module is used to preprocess the acquired 3D cardiac image to obtain various image patches; The image mixing module is used to process the image patches to generate a mixed image; The working process of the image mixing module is as follows: For any two labeled image patches: Randomly select the position and size of the cropping box, and then cut out the image in the cropping box from the two labeled image patches according to the position and size of the cropping box. The images cut out from the two labeled image patches are respectively denoted as x A and x B ; The generated fusion image for: Where M is the mask matrix; According to x A The label and x B Generate mixed images with labels Tags Among them, y A is x A The label, y B is x B The label, λ is the weight; Using the generated fused image Replace the position of the cropped image in the original image patch to obtain a mixed image; For any two unlabeled image patches: Randomly select the position and size of the cropping box, and then cut out the image in the cropping box from the two labeled image patches according to the position and size of the cropping box. The images cut out from the two labeled image patches are respectively recorded as x' A and x' B ; The generated fusion image for: Where M is the mask matrix; Using the generated fused image Replace the position of the cropped image in the original image patch to obtain a mixed image; The semi-supervised attention model is trained based on labeled 3D heart images, unlabeled 3D heart images, and mixed images; The semi-supervised attention model includes a supervision module and a segmentation network, and the segmentation network includes a first segmentation module and a second segmentation module; The supervision module is used to obtain weight maps in both coronal and transverse directions; The first segmentation module is trained based on labeled 3D cardiac images, unlabeled 3D cardiac images, mixed images, and weight maps in the coronal plane direction; The second segmentation module is trained based on labeled 3D cardiac images, unlabeled 3D cardiac images, mixed images, and weight maps in the cross-sectional direction; The working process of the supervision module is as follows: For the coronal direction Step 1: Select the middle slice from the coronal plane of the 3D medical image, and record the weight of the middle slice as weight; Step 2: For any other slice in the coronal direction, calculate the weight of the slice according to the distance d between the slice and the middle slice. d ; Step 3: generating a weight map of the three-dimensional medical image in the coronal direction according to the weights of each slice in the coronal direction; Similarly, a weight map of the three-dimensional medical image in the cross-sectional direction is generated; The trained semi-supervised attention model is used to segment the 3D heart image to be segmented.

2. A 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 1, characterized in that: The 3D heart images acquired by the 3D heart image acquisition module are from the LA dataset and the CETUS dataset.

3. The 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 2, characterized in that: The working process of the 3D cardiac image preprocessing module is as follows: After random rotation, random flipping and scaling, the 3D cardiac images were center cropped and randomly cropped. The 3D cardiac images in the LA dataset were preprocessed to an image patch size of 112×112×80, and the 3D cardiac images in the CETUS dataset were preprocessed to an image patch size of 192×192×64.

4. The 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 3, characterized in that: The first segmentation module includes an encoder part and a decoder part, wherein the encoder part includes four downsampling units and the decoder part includes four upsampling units; The first downsampling unit includes an adaptive pooling subunit, a first convolutional layer, a CBAM subunit, and a second convolutional layer; and the working process of the first downsampling unit is as follows: The input of the first segmentation module is used as the input of the first downsampling unit of the encoder part. In the first downsampling unit, the input first passes through the adaptive pooling subunit, the first convolutional layer, and the CBAM subunit in sequence. Then, the input of the first downsampling unit and the output of the CBAM subunit are element-wise summed to obtain the element-wise summation result A. Finally, the element-wise summation result A passes through the second convolutional layer, and the output of the second convolutional layer is used as the output of the first downsampling unit; The second downsampling unit includes the first convolutional layer, the second convolutional layer, the CBAM subunit and the third convolutional layer; and the working process of the second downsampling unit is: The output of the first downsampling unit is used as the input of the second downsampling unit. In the second downsampling unit, the input first passes through the first convolutional layer, the second convolutional layer, and the CBAM subunit in sequence. Then, the input of the second downsampling unit and the output of the CBAM subunit are element-wise summed to obtain the element-wise summation result B. Finally, the element-wise summation result B passes through the third convolutional layer, and the output of the third convolutional layer is used as the output of the second downsampling unit. The third downsampling unit includes the first convolution layer, the second convolution layer, the third convolution layer, the CBAM subunit and the fourth convolution layer; and the working process of the third downsampling unit is: The output of the second downsampling unit is used as the input of the third downsampling unit. In the third downsampling unit, the input first passes through the first convolutional layer, the second convolutional layer, the third convolutional layer and the CBAM subunit in sequence. Then, the input of the third downsampling unit and the output of the CBAM subunit are element-wise summed to obtain the element-wise summation result C. Finally, the element-wise summation result C passes through the fourth convolutional layer, and the output of the fourth convolutional layer is used as the output of the third downsampling unit. The fourth downsampling unit includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a CBAM subunit, and a fourth convolutional layer; and the working process of the fourth downsampling unit is as follows: The output of the third downsampling unit is used as the input of the fourth downsampling unit. In the fourth downsampling unit, the input first passes through the first convolutional layer, the second convolutional layer, the third convolutional layer and the CBAM subunit in sequence. Then, the input of the fourth downsampling unit and the output of the CBAM subunit are element-wise summed to obtain the element-wise summation result D. Finally, the element-wise summation result D passes through the fourth convolutional layer, and the output of the fourth convolutional layer is used as the output of the fourth downsampling unit. The element-by-element summation result D then passes through three convolutional layers in sequence to obtain the output E, and then E and D are element-by-element summed to obtain F, which then passes through the deconvolution layer to obtain the output F' of the deconvolution layer; The first upsampling unit includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a deconvolutional layer; and the working process of the first upsampling unit is: After feature concatenation of the element-by-element summation result D and the output F' of the deconvolution layer, the feature concatenation result is recorded as D'. In the first upsampling unit, the feature concatenation result D' passes through the first convolution layer, the second convolution layer, and the third convolution layer in sequence. The output of the third convolution layer is then element-wise summed with F'. The element-by-element summation result then passes through the deconvolution layer, and the output of the deconvolution layer is used as the output of the first upsampling unit. The second upsampling unit includes a first convolutional layer, a second convolutional layer, a third convolutional layer, and a deconvolutional layer; and the working process of the second upsampling unit is: After feature concatenation of the element-by-element summation result C and the output of the first upsampling unit, the feature concatenation result is passed through the first convolutional layer, the second convolutional layer, and the third convolutional layer in sequence. The output of the third convolutional layer is then element-wise summed with the output of the first upsampling unit. The element-by-element summation result is then passed through the deconvolution layer, and the output of the deconvolution layer is used as the output of the second upsampling unit. The third upsampling unit includes a first convolutional layer, a second convolutional layer, and a deconvolutional layer; and the working process of the third upsampling unit is: After feature concatenation of the element-by-element summation result B and the output of the second upsampling unit, the feature concatenation result is sequentially passed through the first convolutional layer and the second convolutional layer. The output of the second convolutional layer and the output of the second upsampling unit are then element-wise summed. The element-by-element summation result is then passed through the deconvolution layer, and the output of the deconvolution layer is used as the output of the third upsampling unit. The fourth upsampling unit includes the first convolutional layer; and the working process of the fourth upsampling unit is: After feature concatenation of the element-by-element summation result A and the output of the third upsampling unit, the feature concatenation result is passed through the first convolutional layer. The output of the first convolutional layer is then element-wise summed with the output of the third upsampling unit. The element-by-element sum is used as the output of the fourth upsampling unit, thus obtaining the output of the first segmentation module. The network structure of the second segmentation module is the same as that of the first segmentation module.

5. The 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 4, characterized in that: The working process of the adaptive pooling subunit is as follows: The input of the adaptive pooling subunit is denoted as x. x passes through the adaptive average pooling layer, the sigmoid activation function layer, and the expansion layer in sequence. The output of the expansion layer is multiplied element-wise by x, and the result of the element-wise multiplication is used as the output of the adaptive pooling subunit. The working process of the CBAM subunit is: The input of the CBAM sub-unit is denoted as y. First, y is passed through the convolutional layer, and the output of the convolutional layer is denoted as y'. y' is passed through the channel attention layer, and the output of the channel attention layer is fused with y'. The fusion result is passed through the convolutional layer again to obtain the output y" of the convolutional layer. y" is passed through the spatial attention layer, and the output of the spatial attention layer is fused with y", and the fusion result is used as the output of the CBAM sub-unit.

6. The 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 5, characterized in that: The first segmentation module and the second segmentation module are jointly trained; the specific process of the joint training is: Step 1: Initialize training data batch t=1; Step 2: Select the tth batch of training data from the coronal plane data set; Step 3: Use the training data selected in step 2 to train the first segmentation module. The loss function L used in training is total for: L total =L sup +L semi +λ·L uncertainty Among them, L sup is the segmentation loss of the labeled image dataset, L semi is the segmentation loss of the unlabeled image dataset, L uncertainty is the uncertainty correction loss, λ is the weight coefficient; Segmentation loss L sup and L semi Both include cross loss L ce and Dice loss; among them: For unlabeled images, the cross loss L ce The calculation method is: Among them, w i is the i-th voxel of the weight map W in the coronal plane, H, W and D are the length, width and height of the image, p i represents the probability that the i-th voxel belongs to the foreground, y i represents the pseudo label of the i-th voxel; For unlabeled images, the Dice loss is calculated as: Similarly, calculate the cross loss and Dice loss of labeled images; Uncertainty correction loss L uncertainty for: in, is the predicted value output by the s-th upsampling unit of the i-th voxel, is the average value of the predicted value output by all S upsampling units of the i-th voxel, Represents the correction value of the i-th voxel after the s-th upsampling unit, express uncertainty,||·|| 2 represents the 2-norm; in, yes The value at the jth channel, yes The value at the jth channel, C is the number of channels; Step 4: Using the average exponential parameter transfer method, the parameters of the first segmentation module trained in step 3 are transferred to the second segmentation module, and the second segmentation module updates its own parameters according to the received parameters; Step 5: Select the tth batch of training data from the cross-sectional data set; Step 6: Use the training data selected in step 5 to train the second segmentation module after parameter update; Step 7: Using the average exponential parameter transfer method, the parameters of the second segmentation module trained in step 6 are transferred to the first segmentation module, and the first segmentation module updates its own parameters according to the received parameters; Step 8: Set the batch number t = t + 1, then return to step 2 and continue until the training is completed.

7. The 3D cardiac image segmentation system based on two-view and semi-supervised attention model according to claim 6, characterized in that: The trained semi-supervised attention model is used to segment the 3D heart image to be segmented, specifically: The 3D heart image to be segmented is used as input of the trained first segmentation module and the second segmentation module, respectively, to obtain a segmentation result output by the first segmentation module and a segmentation result output by the second segmentation module; Then, the best segmentation result is selected from the segmentation results output by the first segmentation module and the segmentation results output by the second segmentation module according to the evaluation index.

Images