Mammary gland lesion image semi-supervised segmentation method

Abstract

The invention discloses a breast lesion image semi-supervised segmentation method. The method belongs to the technical field of medical image processing and semi-supervised learning. The technical problem of improving the feature extraction capability of a semi-supervised segmentation network in a complex breast lesion segmentation scene under a small number of labeled samples and realizing accurate segmentation of a breast lesion is solved. A dual-scale image of a mammary gland image is generated, so that the network can learn features of mammary gland lesions from different scales, the ability of the network to extract features of mammary gland lesions with large size and form differences is improved, data perturbation is performed on the dual-scale image, and a generator performs mutual generation on segmentation confidence maps of a source image and a perturbation image. And the network stability is improved. Through cascade inverse residual coding feature fusion and decoding feature fusion, more potential features are mined by using existing features, the segmentation performance of the network is improved under a small number of labeled samples, and accurate segmentation of a breast lesion can be realized by inputting a breast image into a final segmentation network.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing and semi-supervised learning, and in particular relates to a semi-supervised segmentation method for breast lesion images. Background Art

[0002] Breast lesion examination is an important method for screening breast diseases. Early and accurate identification and assessment are crucial for prognosis. Accurate segmentation of breast lesions can help doctors accurately locate and delineate these areas, providing important support for early diagnosis and treatment of breast cancer, and improving diagnostic accuracy and efficiency.

[0003] Deep learning can automatically extract rich features from source images through multi-layer nonlinear transformations, including low-level features (such as pixel brightness and local edges) and high-level abstract features (such as the overall morphology, contextual relationships, and boundary contours of breast lesions). It does not require the manual design of complex feature extractors. This type of ability to process complex, nonlinear image features is conducive to the automatic segmentation of breast images with complex and variable shapes, sizes, and internal structures. The segmentation performance of data-driven deep convolutional neural networks depends on large amounts of high-quality data labels, while manual data labeling is time-consuming, labor-intensive, and costly, resulting in a scarcity of clinically available labeled data. Semi-supervised learning can use a small amount of labeled data and a large amount of unlabeled data to train the network, effectively alleviating the network's dependence on pixel-level labeled data and better meeting the needs of clinical application scenarios.

[0004] Due to the scarcity of clinical breast image labels and the complex internal structure of breast lesions, current semi-supervised segmentation networks remain challenging in extracting complex features from breast images. Improving the network's feature extraction capabilities in complex breast lesion segmentation scenarios with a small number of labeled samples, and achieving accurate breast lesion segmentation, remains a pressing issue for automated clinical breast lesion inspection. Summary of the Invention

[0005] In order to solve the technical problem of improving the feature extraction ability of the semi-supervised segmentation network in complex breast lesion segmentation scenarios with a small number of labeled samples and achieving accurate segmentation of breast lesions, the present invention proposes a semi-supervised segmentation method for breast lesion images.

[0006] The method comprises the following steps: S1. Collect breast source images, including labeled breast source images and unlabeled breast source images ,right Downsampling to obtain ,right Downsampling to obtain ,right Perform data perturbation to obtain ,right Perform data perturbation to obtain ; and , and , and Three sets of dual-scale images including the original scale image size and the downsampled scale image size are respectively formed; S2. Construct a semi-supervised segmentation network for breast lesions. The workflow in the network is as follows: S21, sending the dual-scale image to the shared encoder for feature encoding to generate a dual-scale encoded feature map; S22, sending the dual-scale coding feature map to the cascaded inverse residual coding feature fusion module across layers to perform cross-layer feature fusion to obtain a dual-scale cross-layer fusion feature map; S23, sending the dual-scale encoded feature map to two independent feature decoders, and fusing the dual-scale cross-layer fusion feature map with the decoding features of the corresponding layer through skip connections to obtain a dual-scale decoding feature map and a dual-scale segmentation image; S24, sending the dual-scale decoding feature map to a dual-scale decoding feature fusion device, performing step-by-step feature fusion and decoding on the dual-scale decoding feature map, and obtaining a dual-scale decoding fusion feature map and a breast lesion segmentation image; S3. Jointly train the parameters of the semi-supervised segmentation network for breast lesions based on the supervised loss of labeled images and the consistency loss of unlabeled images to obtain the final segmentation network. S4. Inputting the breast image to be segmented into the final segmentation network to achieve breast lesion segmentation.

[0007] Furthermore, the shared encoder performs feature encoding on the original scale image and the downsampled scale image to obtain the encoded feature map at the original scale. and the encoded feature map at the downsampling scale ,in, The encoding feature map corresponds to the shared encoder The output of the layer, the shared encoder uses the Res2net network.

[0008] Furthermore, cross-layer feature fusion is performed in the cascaded inverse residual coding feature fusion module as follows: ; ; in, express The fused feature map obtained after cross-layer feature fusion, express The fused feature map obtained after cross-layer feature fusion, represents the channel attention mechanism, Indicates the spatial intention mechanism, Indicates channel splicing operation, represents the bilinear interpolation upsampling operation, It represents the cascaded inverse residual module. The data processing flow in the cascaded inverse residual module is as follows: ; ; ; ; in, represents the input feature map, Represents the feature map after channel dimension increase, is a 3×3 depthwise separable convolution, is the batch normalization operation, is the activation function, represents 1×1 convolution, Represents the feature map after residual connection of the dimension-raising feature map; It is the feature map obtained by performing residual connection operation on the feature map after dimensionality reduction and the input feature map. Represents the feature map finally generated by the cascaded inverse residual module.

[0009] Furthermore, the dual-scale encoded feature maps are fed into two independent feature decoders, and and The dual-scale encoded feature map is decoded by passing the feature decoder through the jump connection to obtain the dual-scale decoded feature map and , and dual-scale segmented images and , specifically: ; ; in, represents transposed convolution; ; .

[0010] Furthermore, the dual-scale decoding feature map and The dual-scale decoding feature fusion is sent to the dual-scale feature fusion device to perform step-by-step feature fusion, aggregation and decoding on the dual-scale features to obtain the dual-scale decoding fusion feature map and breast lesion segmentation images : , dual-scale decoding feature fusion The specific operations performed are: ; ; ; ; ; in, GAFM represents the gated attention fusion module, GAFM The specific operations performed are: ; in, represents an element-wise multiplication operation, Represents grouped convolution, where the number of groups is half the number of channels. represents the first feature map of the input, represents the second feature map of the input, represents the output feature map; Decoding fusion features Perform step-by-step decoding to obtain breast lesion segmentation images , specifically: ; .

[0011] Furthermore, when the parameters of the semi-supervised segmentation network for breast lesions are jointly trained based on the supervised loss of labeled images and the consistency loss of unlabeled images, the total loss function is: ,in, represents the dual-scale consistency loss, represents the perturbation consistency loss, represents the generation consistency loss, Indicates supervision loss.

[0012] further, ;in, represents the mean square error loss; is an unlabeled breast source image , For Image obtained by data perturbation ; and The specific solution process is: Will Send it to the semi-supervised segmentation network of breast lesions to obtain the breast lesion segmentation confidence map ,Will Send it to the semi-supervised segmentation network of breast lesions to obtain the breast lesion segmentation confidence map ;Will and Feed into the generator , image mutual generation is performed, specifically: ; ; and Respectively and The perturbed image and the source image generated by the generator; Will Feed into the generator Generated The specific operations are: ; ; ; in, represents the downsampling operation based on maximum pooling, Represents an upsampling operation based on transposed convolution; The generation process and same.

[0013] further,

[0014] ; in, is the cross entropy loss and The average loss of loss, For labeled breast source images Corresponding label map , Represents a downsampling operation based on nearest neighbor interpolation.

[0015] further, ,in, represents the cross entropy loss.

[0016] further,

[0017] ; in, express The breast lesion segmentation prediction image obtained after inputting into the breast lesion semi-supervised segmentation network, express The breast lesion segmentation prediction image obtained after inputting into the breast lesion semi-supervised segmentation network, express and The breast lesion segmentation prediction image is obtained after input into the semi-supervised segmentation network of breast lesions and decoding feature fusion.

[0018] The beneficial effects of the method of the present invention are: 1. The present invention downsamples the source image to generate a dual-scale image of the breast image, enabling the network to learn the features of breast lesions from different scales, improving the network's ability to extract features of breast lesions with large size and morphological differences. At the same time, data perturbation is performed on the dual-scale image, and the segmentation confidence maps of the source image and the perturbation image are mutually generated by the generator, thereby improving the stability of the network's breast lesion feature learning in interference scenarios such as motion artifacts, blur and noise. 2. The present invention combines cascaded inverse residual encoding and decoding feature fusion to exploit existing features to mine more potential features, improving the network's segmentation performance with a small number of labeled samples. By inputting breast images into the final segmentation network, accurate segmentation of breast lesions can be achieved. The present invention can effectively segment breast lesions, assisting doctors in accurately locating and demarcating breast lesion areas, and improving the accuracy and efficiency of clinical diagnosis. This invention solves the problem of accurate segmentation of breast lesions with a small number of labeled clinical samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the method described in an embodiment of the present invention; Figure 2 Schematic diagram of an encoder and a decoder in a network structure according to an embodiment of the present invention; Figure 3 Schematic diagram of a dual-scale decoding feature fusion decoder in a network structure according to an embodiment of the present invention; Figure 4 Schematic diagram of the original-perturbation image generator in the network structure in an embodiment of the present invention; Figure 5 Schematic diagram of a cascaded inverse residual module and a cascaded inverse residual coding feature fusion module in a network structure in an embodiment of the present invention; Figure 6 Schematic diagram of a dual-scale decoding feature fusion module and a gated attention fusion module in a network structure according to an embodiment of the present invention; Figure 7 This is a visualization of the segmentation results after the network is trained on 30% labels on a breast ultrasound image dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following will further explain the technical solutions in the schematic diagrams of the embodiments of the present invention in detail in conjunction with the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, those skilled in the art, without making any creative efforts, can further expand the above technical solutions to application scenarios such as lesion segmentation in breast ultrasound and breast MRI images, and all of these fall within the scope of protection of the present invention.

[0021] Example 1 This embodiment provides a semi-supervised segmentation method for breast lesion images, such as Figure 1 As shown, the process of the method is: S1. Collect breast source images, including labeled breast source images and unlabeled breast source images ,right Downsampling to obtain ,right Downsampling to obtain ,right Perform data perturbation to obtain ,right Perform data perturbation to obtain ; and , and , and Three sets of dual-scale images including the original scale image size and the downsampled scale image size are respectively formed; The breast source images include breast images with lesions and breast images without lesions. The source of the breast source images may be breast ultrasound images or breast MRI images.

[0022] S2. Construct a semi-supervised segmentation network for breast lesions. The workflow in the network is as follows: S21, sending the dual-scale image to the shared encoder for feature encoding to generate a dual-scale encoded feature map; S22, sending the dual-scale coding feature map to the cascaded inverse residual coding feature fusion module across layers to perform cross-layer feature fusion to obtain a dual-scale cross-layer fusion feature map; S23, sending the dual-scale encoded feature map to two independent feature decoders, and fusing the dual-scale cross-layer fusion feature map with the decoding features of the corresponding layer through skip connections to obtain a dual-scale decoding feature map and a dual-scale segmentation image; S24, sending the dual-scale decoding feature map to the dual-scale decoding feature fusion decoder, performing step-by-step feature fusion and decoding on the dual-scale decoding feature map, and obtaining a dual-scale decoding fusion feature map and a breast lesion segmentation image; S3. Jointly train the parameters of the semi-supervised segmentation network for breast lesions based on the supervised loss of labeled images and the consistency loss of unlabeled images to obtain the final segmentation network. S4. Inputting the breast image to be segmented into the final segmentation network to achieve breast lesion segmentation.

[0023] Example 2 This embodiment further limits the embodiment 1 and further explains step S1.

[0024] The breast ultrasound source image is downsampled, specifically, by using the nearest neighbor interpolation method, with the downsampling scale being approximately 1 / 2 of the source ultrasound image size, and ensuring that its size is divisible during the encoding feature extraction process; data perturbation, specifically, by randomly changing the degree of offset of the brightness and hue of the breast ultrasound source image to generate a perturbed breast ultrasound image.

[0025] Example 3 This embodiment further limits the embodiment 1 and further explains step S2.

[0026] In this embodiment, if Figure 2 As shown in the encoder: a shared encoder is used to encode the features of the original scale image and the downsampled scale image to obtain the encoded feature map at the original scale and the encoded feature map at the downsampling scale ,in, The encoding feature map corresponds to the shared encoder The output of the layer, the shared encoder uses the Res2net network.

[0027] In this embodiment, Figure 2 The encoder and Figure 5 Cascaded inverse residual module and cascaded inverse residual coding feature fusion module in As shown: The cross-layer feature fusion in the cascaded inverse residual coding feature fusion module is specifically as follows: ; ; in, express The fused feature map obtained after cross-layer feature fusion, express The fused feature map obtained after cross-layer feature fusion, represents the channel attention mechanism, Indicates the spatial intention mechanism, Indicates channel splicing operation, represents the bilinear interpolation upsampling operation, It represents the cascaded inverse residual module, which is used to retain the underlying features while reducing network parameters and improve the feature extraction ability of the breast lesion segmentation network. The data processing flow in the cascaded inverse residual module is as follows: ; ; ; ; in, represents the input feature map, It represents the feature map after channel dimension increase, that is, the number of channels becomes twice the original number of channels to improve the feature extraction ability of the model; is a 3×3 depthwise separable convolution, is the batch normalization operation, is the activation function, represents 1×1 convolution, Represents the feature map after residual connection of the dimension-raising feature map; It is the feature map obtained by performing residual connection operation on the feature map after dimensionality reduction and the input feature map. Represents the feature map finally generated by the cascaded inverse residual module.

[0028] In this embodiment, if Figure 2 The decoder and Figure 5 As shown in the cascaded inverse residual module in

[15] : the dual-scale encoded feature map is fed into two independent feature decoders respectively. and , and and The dual-scale encoded feature map is decoded by passing the feature decoder through the jump connection to obtain the dual-scale decoded feature map and , and dual-scale segmented images and , the feature decoder uses transposed convolution for upsampling to restore the size of the image, and uses the cascaded inverse residual module to decode the features; the cross-layer fusion features are passed into the decoder through the skip connection, and the channel splicing operation is used to fuse the features with the decoded features, specifically: ; ; in, represents transposed convolution; ; .

[0029] In this embodiment, if Figure 3The dual-scale decoding feature fusion decoder and Figure 6 As shown in the dual-scale decoding feature fusion module and gated attention fusion module: the dual-scale decoding feature map and Feed into the dual-scale decoding feature fusion decoder , perform step-by-step feature fusion aggregation and decoding on the dual-scale features to obtain the dual-scale decoding fusion feature map of breast ultrasound and breast lesion segmentation images : , dual-scale decoding feature fusion decoder The specific operations performed are: ; ; ; ; ; in, GAFM represents the gated attention fusion module, GAFM The specific operations performed are: ; in, represents an element-wise multiplication operation, Represents grouped convolution, where the number of groups is half the number of channels. represents the first feature map of the input, represents the second feature map of the input, represents the output feature map; Decoding fusion features Perform step-by-step decoding to obtain breast lesion segmentation images , specifically: ; .

[0030] Example 4 This embodiment further limits the embodiment 1 and further explains step S3.

[0031] When the parameters of the semi-supervised segmentation network for breast lesions are jointly trained based on the supervised loss of labeled images and the consistency loss of unlabeled images, the total loss function is: ,in, represents the dual-scale consistency loss, represents the perturbation consistency loss, represents the generation consistency loss, Indicates supervision loss.

[0032] ;in, represents the mean square error loss; is an unlabeled breast ultrasound source image , For Image obtained by data perturbation ; and The specific solution process is: Will Send it to the semi-supervised segmentation network of breast lesions to obtain the breast lesion segmentation confidence map ,Will Send it to the semi-supervised segmentation network of breast lesions to obtain the breast lesion segmentation confidence map The difference between the segmentation confidence map and the segmentation prediction image is that the segmentation prediction image is binarized on the basis of the segmentation confidence map. After being sent to the semi-supervised segmentation network of breast lesions, the breast lesion segmentation confidence map is obtained after encoding and decoding. .

[0033] Will and Feed into the generator , image mutual generation is performed, specifically: ; ; and Respectively and The perturbed image and the source image generated by the generator; In this embodiment, Figure 4 As shown, the generator Its structure adopts a simplified Unet network, retaining only the U-shaped structure to encode and decode images.

[0034] Will Feed into the generator Generated The specific operations are: ; ; ; in, represents the downsampling operation based on maximum pooling, Represents an upsampling operation based on transposed convolution; The generation process and same.

[0035] ; in, is the cross entropy loss and The average loss of loss, Breast ultrasound source image Corresponding label map , Represents a downsampling operation based on nearest neighbor interpolation.

[0036] ,in, represents the cross entropy loss.

[0037] ; in, express The breast lesion segmentation prediction image obtained after inputting into the breast lesion semi-supervised segmentation network, express The breast lesion segmentation prediction image obtained after inputting into the breast lesion semi-supervised segmentation network, express and The breast lesion segmentation prediction image is obtained after input into the semi-supervised segmentation network of breast lesions and decoding feature fusion.

[0038] Example 5 This example further limits Example 1. In order to verify the effectiveness and superiority of this method, a method comparison was conducted. URPC, DCNet, SSNet, and MCNet semi-supervised segmentation methods were additionally selected for parallel comparison. Each method was segmented under 30% labels in the breast ultrasound dataset for quantitative comparison. The mDice, mIoU, and 、 And MAE evaluation index value, the segmentation results are as follows Figure 7 The comparison results are shown in Table 1. As can be seen from the chart, the evaluation index values ​​of the method in this embodiment on the breast ultrasound dataset with 30% labels are better than those of similar comparison methods, and the segmentation results of benign and malignant lesions are closer to the actual segmentation labels. It can be seen that the prediction results of the segmentation method in this embodiment effectively improve the segmentation performance of breast ultrasound lesions with a small number of labeled samples.

[0039] Table 1 Comparison of segmentation results of various methods under 30% labels of breast ultrasound dataset

[0040] Example 6 This embodiment further provides a semi-supervised segmentation system for breast lesion images, which is applied to the semi-supervised segmentation method for breast lesion images in Embodiments 1-4, including: A network building unit, used to construct a semi-supervised segmentation network for breast lesion images; A training unit, used to train parameters of a semi-supervised segmentation network for breast lesion images; The output unit is used to output the visualization results of breast lesion image segmentation.

[0041] The present invention also provides a semi-supervised segmentation device for breast lesion images. The above-mentioned device includes: a processor and a memory storing computer program instructions; The processor implements the semi-supervised segmentation method for breast lesion images when executing the computer program instructions.

Claims

1. A semi-supervised segmentation method for breast lesion images, characterized in that: The method comprises the following steps: S1. Collect breast source images, including labeled breast source images and unlabeled breast source images ,right Downsampling to obtain ,right Downsampling to obtain ,right Perform data perturbation to obtain ,right Perform data perturbation to obtain ; and , and , and Three sets of dual-scale images including the original scale image size and the downsampled scale image size are respectively formed; S2. Construct a semi-supervised segmentation network for breast lesions. The workflow in the network is as follows: S21, sending the dual-scale image to the shared encoder for feature encoding to generate a dual-scale encoded feature map; S22, sending the dual-scale coding feature map to the cascaded inverse residual coding feature fusion module across layers to perform cross-layer feature fusion to obtain a dual-scale cross-layer fusion feature map; S23, sending the dual-scale encoded feature map to two independent feature decoders, and fusing the dual-scale cross-layer fusion feature map with the decoding features of the corresponding layer through skip connections to obtain a dual-scale decoding feature map and a dual-scale segmentation image; S24, sending the dual-scale decoding feature map to the dual-scale decoding feature fusion decoder, performing step-by-step feature fusion and decoding on the dual-scale decoding feature map, and obtaining a dual-scale decoding fusion feature map and a breast lesion segmentation image; S3. Jointly train the parameters of the semi-supervised segmentation network for breast lesions based on the supervised loss of labeled images and the consistency loss of unlabeled images to obtain the final segmentation network. S4. Inputting the breast image to be segmented into the final segmentation network to achieve breast lesion segmentation.

2. The semi-supervised segmentation method for breast lesion images according to claim 1, wherein The shared encoder performs feature encoding on the original scale image and the downsampled scale image to obtain the encoded feature map at the original scale. and the encoded feature map at the downsampling scale ,in, The encoding feature map corresponds to the shared encoder The output of the layer, the shared encoder uses the Res2net network.

3. The semi-supervised segmentation method for breast lesion images according to claim 2, wherein: The cross-layer feature fusion in the cascaded inverse residual coding feature fusion module is specifically as follows: ； ； in, express The fused feature map obtained after cross-layer feature fusion, express The fused feature map obtained after cross-layer feature fusion, represents the channel attention mechanism, Indicates the spatial intention mechanism, Indicates channel splicing operation, represents the bilinear interpolation upsampling operation, It represents the cascaded inverse residual module. The data processing flow in the cascaded inverse residual module is as follows: ； ； ； ； in, represents the input feature map, Represents the feature map after channel dimension increase, is a 3×3 depthwise separable convolution, is the batch normalization operation, is the activation function, represents 1×1 convolution, Represents the feature map after residual connection of the dimension-raising feature map; It is the feature map obtained by performing residual connection operation on the feature map after dimensionality reduction and the input feature map. Represents the feature map finally generated by the cascaded inverse residual module.

4. The semi-supervised segmentation method for breast lesion images according to claim 3, wherein: The dual-scale encoded feature maps are fed into two independent feature decoders, and and The dual-scale encoded feature map is decoded by passing the feature decoder through the jump connection to obtain the dual-scale decoded feature map and , and dual-scale segmented images and , specifically: ； ； in, represents transposed convolution; ; 。 5. The semi-supervised segmentation method for breast lesion images according to claim 4, wherein: The dual-scale decoding feature map and The dual-scale decoding feature fusion decoder is sent to perform step-by-step feature fusion aggregation and decoding on the dual-scale features to obtain the dual-scale decoding fusion feature map. and breast lesion segmentation images : , dual-scale decoding feature fusion decoder The specific operations performed are: ； ； ； ； ； in, GAFM represents the gated attention fusion module, GAFM The specific operations performed are: ； in, represents an element-wise multiplication operation, Represents grouped convolution, where the number of groups is half the number of channels. represents the first feature map of the input, represents the second feature map of the input, represents the output feature map; Decoding fusion features Perform step-by-step decoding to obtain breast lesion segmentation images , specifically: ； 。 6. The semi-supervised segmentation method for breast lesion images according to claim 5, characterized in that: When the parameters of the semi-supervised segmentation network for breast lesions are jointly trained based on the supervised loss of labeled images and the consistency loss of unlabeled images, the total loss function is: ,in, represents the dual-scale consistency loss, represents the perturbation consistency loss, represents the generation consistency loss, Indicates supervision loss.

7. The semi-supervised segmentation method for breast lesion images according to claim 6, wherein: ;in, represents the mean square error loss; is an unlabeled breast source image , For Image obtained by data perturbation ; and The specific solution process is: Will Send it to the semi-supervised segmentation network of breast lesions to obtain the breast lesion segmentation confidence map ,Will Send it to the semi-supervised segmentation network of breast lesions to obtain the breast lesion segmentation confidence map ;Will and Feed into the generator , image mutual generation is performed, specifically: ； ； and Respectively and The perturbed image and the source image generated by the generator; Will Feed into the generator Generated The specific operations are: ； ； ； in, represents the downsampling operation based on maximum pooling, Represents an upsampling operation based on transposed convolution; The generation process and same.

8. The semi-supervised segmentation method for breast lesion images according to claim 7, wherein: ； in, is the cross entropy loss and The average loss of loss, For labeled breast source images Corresponding label map , Represents a downsampling operation based on nearest neighbor interpolation.

9. The semi-supervised segmentation method for breast lesion images according to claim 8, characterized in that: ,in, represents the cross entropy loss.

10. The semi-supervised segmentation method for breast lesion images according to claim 9, characterized in that: ； in, express The breast lesion segmentation prediction image obtained after inputting into the breast lesion semi-supervised segmentation network, express The breast lesion segmentation prediction image obtained after inputting into the breast lesion semi-supervised segmentation network, express and The breast lesion segmentation prediction image is obtained after input into the semi-supervised segmentation network of breast lesions and decoding feature fusion.

Images