Mammography image processing method, device and computer-readable storage medium

Abstract

The present application provides a method, apparatus, and computer-readable storage medium for processing mammographic images. The method comprises: encoding an image to be processed using an encoder to obtain encoding information corresponding to the image to be processed; decoding the encoding information corresponding to the image to be processed using a decoder to obtain calcification point segmentation information corresponding to the image to be processed; extracting features from the image to be processed using the first layer of the encoder or any layer of the encoder or decoder from the first layer to the next layer to obtain a feature map corresponding to the image to be processed; and inputting the feature map corresponding to the image to be processed into a fully connected layer to obtain benign and malignant classification information of the calcification clusters corresponding to the image to be processed. The present application can reduce the time and effort doctors spend on visual marking, improve the efficiency of mammographic image processing, and has a wide range of applications.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device and computer-readable storage medium for processing mammary mammography images. Background Art

[0002] Breast cancer is one of the leading causes of death among women worldwide. Mammography is a key diagnostic tool for early detection of breast cancer, offering advantages such as low cost and high sensitivity. Mammography, also known as mammographic X-rays, is currently the simplest and most reliable non-invasive test for diagnosing breast disease. It is relatively painless, easy to perform, and offers high resolution and good repeatability. The images captured allow for before-and-after comparisons, and are not restricted by age or body shape. It has become a routine screening procedure.

[0003] During mammography, breast calcifications are of great concern to doctors, including both benign and malignant calcifications. Generally speaking, benign calcifications are often found as independent, scattered points. Calcifications associated with malignant lesions are typically smaller and more densely distributed, appearing as microcalcifications distributed in clusters. The clustering of calcifications is a reference indicator for breast cancer and an important intermediate indicator of early-stage breast cancer. A calcification cluster is defined as a cluster of five or more calcifications within 1 cm, and the size of the calcification cluster is generally less than 2 cm.

[0004] In existing mammography examinations, doctors usually manually mark calcification clusters in breast mammography images with their naked eyes. Due to the small size of calcification clusters, the doctor's marking work is time-consuming and laborious. Summary of the Invention

[0005] The purpose of this application is to provide a method, device and computer-readable storage medium for processing mammography images, which can reduce the time and energy doctors spend on marking with the naked eye, improve the efficiency of mammography image processing and have a wide range of applications.

[0006] The purpose of this application is achieved by the following technical solutions:

[0007] In a first aspect, the present application provides a method for processing mammographic images, the method comprising: encoding an image to be processed using an encoder to obtain encoding information corresponding to the image to be processed; decoding the encoding information corresponding to the image to be processed using a decoder to obtain calcification point segmentation information corresponding to the image to be processed; extracting features from the image to be processed using the first layer of the encoder or any layer of the encoder or decoder from the first layer to the first layer of the encoder to obtain a feature map corresponding to the image to be processed; and inputting the feature map corresponding to the image to be processed into a fully connected layer to obtain benign and malignant classification information of calcification clusters corresponding to the image to be processed. The beneficial effects of this technical solution are that, on the one hand, the encoding information corresponding to the image to be processed is obtained using the encoder, and then decoded using the decoder to obtain corresponding calcification point segmentation information; on the other hand, the feature extraction of the image to be processed using the first layer of the encoder or any layer of the encoder or decoder from the first layer to the first layer of the encoder to obtain a corresponding feature map, and then inputting the feature map into the fully connected layer to obtain benign and malignant classification information of the calcification clusters corresponding to the image to be processed. In summary, this method can use the encoder, decoder, and fully connected layer to obtain the calcification point segmentation information and benign and malignant classification information of the calcification cluster corresponding to the image to be processed, reducing the time and energy of doctors using naked eye marking, improving the efficiency of breast mammography image processing, and has a wide range of applications.

[0008] In some optional embodiments, the method further includes: copying the weights of the fully connected layer to the first convolutional layer, inputting the feature map corresponding to the image to be processed into the first convolutional layer, outlining the calcification clusters in the image to be processed to obtain calcification cluster contour information corresponding to the image to be processed. The beneficial effect of this technical solution is that the method can not only use the encoder, decoder, and fully connected layer to obtain calcification point segmentation information and benign and malignant classification information of the calcification clusters corresponding to the image to be processed, but also copy the weights of the fully connected layer to the first convolutional layer, use the first convolutional layer to outline the calcification clusters in the image to be processed, thereby obtaining calcification cluster contour information corresponding to the image to be processed, further improving the ability to process mammography images.

[0009] In some optional embodiments, inputting the feature map corresponding to the image to be processed into a fully connected layer to obtain benign and malignant classification information of the calcification cluster corresponding to the image to be processed includes: inputting the feature map corresponding to the image to be processed into a pooling layer to obtain multiple pooled features corresponding to the image to be processed; and inputting the multiple pooled features corresponding to the image to be processed into the fully connected layer to obtain benign and malignant classification information of the calcification cluster corresponding to the image to be processed. The beneficial effect of this technical solution is that the pooling layer has the functions of dimensionality reduction, removing redundant information, and compressing features. By inputting the feature map corresponding to the image to be processed into the pooling layer to obtain multiple pooled features corresponding to the image to be processed, and then inputting the multiple pooled features into the fully connected layer, the number of parameters in the connection layer can be reduced, the calculation speed can be accelerated, and overfitting can be prevented.

[0010] In some optional embodiments, the outlining training process is as follows: copying the weights of the fully connected layer to the first convolutional layer, and inputting the feature map corresponding to the training image into the first convolutional layer, outlining the calcification clusters in the training image to obtain the calcification cluster contour information corresponding to the training image; based on the calcification cluster contour information corresponding to the training image, obtaining the calcification cluster contour information loss value corresponding to the training image; based on the calcification cluster contour information loss value corresponding to the training image, updating the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer. The beneficial effect of this technical solution is that, by copying the weights of the fully connected layer to the first convolutional layer and inputting the feature map corresponding to the training image into the first convolutional layer, the calcification clusters in the training image are outlined to obtain the calcification cluster contour information corresponding to the training image, based on the calcification cluster contour information corresponding to the training image, the corresponding calcification cluster contour information loss value is obtained, and based on the calcification cluster contour information loss value corresponding to the training image, the parameters of the first layer of the encoder or any layer of the encoder or decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer are updated, thereby enabling the encoder, decoder and fully connected layer to better fit the data, thereby improving the accuracy of outlining the calcification clusters.

[0011] In some optional embodiments, the classification training process is as follows: using the first layer of the encoder or any layer of the encoder or the decoder from the first layer to the first layer of the encoder to obtain a feature map corresponding to the training image; inputting the feature map corresponding to the training image into the fully connected layer to obtain benign and malignant classification information of the calcification cluster corresponding to the training image; based on the benign and malignant classification information of the calcification cluster corresponding to the training image, obtaining a classification loss value corresponding to the training image; based on the classification loss value corresponding to the training image, updating the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer to the first layer of the encoder and the fully connected layer; the segmentation training process is as follows: using the encoder to encode the training image to obtain encoding information corresponding to the training image; using the decoder to decode the encoding information corresponding to the training image to obtain calcification point segmentation information corresponding to the training image; based on the calcification point segmentation information corresponding to the training image, obtaining a segmentation loss value corresponding to the training image; and updating the parameters of the encoder and the decoder based on the segmentation loss value corresponding to the training image. The beneficial effect of this technical solution is that the encoder, decoder and fully connected layer can be trained using training images. On the one hand, the first layer of the encoder or any layer of the encoder or decoder from the first layer of the encoder to the first layer of the encoder is used to extract features of the training image to obtain a corresponding feature map; the feature map is input into the fully connected layer to obtain the benign and malignant classification information of the calcification cluster corresponding to the training image, and the classification loss value corresponding to the training image is obtained based on the benign and malignant classification information of the calcification cluster. Based on the classification loss value corresponding to the training image, the parameters of any layer from the first layer of the encoder to the encoder or decoder and the fully connected layer are updated. Thus, the encoder, decoder and fully connected layer can Better fit the data, thereby improving the accuracy of benign and malignant classification of calcification clusters; on the other hand, the training images can be used to train the encoder, decoder and fully connected layer, and the encoder is used to encode the training image to obtain the encoding information corresponding to the training image; the decoder is used to decode the encoding information corresponding to the training image to obtain the calcification point segmentation information corresponding to the training image, and based on the calcification point segmentation information corresponding to the training image, the segmentation loss value corresponding to the training image is obtained, and based on the segmentation loss value corresponding to the training image, the parameters of the encoder, decoder and fully connected layer are updated. As a result, the encoder, decoder and fully connected layer can better fit the data, thereby improving the accuracy of calcification point segmentation.

[0012] In some optional embodiments, the classification training process further includes: when the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is malignant, using the calcification cluster area corresponding to the calcification cluster contour information corresponding to the training image as the to-be-erased area of ​​the training image; erasing the to-be-erased area in the training image to obtain an erasure result of the training image; copying the first layer of the encoder or any layer of the encoder or decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer to obtain a copied classification network; and copying the erasure result of the training image. The result is input into the replication classification network to obtain benign or malignant classification information of the calcification cluster corresponding to the erasure result of the training image; when the benign or malignant classification information of the calcification cluster corresponding to the erasure result of the training image is malignant, the classification loss value corresponding to the erasure result is obtained as the self-adversarial loss value corresponding to the training image; based on the self-adversarial loss value corresponding to the training image, the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer are updated, so that the replication classification network can classify the erasure result of the training image as benign. The beneficial effect of this technical solution is that, when the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is malignant, the calcification cluster area corresponding to the calcification cluster contour information corresponding to the training image is used as the to-be-erased area of ​​the training image, and the to-be-erased area in the training image is erased to obtain an erased result of the training image. The erased result of the training image is input into the replication classification network to obtain benign or malignant classification information of the calcification cluster corresponding to the erased result of the training image. When the benign or malignant classification information of the calcification cluster corresponding to the erased result of the training image is malignant, the parameters of the first layer of the encoder or any layer of the encoder or decoder from the first layer to the first layer after the first layer of the encoder, and the fully connected layer can be updated based on the self-adversarial loss value corresponding to the training image, so that the replication classification network can classify the erased result of the training image as benign. As a result, the encoder, decoder, and fully connected layer can have more comprehensive coverage of the calcification cluster area when performing benign or malignant classification of the calcification cluster, thereby improving the classification accuracy.

[0013] In some optional embodiments, the classification training process further includes: when the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is benign, the area outside the calcification cluster area corresponding to the calcification cluster contour information corresponding to the training image is used as the area to be erased of the training image; erasing the area to be erased in the training image to obtain the erasing result of the training image; copying the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer to obtain a copied classification network; The erasure result is input into the replica classification network to obtain benign or malignant classification information of the calcification cluster corresponding to the erasure result of the training image; when the benign or malignant classification information of the calcification cluster corresponding to the erasure result of the training image is malignant, the classification loss value corresponding to the erasure result is obtained as the self-adversarial loss value corresponding to the training image; based on the self-adversarial loss value corresponding to the training image, the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer are updated, so that the replica classification network can classify the erasure result of the training image as benign. The beneficial effect of this technical solution is that, when the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is benign, the area outside the calcification cluster area corresponding to the calcification cluster contour information corresponding to the training image is used as the to-be-erased area of ​​the training image, and the to-be-erased area in the training image is erased to obtain an erased result of the training image. The erased result of the training image is input into the replication classification network to obtain benign or malignant classification information of the calcification cluster corresponding to the erased result of the training image. When the benign or malignant classification information of the calcification cluster corresponding to the erased result of the training image is malignant, the parameters of the first layer of the encoder, or any layer of the encoder or decoder from the first layer to the first layer of the encoder, and the fully connected layer can be updated based on the self-adversarial loss value corresponding to the training image, so that the replication classification network can classify the erased result of the training image as benign. As a result, the encoder, decoder, and fully connected layer do not cover areas that do not contain malignant calcification clusters when performing benign or malignant classification of calcification clusters, thereby improving classification accuracy.

[0014] In some optional embodiments, the classification training process, the segmentation training process, and the outline training process are performed in a joint training manner. This technical solution has the beneficial effect of further improving the accuracy of benign and malignant calcification cluster classification, the accuracy of calcification cluster outline, and the accuracy of calcification point segmentation.

[0015] In a second aspect, the present application provides a breast mammography image processing device, the device comprising: an encoding module for encoding an image to be processed using an encoder to obtain encoding information corresponding to the image to be processed; a decoding module for decoding the encoding information corresponding to the image to be processed using a decoder to obtain calcification point segmentation information corresponding to the image to be processed; an extraction module for extracting features from the image to be processed using the first layer of the encoder or any layer of the encoder or decoder from the first layer of the encoder to the first layer of the encoder, to obtain a feature map corresponding to the image to be processed; a classification module for inputting the feature map corresponding to the image to be processed into a fully connected layer to obtain benign and malignant classification information of the calcification cluster corresponding to the image to be processed.

[0016] In some optional embodiments, the device further includes: a delineation module, configured to copy the weights of the fully connected layer to the first convolutional layer, and input the feature map corresponding to the image to be processed into the first convolutional layer, to delineate the calcification clusters in the image to be processed to obtain contour information of the calcification clusters corresponding to the image to be processed.

[0017] In some optional embodiments, the classification module includes: a pooling unit, used to input the feature map corresponding to the image to be processed into the pooling layer to obtain multiple pooling features corresponding to the image to be processed; a classification information unit, used to input the multiple pooling features corresponding to the image to be processed into the fully connected layer to obtain benign and malignant classification information of the calcification cluster corresponding to the image to be processed.

[0018] In some optional embodiments, the outlining training process is as follows: copying the weights of the fully connected layer to the first convolutional layer, and inputting the feature map corresponding to the training image into the first convolutional layer, outlining the calcification clusters in the training image to obtain the calcification cluster contour information corresponding to the training image; based on the calcification cluster contour information corresponding to the training image, obtaining the calcification cluster contour information loss value corresponding to the training image; based on the calcification cluster contour information loss value corresponding to the training image, updating the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer.

[0019] In some optional embodiments, the classification training process is as follows: using the first layer of the encoder or any layer of the encoder or the decoder from the first layer to the first layer of the encoder to obtain a feature map corresponding to the training image; inputting the feature map corresponding to the training image into the fully connected layer to obtain benign and malignant classification information of the calcification cluster corresponding to the training image; based on the benign and malignant classification information of the calcification cluster corresponding to the training image, obtaining a classification loss value corresponding to the training image; based on the classification loss value corresponding to the training image, updating the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer to the first layer of the encoder and the fully connected layer; the segmentation training process is as follows: using the encoder to encode the training image to obtain encoding information corresponding to the training image; using the decoder to decode the encoding information corresponding to the training image to obtain calcification point segmentation information corresponding to the training image; based on the calcification point segmentation information corresponding to the training image, obtaining a segmentation loss value corresponding to the training image; and updating the parameters of the encoder and the decoder based on the segmentation loss value corresponding to the training image.

[0020] In some optional embodiments, the classification training process further includes: when the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is malignant, using the calcification cluster area corresponding to the calcification cluster contour information corresponding to the training image as the to-be-erased area of ​​the training image; erasing the to-be-erased area in the training image to obtain an erasure result of the training image; copying the first layer of the encoder or any layer of the encoder or decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer to obtain a copied classification network; and copying the erasure result of the training image. The result is input into the replication classification network to obtain benign or malignant classification information of the calcification cluster corresponding to the erasure result of the training image; when the benign or malignant classification information of the calcification cluster corresponding to the erasure result of the training image is malignant, the classification loss value corresponding to the erasure result is obtained as the self-adversarial loss value corresponding to the training image; based on the self-adversarial loss value corresponding to the training image, the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer are updated, so that the replication classification network can classify the erasure result of the training image as benign.

[0021] In some optional embodiments, the classification training process further includes: when the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is benign, the area outside the calcification cluster area corresponding to the calcification cluster contour information corresponding to the training image is used as the area to be erased of the training image; erasing the area to be erased in the training image to obtain the erasing result of the training image; copying the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer to obtain a copied classification network; The erasure result is input into the replica classification network to obtain benign or malignant classification information of the calcification cluster corresponding to the erasure result of the training image; when the benign or malignant classification information of the calcification cluster corresponding to the erasure result of the training image is malignant, the classification loss value corresponding to the erasure result is obtained as the self-adversarial loss value corresponding to the training image; based on the self-adversarial loss value corresponding to the training image, the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer are updated, so that the replica classification network can classify the erasure result of the training image as benign.

[0022] In some optional embodiments, multiple training processes among the classification training process, the segmentation training process and the outline training process are trained in a joint training manner.

[0023] In a third aspect, the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above-mentioned breast mammography image processing methods when executing the computer program.

[0024] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the steps of any of the above-mentioned breast mammography image processing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present application is further described below with reference to the accompanying drawings and examples.

[0026] Figure 1 1 is a flow chart of a mammography image processing method provided in an embodiment of the present application;

[0027] Figure 2 1 is a flow chart of another method for processing mammographic images provided in an embodiment of the present application;

[0028] Figure 3This is a schematic diagram of a process for obtaining benign and malignant classification information of calcification clusters provided in an embodiment of the present application;

[0029] Figure 4 This is a flow chart of a classification training process provided by an embodiment of the present application;

[0030] Figure 5 This is a flow chart of a segmentation training process provided by an embodiment of the present application;

[0031] Figure 6 This is a flow chart of a sketching training process provided by an embodiment of the present application;

[0032] Figure 7 This is a flowchart of a self-adversarial training method provided by an embodiment of the present application;

[0033] Figure 8 This is a flowchart of another self-adversarial training method provided by an embodiment of the present application;

[0034] Figure 9 is a schematic diagram of a calcification cluster detection result provided in an embodiment of the present application;

[0035] Figure 10 This is a schematic diagram of the structure of a U-shaped network provided in an embodiment of the present application;

[0036] Figure 11 Schematic diagram of the structure of a mammography image processing device provided in an embodiment of the present application;

[0037] Figure 12 1 is a schematic structural diagram of another breast mammography image processing device provided in an embodiment of the present application;

[0038] Figure 13 This is a structural diagram of a classification module provided in an embodiment of the present application;

[0039] Figure 14 This is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0040] Figure 15 This is a structural diagram of a program product for implementing a breast mammography image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] Below, the present application is further described in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0042] See also Figure 1, an embodiment of the present application provides a breast mammography image processing method, the method comprising steps S101 to S104.

[0043] Step S101: Encode an image to be processed using an encoder to obtain encoding information corresponding to the image to be processed. The image to be processed can be obtained from a mammographic image, and the image to be processed can be part of or all of the mammographic image. The resolution of the image to be processed can be, for example, 256 pixels * 256 pixels, 512 pixels * 512 pixels, or 800 pixels * 800 pixels.

[0044] Step S102: using a decoder to decode the coded information corresponding to the image to be processed to obtain calcification point segmentation information corresponding to the image to be processed.

[0045] See also Figure 10 In a specific application, the encoder and decoder can be provided with a bypass connection to increase the encoding and decoding speed. The number of bypass connections can be one or more, where one bypass connection means that the output result of one layer of the encoder is provided to one layer of the decoder.

[0046] Step S103: using the first layer of the encoder or any layer of the encoder or the decoder from the first layer to the first layer of the encoder to perform feature extraction on the image to be processed to obtain a feature map corresponding to the image to be processed.

[0047] In a specific application, step S103 may include: performing feature extraction on the image to be processed using the first layer of the encoder to the first upsampling layer of the decoder to obtain a feature map corresponding to the image to be processed.

[0048] Step S104: Input the feature map corresponding to the image to be processed into a fully connected layer to obtain benign or malignant classification information of the calcification cluster corresponding to the image to be processed. The fully connected layer can be represented by FC.

[0049] Thus, on the one hand, the encoder is used to obtain the encoded information corresponding to the image to be processed, which is then decoded by the decoder to obtain the corresponding calcification point segmentation information. On the other hand, the first layer of the encoder, or any layer of the encoder or decoder from the first layer to the first layer of the encoder, is used to extract features from the image to be processed, obtaining a corresponding feature map. The feature map is then input into the fully connected layer to obtain the benign and malignant classification information of the calcification clusters corresponding to the image to be processed. In summary, this method can utilize the encoder, decoder, and fully connected layer to obtain the calcification point segmentation information and benign and malignant classification information corresponding to the image to be processed, reducing the time and effort doctors spend on visual marking, improving the efficiency of mammography image processing, and having a wide range of applications.

[0050] See also Figure 2 In some implementations, the method may further include step S105.

[0051] Step S105: copy the weights of the fully connected layer to the first convolutional layer, and input the feature map corresponding to the image to be processed into the first convolutional layer to outline the calcification clusters in the image to be processed to obtain the contour information of the calcification clusters corresponding to the image to be processed. The first convolutional layer can be represented by Conv, and the size of the convolution kernel of the first convolutional layer can be 1*1. Weight copying, that is, weight sharing, refers to copying the parameters of the fully connected layer to the first convolutional layer, wherein the first convolutional layer can be a 1*1 convolutional layer. The number of parameters of the fully connected layer and the 1*1 convolutional layer is the same, but the dimensions of the arrangement are different.

[0052] Therefore, this method can not only use the encoder, decoder and fully connected layer to obtain the calcification point segmentation information and benign and malignant classification information of the calcification cluster corresponding to the image to be processed, but also copy the weights of the fully connected layer to the first convolutional layer, and use the first convolutional layer to outline the calcification clusters in the image to be processed, thereby obtaining the calcification cluster contour information corresponding to the image to be processed, further improving the ability of breast mammography image processing.

[0053] See also Figure 6 , the outlining training process may include steps S501 to S503.

[0054] Step S501: copying the weights of the fully connected layer to the first convolutional layer, inputting the feature map corresponding to the training image into the first convolutional layer, outlining the calcification clusters in the training image to obtain calcification cluster contour information corresponding to the training image.

[0055] Step S502: Based on the calcification cluster contour information corresponding to the training image, obtain the calcification cluster contour information loss value corresponding to the training image. The calcification cluster contour information loss value can be represented by Attention Loss.

[0056] Step S503: Based on the calcification cluster contour information loss value corresponding to the training image, update the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer.

[0057] Therefore, by copying the weights of the fully connected layer to the first convolutional layer and inputting the feature map corresponding to the training image into the first convolutional layer, the calcification clusters in the training image are outlined to obtain the calcification cluster contour information corresponding to the training image, and based on the calcification cluster contour information corresponding to the training image, the corresponding calcification cluster contour information loss value is obtained. Based on the classification loss value and the calcification cluster contour information loss value corresponding to the training image, the parameters of the first layer of the encoder or any layer of the encoder or decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer are updated. As a result, the encoder, decoder and fully connected layer can better fit the data, thereby improving the accuracy of outlining the calcification clusters.

[0058] See also Figure 3 In some implementations, step S104 may include steps S201 to S202.

[0059] Step S201: Input the feature map corresponding to the image to be processed into a pooling layer to obtain multiple pooling features corresponding to the image to be processed. The pooling layer can be represented by pooling, and the multiple pooling features can be multiple vectors.

[0060] Step S202: Input the multiple pooled features corresponding to the image to be processed into the fully connected layer to obtain benign or malignant classification information of the calcification cluster corresponding to the image to be processed. The fully connected layer can use two neurons to output the benign or malignant classification information of the calcification cluster corresponding to the image to be processed. The benign or malignant classification information of the calcification cluster indicates whether the calcification cluster in the image to be processed is benign or malignant. Benign can be represented by Benign, and malignant can be represented by Malignant.

[0061] Therefore, the pooling layer has the functions of dimensionality reduction, removing redundant information, and compressing features. By inputting the feature map corresponding to the image to be processed into the pooling layer, multiple pooling features corresponding to the image to be processed are obtained, and then the multiple pooling features are input into the fully connected layer. This can reduce the number of parameters in the connection layer, speed up the calculation, and prevent overfitting.

[0062] See also Figure 4 and Figure 5 In some implementations, the classification training process may include steps S301 to S304.

[0063] Step S301: extract features of a training image using the first layer of the encoder or any layer of the encoder or the decoder from the first layer to the first layer of the encoder to obtain a feature map corresponding to the training image.

[0064] Step S302: Inputting the feature map corresponding to the training image into the fully connected layer to obtain benign or malignant classification information of the calcification cluster corresponding to the training image.

[0065] Step S303: Based on the benign or malignant classification information of the calcification cluster corresponding to the training image, a classification loss value corresponding to the training image is obtained, wherein the classification loss value can be represented by Classification loss.

[0066] Step S304: Based on the classification loss value corresponding to the training image, update the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer.

[0067] The segmentation training process may include steps S401 to S404.

[0068] Step S401: Encode the training image using the encoder to obtain encoding information corresponding to the training image.

[0069] Step S402: using the decoder to decode the coded information corresponding to the training image to obtain calcification point segmentation information corresponding to the training image.

[0070] Step S403: Based on the calcification point segmentation information corresponding to the training image, a segmentation loss value corresponding to the training image is obtained, wherein the segmentation loss value can be represented by Segmentation loss.

[0071] Step S404: Based on the segmentation loss value corresponding to the training image, the parameters of the encoder, the decoder and the fully connected layer are updated.

[0072] Thus, the encoder, decoder, and fully connected layer can be trained using the training image. On the one hand, the first layer of the encoder or any layer of the encoder or decoder from the first layer of the encoder to the first layer of the encoder is used to extract features of the training image to obtain the corresponding feature map; the feature map is input into the fully connected layer to obtain the benign and malignant classification information of the calcification cluster corresponding to the training image, and the classification loss value corresponding to the training image is obtained based on the benign and malignant classification information of the calcification cluster. Based on the classification loss value corresponding to the training image, the parameters of the first layer of the encoder to any layer of the encoder or decoder and the fully connected layer are updated. Thus, the encoder, decoder, and fully connected layer can better simulate the image. Combined data can improve the accuracy of benign and malignant classification of calcification clusters; on the other hand, the training images can be used to train the encoder, decoder and fully connected layer, and the encoder is used to encode the training image to obtain the encoding information corresponding to the training image; the decoder is used to decode the encoding information corresponding to the training image to obtain the calcification point segmentation information corresponding to the training image, based on the calcification point segmentation information corresponding to the training image, the segmentation loss value corresponding to the training image is obtained, and based on the segmentation loss value corresponding to the training image, the parameters of the encoder, decoder and fully connected layer are updated, thereby, the encoder, decoder and fully connected layer can better fit the data, thereby improving the accuracy of calcification point segmentation.

[0073] See also Figure 7 In some implementations, the classification training process may further include steps S601 to S606.

[0074] Step S601: When the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is malignant, the calcification cluster area corresponding to the calcification cluster contour information corresponding to the training image is used as the area to be erased in the training image.

[0075] Step S602: Erasing the area to be erased in the training image to obtain an erasing result of the training image.

[0076] Step S603: Copy the first layer of the encoder or any layer of the encoder or decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer to obtain a copy classification network. The copy classification network can be represented by a self-discriminator.

[0077] Step S604: inputting the erasing result of the training image into the replication classification network to obtain benign or malignant classification information of the calcification cluster corresponding to the erasing result of the training image.

[0078] Step S605: When the benign or malignant classification information of the calcification cluster corresponding to the erasure result of the training image is malignant, a classification loss value corresponding to the erasure result is obtained as a self-adversarial loss value corresponding to the training image. The self-adversarial loss value can be represented by a self-adversarial loss.

[0079] Step S606: Based on the self-adversarial loss value corresponding to the training image, update the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer, so that the copy classification network can classify the erasure result of the training image as benign.

[0080] Therefore, when the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is malignant, the calcification cluster area corresponding to the calcification cluster contour information corresponding to the training image is used as the to-be-erased area of ​​the training image, and the to-be-erased area in the training image is erased to obtain the erased result of the training image. The erased result of the training image is input into the copy classification network to obtain the benign or malignant classification information of the calcification cluster corresponding to the erased result of the training image. When the benign or malignant classification information of the calcification cluster corresponding to the erased result of the training image is malignant, the parameters of the first layer of the encoder or any layer of the encoder or decoder from the first layer to the first layer of the encoder and the fully connected layer can be updated based on the self-adversarial loss value corresponding to the training image, so that the copy classification network can classify the erased result of the training image as benign. As a result, the encoder, decoder, and fully connected layer can have more comprehensive coverage of the calcification cluster area when performing benign or malignant classification of the calcification cluster, and the classification accuracy can be higher.

[0081] See also Figure 8 In some implementations, the classification training process may further include steps S701 to S706.

[0082] Step S701: When the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is benign, the area outside the calcification cluster area corresponding to the calcification cluster contour information corresponding to the training image is used as the area to be erased of the training image.

[0083] Step S702: Erasing the area to be erased in the training image to obtain an erasing result of the training image.

[0084] Step S703: Copy the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer after the encoder, and the fully connected layer to obtain a copied classification network.

[0085] Step S704: inputting the erasing result of the training image into the replication classification network to obtain benign or malignant classification information of the calcification cluster corresponding to the erasing result of the training image.

[0086] Step S705: When the benign or malignant classification information of the calcification cluster corresponding to the erasing result of the training image is malignant, obtaining the classification loss value corresponding to the erasing result as the self-adversarial loss value corresponding to the training image.

[0087] Step S706: Based on the self-adversarial loss value corresponding to the training image, update the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer, so that the copy classification network can classify the erasure result of the training image as benign.

[0088] Thus, when the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is benign, the area outside the calcification cluster area corresponding to the calcification cluster contour information corresponding to the training image is used as the to-be-erased area of ​​the training image. The to-be-erased area in the training image is erased to obtain an erased result of the training image. The erased result of the training image is input into the replication classification network to obtain benign or malignant classification information of the calcification cluster corresponding to the erased result of the training image. When the benign or malignant classification information of the calcification cluster corresponding to the erased result of the training image is malignant, the parameters of the first layer of the encoder, or any layer of the encoder or decoder from the first layer to the first layer of the encoder and thereafter, and the fully connected layer can be updated based on the self-adversarial loss value corresponding to the training image, so that the replication classification network can classify the erased result of the training image as benign. As a result, the encoder, decoder, and fully connected layer do not cover areas that do not contain malignant calcification clusters when performing benign or malignant classification of calcification clusters, thereby improving classification accuracy.

[0089] In some implementations, multiple training processes among the classification training process, the segmentation training process, and the outline training process may be trained in a joint training manner.

[0090] In this way, the accuracy of classifying benign and malignant calcification clusters, the accuracy of outlining calcification clusters, and the accuracy of calcification point segmentation can be further improved.

[0091] In one specific application, calcification clusters can be delineated in the image being processed. The specific process is as follows: the weights of the fully connected layer are copied to a 1x1 convolutional layer, and the feature map corresponding to the image being processed is then fed into this 1x1 convolutional layer to generate an attention map corresponding to the image being processed. The highlighted areas in the attention map represent the calcification cluster detection results, and the edges of the highlighted areas represent the outlines of the calcification clusters.

[0092] See also Figure 9 The image to be processed is shown in (a), and the area in the rectangular box is the malignant area marked by the doctor; the attention map corresponding to the image to be processed is shown in (b), and the highlighted area is the calcification cluster area; the outline of the calcification cluster outlined according to Figure (b) is shown in (c).

[0093] Existing breast calcification cluster detection algorithms mainly include two stages: segmentation of individual calcification points and analysis of the segmented calcification points to determine whether they are clustered. The calcification cluster detection results obtained using the existing breast calcification cluster detection algorithm are shown in (d).

[0094] The segmentation of a single calcification is generally divided into two categories: the first is based on traditional image processing methods, which construct a binary classifier by extracting different image features to classify all pixels in the image that belong to calcifications. The image features here are generally: Haar-like features (a common feature description operator in the field of computer vision), shape-based features, and texture-based features. The second category is segmentation methods based on deep learning. For example, the use of a U-Net-based calcification segmentation network can achieve a recall rate of 85.31% while recalling an average of one false-positive calcification point per image.

[0095] The method for analyzing and determining whether the segmented calcifications are clustered is to use post-processing methods to analyze the spatial distribution of the segmented calcifications based on the results of the segmentation of individual calcifications to determine whether they are clustered. Generally, methods used to analyze the spatial distribution of calcifications include adaptive Gaussian mixture models and expert rule-based clustering methods.

[0096] Combine Figure 9 It can be seen that the existing calcification cluster detection methods have the following problems:

[0097] 1. The segmentation of individual calcifications is poor, especially for small calcifications (less than 0.1mm in diameter). Given the complex interference of surrounding tissues and the varying shapes of calcifications, it is easy to miss calcifications. These problematic segmentation results further impact the performance of subsequent calcification clustering analysis algorithms.

[0098] 2. Algorithms for spatial analysis of calcifications are all based on manual rules or traditional manually designed models, and these algorithms have limited performance.

[0099] 3. The effect of outlining the boundaries of clusters based on the segmentation of calcification points is limited, and it is impossible to obtain high-quality results that fit the contours of calcification clusters.

[0100] In contrast, this application copies the weights of the fully connected layer to the first convolutional layer, and inputs the feature map corresponding to the image to be processed into the first convolutional layer, and uses CAM to outline the calcification clusters in the image to be processed. The outline of the calcification cluster is shown in (c), and the effect of outlining the cluster boundary is significantly better than (d).

[0101] At the same time, to enhance the impact of the attention mechanism on calcification cluster detection, different erasure strategies can be formulated based on the results of the attention map and the benign and malignant annotated information of the image to be processed. The corresponding areas in the image to be processed are erased to obtain the corresponding erasure results of the image to be processed. The replicated classification network is used as a self-discriminator to determine the benign and malignant classification information of the calcification cluster corresponding to the erasure results. The benign and malignant annotated information of the image to be processed refers to the benign or malignant areas in the image to be processed that are marked by the doctor.

[0102] The process of developing an erasure strategy is as follows:

[0103] When the benign or malignant labeling of the image to be processed indicates malignancy, the erasure rule is (1-CAM)*image, where CAM represents the highlighted area of ​​the attention map. The CAM value of the highlighted area is 1, and the CAM value of the remaining area is 0. Using this erasure rule, the calcification clusters in the image to be processed can be erased, resulting in the corresponding erasure result for the image to be processed. The erasure result for the image to be processed can be represented by Erased Image.

[0104] The erasure result is then input into the replication classification network. When the benign or malignant classification information of the calcification cluster corresponding to the erasure result is malignant, the self-adversarial loss value corresponding to the image to be processed is obtained. Based on the self-adversarial loss value, the parameters of the encoder, decoder, and fully connected layer are updated so that the replication classification network can classify the erasure result as benign. The specific expression is as follows:

[0105]

[0106] Through the above operations, the attention map can cover the calcification cluster area more comprehensively.

[0107] When the benign or malignant labeling information of the image to be processed is displayed as benign, the erasing rule is CAM*image. The image to be processed is erased using this erasing rule. The area outside the calcification cluster area in the image to be processed can be erased to obtain the erasing result corresponding to the image to be processed.

[0108] The erasure result is then input into the replication classification network. When the benign or malignant classification information of the calcification cluster corresponding to the erasure result is malignant, the self-adversarial loss value corresponding to the image to be processed is obtained. Based on the self-adversarial loss value, the parameters of the encoder, decoder, and fully connected layer are updated so that the replication classification network can classify the erasure result as benign.

[0109] The specific expressions are as follows:

[0110]

[0111] Generally speaking, for some images with benign or malignant calcification cluster classification information, there are some features similar to malignant calcification clusters, such as high-density glandular areas. The attention map tends to highlight these areas, resulting in deviations in the detection results. Through the above operation, the false positive areas covered by the attention map can be reduced, thereby improving the calcification cluster detection effect.

[0112] See also Figure 10 The present application also provides a multi-task U-net that can simultaneously perform calcification point segmentation, benign and malignant classification of calcification clusters, and localization and delineation of calcification clusters in an end-to-end, one-step process. The multi-tasks include calcification point segmentation, benign and malignant classification of calcification clusters, and localization and delineation of calcification clusters. The multi-task U-net can be represented by a Multi-task U-Net, the calcification point segmentation task can be represented by a Segmentation Task, the benign and malignant classification of calcification clusters can be represented by a Classification Task, and the localization and delineation of calcification clusters can be represented by a CAM Task.

[0113] The U-shaped network may include an encoder, a decoder, a pooling layer, and a fully connected layer. The above-mentioned breast mammography image processing method can be implemented using the U-shaped network.

[0114] See also Figure 11 The embodiment of the present application also provides a breast mammography image processing device, the specific implementation method of which is consistent with the implementation method and technical effects recorded in the embodiment of the above-mentioned breast mammography image processing method, and some contents will not be repeated here.

[0115] The apparatus includes: an encoding module 101 for encoding an image to be processed using an encoder to obtain encoding information corresponding to the image to be processed; a decoding module 102 for decoding the encoding information corresponding to the image to be processed using a decoder to obtain calcification point segmentation information corresponding to the image to be processed; an extraction module 103 for extracting features from the image to be processed using the first layer of the encoder or any layer of the encoder or decoder from the first layer to the next layer of the encoder to obtain a feature map corresponding to the image to be processed; and a classification module 104 for inputting the feature map corresponding to the image to be processed into a fully connected layer to obtain benign or malignant classification information of the calcification cluster corresponding to the image to be processed.

[0116] See also Figure 12 In some embodiments, the device may further include: a delineation module 105, configured to copy the weights of the fully connected layer to the first convolutional layer, and input the feature map corresponding to the image to be processed into the first convolutional layer, to delineate the calcification clusters in the image to be processed to obtain contour information of the calcification clusters corresponding to the image to be processed.

[0117] See also Figure 13 In some embodiments, the classification module 104 may include: a pooling unit 201, used to input the feature map corresponding to the image to be processed into the pooling layer to obtain multiple pooling features corresponding to the image to be processed; a classification information unit 202, used to input the multiple pooling features corresponding to the image to be processed into the fully connected layer to obtain benign and malignant classification information of the calcification cluster corresponding to the image to be processed.

[0118] In some embodiments, the outlining training process is as follows: copying the weights of the fully connected layer to the first convolutional layer, and inputting the feature map corresponding to the training image into the first convolutional layer, outlining the calcification clusters in the training image to obtain the calcification cluster contour information corresponding to the training image; based on the calcification cluster contour information corresponding to the training image, obtaining the calcification cluster contour information loss value corresponding to the training image; based on the calcification cluster contour information loss value corresponding to the training image, updating the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer.

[0119] In some embodiments, the classification training process is as follows: using the first layer of the encoder or any layer of the encoder or the decoder from the first layer to the first layer of the encoder to obtain a feature map corresponding to the training image; inputting the feature map corresponding to the training image into the fully connected layer to obtain benign and malignant classification information of the calcification cluster corresponding to the training image; based on the benign and malignant classification information of the calcification cluster corresponding to the training image, obtaining a classification loss value corresponding to the training image; based on the classification loss value corresponding to the training image, updating the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer to the first layer of the encoder and the fully connected layer; the segmentation training process is as follows: using the encoder to encode the training image to obtain encoding information corresponding to the training image; using the decoder to decode the encoding information corresponding to the training image to obtain calcification point segmentation information corresponding to the training image; based on the calcification point segmentation information corresponding to the training image, obtaining a segmentation loss value corresponding to the training image; based on the segmentation loss value corresponding to the training image, updating the parameters of the encoder and the decoder.

[0120] In some embodiments, the classification training process may further include: when the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is malignant, using the calcification cluster area corresponding to the calcification cluster contour information corresponding to the training image as the to-be-erased area of ​​the training image; erasing the to-be-erased area in the training image to obtain an erasure result of the training image; copying the first layer of the encoder or any layer of the encoder or decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer to obtain a copied classification network; and copying the erasure result of the training image. The result is input into the replication classification network to obtain benign or malignant classification information of the calcification cluster corresponding to the erasure result of the training image; when the benign or malignant classification information of the calcification cluster corresponding to the erasure result of the training image is malignant, the classification loss value corresponding to the erasure result is obtained as the self-adversarial loss value corresponding to the training image; based on the self-adversarial loss value corresponding to the training image, the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer are updated, so that the replication classification network can classify the erasure result of the training image as benign.

[0121] In some embodiments, the classification training process may further include: when the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is benign, using the area outside the calcification cluster area corresponding to the calcification cluster contour information corresponding to the training image as the area to be erased of the training image; erasing the area to be erased in the training image to obtain the erasing result of the training image; copying the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer to obtain a copied classification network; The erasure result is input into the replica classification network to obtain benign or malignant classification information of the calcification cluster corresponding to the erasure result of the training image; when the benign or malignant classification information of the calcification cluster corresponding to the erasure result of the training image is malignant, the classification loss value corresponding to the erasure result is obtained as the self-adversarial loss value corresponding to the training image; based on the self-adversarial loss value corresponding to the training image, the parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer are updated, so that the replica classification network can classify the erasure result of the training image as benign.

[0122] In some implementations, multiple training processes among the classification training process, the segmentation training process, and the outline training process may be trained in a joint training manner.

[0123] See also Figure 14 , an embodiment of the present application further provides an electronic device 200, which includes at least one memory 210, at least one processor 220, and a bus 230 connecting different platform systems.

[0124] The memory 210 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 211 and / or a cache memory 212 , and may further include a read-only memory (ROM) 213 .

[0125] Among them, the memory 210 also stores a computer program, which can be executed by the processor 220, so that the processor 220 performs the steps of the breast mammography image processing method in the embodiment of the present application. Its specific implementation method is consistent with the implementation method and the technical effect achieved in the embodiment of the above-mentioned breast mammography image processing method, and some contents will not be repeated here.

[0126] Memory 210 may also include a utility 214 having at least one program module 215, such program modules 215 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0127] Accordingly, the processor 220 may execute the aforementioned computer program and the utility 214 .

[0128] The bus 230 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0129] The electronic device 200 may also communicate with one or more external devices 240, such as a keyboard, pointing device, Bluetooth device, etc., and may also communicate with one or more devices capable of interacting with the electronic device 200, and / or any device that enables the electronic device 200 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication may be performed via an input / output interface 250. Furthermore, the electronic device 200 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 260. The network adapter 260 may communicate with other modules of the electronic device 200 via the bus 230. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the electronic device 200, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0130] The embodiment of the present application also provides a computer-readable storage medium for storing a computer program. When the computer program is executed, the steps of the breast mammography image processing method in the embodiment of the present application are implemented. The specific implementation method is consistent with the implementation method and technical effects described in the embodiment of the above-mentioned breast mammography image processing method, and some contents are not repeated here.

[0131] Figure 15The program product 300 provided in this embodiment for implementing the above-described mammography image processing method is shown. It can be implemented in a portable compact disk read-only memory (CD-ROM) and include program code, and can be executed on a terminal device, such as a personal computer. However, the program product 300 of the present invention is not limited thereto. In this application, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The program product 300 can utilize any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] A computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, or any suitable combination thereof. The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. Where a remote computing device is involved, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0133] This application is explained from the perspectives of purpose of use, effectiveness, progress and novelty, and has complied with the functional enhancement and use requirements emphasized by the Patent Law. The above description and drawings of this application are only preferred embodiments of this application and are not intended to limit this application. Therefore, all structures, devices, features, etc. that are similar or identical to those of this application, that is, all equivalent replacements or modifications made in accordance with the scope of the patent application of this application, should fall within the scope of protection of the patent application of this application.

Claims

1. A method for processing mammographic images, characterized in that: The method comprises: Encoding the image to be processed using an encoder to obtain encoding information corresponding to the image to be processed; Decoding the encoded information corresponding to the image to be processed using a decoder to obtain calcification point segmentation information corresponding to the image to be processed; Performing feature extraction on the image to be processed using the first layer of the encoder to the first upsampling layer of the decoder to obtain a feature map corresponding to the image to be processed; Inputting the feature map corresponding to the image to be processed into a fully connected layer to obtain benign and malignant classification information of the calcification cluster corresponding to the image to be processed; Copying the weights of the fully connected layer to the first convolutional layer, inputting the feature map corresponding to the image to be processed into the first convolutional layer, outlining the calcification clusters in the image to be processed to obtain calcification cluster contour information corresponding to the image to be processed; Among them, multiple training processes in the classification training process, the segmentation training process and the outline training process are trained in a joint training manner; During the classification training process, self-adversarial training is also included: When the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is malignant, the calcification cluster area corresponding to the calcification cluster contour information is erased using the erasing rule (1-CAM) × training image to obtain an erasing result of the training image, where CAM is the highlighted area of ​​the attention map generated based on the calcification cluster contour information, the CAM value of the calcification cluster area is 1, and the rest of the area is 0; Copying the first layer of the encoder to the first upsampling layer of the decoder and the fully connected layer to obtain a copied classification network; Inputting the erasing result of the training image into the replication classification network to obtain benign or malignant classification information of the calcification cluster corresponding to the erasing result; When the benign or malignant classification information of the calcification cluster corresponding to the erasure result is malignant, obtaining the classification loss value corresponding to the erasure result as the self-combat loss value; Based on the self-adversarial loss value, updating parameters from the first layer of the encoder to the first upsampling layer of the decoder and the fully connected layer so that the copy classification network classifies the erasure result as benign; When the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is benign, an erasing rule CAM×training image is used to erase the area outside the calcification cluster area corresponding to the calcification cluster contour information to obtain an erasing result of the training image; The same steps of copying, inputting, obtaining loss values, and updating parameters are performed so that the copy classification network classifies the erasure result as benign.

2. The method for processing mammographic images according to claim 1, wherein: The training process is outlined as follows: Copying the weights of the fully connected layer to the first convolutional layer, inputting the feature map corresponding to the training image into the first convolutional layer, outlining the calcification clusters in the training image to obtain calcification cluster contour information corresponding to the training image; Based on the calcification cluster contour information corresponding to the training image, obtaining a calcification cluster contour information loss value corresponding to the training image; Based on the calcification cluster contour information loss value corresponding to the training image, parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer are updated.

3. The method for processing mammographic images according to claim 1, wherein: The classification training process is as follows: performing feature extraction on a training image using the first layer of the encoder or any layer of the encoder from the first layer to the first layer of the encoder or the decoder to obtain a feature map corresponding to the training image; Inputting the feature map corresponding to the training image into the fully connected layer to obtain benign and malignant classification information of the calcification cluster corresponding to the training image; Based on the benign and malignant classification information of the calcification cluster corresponding to the training image, obtaining a classification loss value corresponding to the training image; Based on the classification loss value corresponding to the training image, updating parameters of the first layer of the encoder or any layer of the encoder or the decoder from the first layer of the encoder to the first layer of the encoder and the fully connected layer; The segmentation training process is as follows: Encoding the training image using the encoder to obtain encoding information corresponding to the training image; Decoding the encoded information corresponding to the training image using the decoder to obtain calcification point segmentation information corresponding to the training image; Based on the calcification point segmentation information corresponding to the training image, obtaining a segmentation loss value corresponding to the training image; Based on the segmentation loss value corresponding to the training image, parameters of the encoder and the decoder are updated.

4. A mammography image processing device, characterized in that: The device comprises: An encoding module, configured to encode the image to be processed using an encoder to obtain encoding information corresponding to the image to be processed; A decoding module, configured to decode the coded information corresponding to the image to be processed using a decoder to obtain calcification point segmentation information corresponding to the image to be processed; an extraction module, performing feature extraction on the image to be processed using the first layer of the encoder to the first upsampling layer of the decoder to obtain a feature map corresponding to the image to be processed; A classification module, configured to input the feature map corresponding to the image to be processed into a fully connected layer to obtain benign and malignant classification information of the calcification cluster corresponding to the image to be processed; a delineation module, configured to copy the weights of the fully connected layer to the first convolutional layer, input the feature map corresponding to the image to be processed into the first convolutional layer, and delineate the calcification clusters in the image to be processed to obtain contour information of the calcification clusters corresponding to the image to be processed; Among them, multiple training processes in the classification training process, the segmentation training process and the outline training process are trained in a joint training manner; During the classification training process, self-adversarial training is also included: When the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is malignant, the calcification cluster area corresponding to the calcification cluster contour information is erased using the erasing rule (1-CAM) × training image to obtain an erasing result of the training image, where CAM is the highlighted area of ​​the attention map generated based on the calcification cluster contour information, the CAM value of the calcification cluster area is 1, and the rest of the area is 0; Copying the first layer of the encoder to the first upsampling layer of the decoder and the fully connected layer to obtain a copied classification network; Inputting the erasing result of the training image into the replication classification network to obtain benign or malignant classification information of the calcification cluster corresponding to the erasing result; When the benign or malignant classification information of the calcification cluster corresponding to the erasure result is malignant, obtaining the classification loss value corresponding to the erasure result as the self-combat loss value; Based on the self-adversarial loss value, updating parameters from the first layer of the encoder to the first upsampling layer of the decoder and the fully connected layer so that the copy classification network classifies the erasure result as benign; When the benign or malignant classification information of the calcification cluster corresponding to the training image indicates that the calcification cluster in the training image is benign, an erasing rule CAM×training image is used to erase the area outside the calcification cluster area corresponding to the calcification cluster contour information to obtain an erasing result of the training image; The same steps of copying, inputting, obtaining loss values, and updating parameters are performed so that the copy classification network classifies the erasure result as benign.

5. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the breast mammography image processing method according to any one of claims 1 to 3 are implemented.

Images