A breast CEM image lesion recognition system and device based on deep learning

Abstract

The application discloses a breast CEM image lesion recognition system and equipment based on deep learning, and relates to the field of medical image processing. By acquiring a low-energy image and a subtraction image of an original CEM image and performing pretreatment, lesions in a breast region of the image are labeled, and two ResNet50 networks are used to extract preliminary extraction features of the low-energy image and the subtraction image respectively. Double-mode auxiliary feature fusion is performed to obtain a multi-scale pyramid feature layer. A detection head with an attention mechanism is connected after each pyramid feature layer to finally position and classify breast lesions. The application realizes rapid detection, positioning and classification of breast lesions based on a multi-modal CEM image based on a deep learning method, and improves breast lesion recognition efficiency and precision.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a deep learning-based system and device for identifying lesions in breast CEM images. Background Technology

[0002] Contrast-Enhanced Mammography (CEM) is a relatively new imaging technique that combines mammography with intravenous injection of iodine contrast agents. First, the contrast agent is injected intravenously. Utilizing the K-edge effect of the iodine contrast agent (significant differences in attenuation characteristics around 33.2 keV), high-energy (above 33.2 keV) and low-energy (below 33.2 keV) exposures are performed to obtain high-energy and low-energy images. Then, the difference in X-ray attenuation between iodine and breast tissue at these two energy levels is used to suppress background breast tissue. Specific weighted reconstruction (subtraction) is performed on the low-energy and high-energy images, ultimately yielding low-energy and subtraction images of the bilateral breast in cephalothorax and oblique views. The low-energy images possess morphological details similar to those of mammographic images, while the subtraction images remove surrounding normal overlapping glandular tissue, making lesions clearer. Therefore, CEM images provide both tumor morphology and perfusion information.

[0003] Currently, many target recognition methods based on natural images exist, capable of accurately and quickly locating and identifying target objects. However, compared to natural images, the proportion of key information in breast imaging is small, and lesions and non-lesions often have similar appearances, requiring the network to focus on more fine-grained category features. Clearly, directly transferring recognition algorithms based on natural images to breast medical images is unreasonable. Furthermore, the multimodal information in current CEM images lacks effective utilization. Therefore, there is an urgent need in this field to develop a method specifically for the rapid detection and classification of breast lesions simultaneously in bimodal CEM images. Summary of the Invention

[0004] To address the problems mentioned in the background art, the present invention provides a breast lesion recognition system and device based on deep learning for CEM images, so as to realize rapid localization, detection and classification of breast lesions based on CEM images, and improve the accuracy of breast lesion recognition.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] On one hand, the present invention provides a deep learning-based breast CEM image lesion recognition system, comprising:

[0007] The image acquisition and preprocessing module is used to acquire CEM images and perform image preprocessing to obtain preprocessed sample images; the CEM images include low-energy images and subtraction images of bilateral breast head-to-tail and internal-external oblique views; the preprocessed sample images include preprocessed low-energy images and subtraction images.

[0008] The preliminary feature extraction module is used to input the preprocessed low-energy image and subtraction image into two identical ResNet50 deep learning branch networks for preliminary feature extraction, so as to obtain preliminary extracted features at different scales.

[0009] The multimodal feature fusion module is used to fuse the preliminary extracted features at different scales using a multimodal feature fusion network to obtain dual-modal fused features at different scales.

[0010] The multi-scale pyramid feature fusion module is used to input dual-modal fusion features of different scales into the multi-scale pyramid feature fusion network to obtain a multi-scale pyramid feature layer.

[0011] The detection head lesion identification module is used to input the multi-scale pyramid feature layer into the corresponding detection head and output the breast lesion identification result; the detection head includes a regression sub-network and a classification sub-network; the breast lesion identification result includes the lesion location and category.

[0012] Optionally, the image collection and preprocessing module specifically includes:

[0013] The labeling unit is used to mark the region of interest of the lesion in the acquired CEM image to generate the location label of the lesion, and at the same time mark the lesion as benign or malignant as a classification label;

[0014] The binarization unit is used to binarize the labeled CEM image to obtain a binarized image.

[0015] The cropping and scaling unit is used to crop out the smallest bounding rectangle region containing the breast foreground region from the binarized image and scale it to a uniform size to obtain the scaled image.

[0016] The normalization unit is used to normalize the scaled image to obtain a normalized image;

[0017] The image overlay unit is used to overlay three identical single-channel normalized images to obtain a preprocessed sample image.

[0018] Optionally, the preliminary feature extraction module specifically includes:

[0019] The preliminary feature extraction unit is used to input the preprocessed low-energy map and subtraction map into two identical ResNet50 deep learning branch networks. First, they pass through a 3×3 convolutional layer and a max pooling layer to obtain a feature map of size 256×256. Then, they pass through three residual modules with the same structure to obtain feature maps of size 128×128, 64×64 and 32×32 respectively, which together constitute four different scales of preliminary extracted features.

[0020] Optionally, the multimodal feature fusion module specifically includes:

[0021] The multimodal feature fusion unit is used to process feature maps of 128×128, 64×64, and 32×32 sizes. First, the feature maps corresponding to the low-energy map and the subtraction map at the same scale are processed separately using the same dual-convolutional attention mechanism to obtain dual-convolutional attention features. Then, the dual-convolutional attention features of the subtraction map are inverted using a 1-sigmoid function to obtain their inverse features. Next, the inverse features of the subtraction map and the dual-convolutional attention features of the low-energy map are multiplied element-wise to obtain new synthetic auxiliary features. Finally, the feature map corresponding to the original subtraction map is added element-wise to the synthetic auxiliary features to obtain dual-modal fusion features of different scales.

[0022] Optionally, the multi-scale pyramid feature fusion module specifically includes:

[0023] The multi-scale pyramid feature fusion unit is used to generate feature maps of the same size as the next level fusion feature from top to bottom by using a neighbor interpolation 2x upsampling algorithm for dual-modal fusion features of different scales. Then, a horizontal addition fusion operation is performed. After fusion, a 3*3 convolution kernel is used to convolve each fusion result to eliminate the aliasing effect of upsampling. Finally, multi-scale pyramid features of size 32×32, 64×64 and 128×128 are obtained from top to bottom.

[0024] Optionally, the detection head lesion identification module specifically includes:

[0025] The lesion location identification unit is used to input multi-scale pyramid features into the corresponding regression sub-network and output the lesion location.

[0026] The lesion category identification unit is used to input multi-scale pyramid features into the corresponding classification subnetwork and output the lesion category.

[0027] On the other hand, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a deep learning-based method for identifying lesions in breast CEM images. The deep learning-based method for identifying lesions in breast CEM images includes:

[0028] CEM images are acquired and preprocessed to obtain preprocessed sample images; the CEM images include low-energy images and subtraction images of bilateral breast head-to-tail and internal-external oblique views; the preprocessed sample images include preprocessed low-energy images and subtraction images.

[0029] The preprocessed low-energy image and subtraction image are respectively input into two identical ResNet50 deep learning branch networks for preliminary feature extraction to obtain preliminary extracted features at different scales;

[0030] A multimodal feature fusion network is used to fuse the preliminary extracted features at different scales to obtain bimodal fused features at different scales.

[0031] The dual-modal fusion features at different scales are input into the multi-scale pyramid feature fusion network to obtain the multi-scale pyramid feature layer.

[0032] The multi-scale pyramid feature layer is input into the corresponding detection head, and the breast lesion identification result is output; the detection head includes a regression sub-network and a classification sub-network; the breast lesion identification result includes the lesion location and category.

[0033] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0034] This invention provides a deep learning-based system and device for breast lesion recognition in CEM images. It acquires and preprocesses low-energy and subtraction images of the original CEM image, marking lesions within the breast region. Preliminary features are extracted from the low-energy and subtraction images using two ResNet50 networks. Dual-modal auxiliary feature fusion is then performed to obtain multi-scale pyramid feature layers. Each pyramid feature layer is followed by a detection head with an attention mechanism for final localization and classification of breast lesions. This invention, based on deep learning, achieves rapid detection, localization, and classification of breast lesions from multi-modal CEM images, improving the efficiency and accuracy of breast lesion recognition. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of the structure of a breast CEM image lesion recognition system based on deep learning according to the present invention;

[0037] Figure 2 This is a schematic diagram of the image processing process of the deep learning-based breast CEM image lesion recognition system of the present invention;

[0038] Figure 3 This is a schematic diagram of the feature processing process of the multimodal feature fusion network of the present invention;

[0039] Figure 4 This is a schematic diagram of the feature processing process of the dual convolutional attention mechanism of the present invention;

[0040] Figure 5 This is a schematic diagram of the feature processing of the channel attention operation in the dual convolutional attention mechanism of the present invention;

[0041] Figure 6 This is a schematic diagram of the feature processing of spatial attention operation in the dual convolutional attention mechanism of the present invention;

[0042] Figure 7 This is a schematic diagram of the feature processing process of the detection head of the present invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Typically, lesion outlines are more clearly displayed in subtraction images, a feature that helps the network better locate lesions; while low-energy images often provide richer information about breast tissue and calcifications, which is more helpful for lesion classification. However, no research has yet proposed a method for CEM images that can fully learn the complementary information of the two modalities. The purpose of this invention is to provide a deep learning-based breast CEM image lesion recognition system and device to achieve rapid localization, detection, and classification of breast lesions based on CEM images, thereby improving the accuracy of breast lesion recognition.

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Figure 1 This is a schematic diagram of the structure of a deep learning-based breast CEM image lesion recognition system according to the present invention. See also Figure 1 A deep learning-based breast CEM image lesion recognition system includes:

[0047] The image acquisition and preprocessing module 101 is used to acquire CEM images and perform image preprocessing to obtain preprocessed sample images.

[0048] This invention collected breast images from 1790 patients who underwent contrast-enhanced mammography at Yantai Yuhuangding Hospital between 2019 and 2022. Each CEM image included low-energy images and subtraction images in bilateral cephalothorax and endoscopic oblique views, totaling four images. To further expand the dataset, this invention used two views (bilateral cephalothorax and endoscopic oblique views) from each patient as two samples, each sample containing one low-energy image and one subtraction image. That is, in this invention, the CEM images include low-energy images and subtraction images in bilateral cephalothorax and endoscopic oblique views; the preprocessed sample images also correspondingly include preprocessed low-energy images and subtraction images.

[0049] Specifically, the image collection and preprocessing module includes:

[0050] The labeling unit is used to mark the region of interest of the lesion in the acquired CEM image to generate the location label of the lesion, and at the same time, the marked lesions are labeled as benign or malignant as classification labels.

[0051] Specifically, the graphic annotation tool LabelMe is used to annotate the regions of interest (ROIs) of lesions using rectangular bounding boxes. LabelMe then automatically obtains the coordinates of the top-left vertex (Xmin, Ymin) and bottom-right vertex (Xmax, Ymax) of the bounding box by locating its position. These coordinates serve as the location label information for the lesions. Simultaneously, the delineated lesions are labeled as benign or malignant to serve as classification labels. The location labels are used for training the model for the detection task, while the classification labels are used for training the classification task. Training labels for the network are created because deep learning networks require real-world labels as constraints to guide the training model, making the prediction results increasingly closer to the true values. Since most lesions are clearer on subtraction angiography, lesion delineation is primarily performed on subtraction angiography. Lesions that are not clearly visible on subtraction angiography are delineated with the assistance of low-energy angiography. The tumor coordinates and classification label information from the low-energy angiography and subtraction angiography of the same sample are shared.

[0052] The binarization unit is used to binarize the labeled CEM image to obtain a binarized image.

[0053] For the original low-energy image and the subtraction image, an optimal threshold is determined to divide the pixel values ​​of the original image into two groups and maximize the inter-class variance of the pixel values ​​of these two groups. Then, the value of pixels greater than the threshold is set to 255 (breast region), and the value of pixels less than the threshold is set to 0 (background region), thereby binarizing the original image and separating the breast region from the background region, thus obtaining the binarized image corresponding to the original low-energy image or the subtraction image.

[0054] The cropping and scaling unit is used to crop the smallest bounding rectangle region containing the breast foreground region from the binarized image and scale it to a uniform size to obtain the scaled image.

[0055] Specifically, the nonzero method integrated in the NumPy package in Python is used to identify nonzero pixels in the binarized image and return their coordinates. Then, based on the returned minimum and maximum coordinates, a minimum bounding rectangle region that can just contain the foreground region of the breast is determined, and this region is cut out from the original image and scaled to 512×1024 to obtain the scaled image.

[0056] The normalization unit is used to normalize the scaled image to obtain a normalized image.

[0057] Specifically, the cropped and scaled image is processed using the formula... Normalization is performed to obtain a normalized image. Where x... i Let x be the pixel value of the image, and min(x) and max(x) be the maximum and minimum values ​​of the image pixels, respectively.

[0058] The image overlay unit is used to overlay three identical single-channel normalized images to obtain a preprocessed sample image.

[0059] Both the low-energy image and the subtraction image of a CEM are single-channel images. By superimposing a low-energy image-low-energy image-low-energy image, a three-channel low-energy image can be formed, and the same applies to the subtraction image. Therefore, this invention superimposes three identical single-channel normalized images corresponding to the low-energy image or subtraction image to obtain a three-channel RGB image, which serves as the preprocessed low-energy image or subtraction image.

[0060] When performing the above cropping, normalization, and scaling on the image, the coordinates of the original corresponding lesion labels are also calculated and transformed in the same way as the pixel information of the original image, to ensure that the coordinates of the original lesion location can still correspond to the preprocessed image after the image is preprocessed.

[0061] The preliminary feature extraction module 102 is used to input the preprocessed low-energy image and subtraction image into two identical ResNet50 deep learning branch networks for preliminary feature extraction, so as to obtain preliminary extracted features at different scales.

[0062] Specifically, the preprocessed low-energy image and subtraction image are input into two identical ResNet50 deep learning branches for initial feature extraction. First, a 3×3 convolutional layer and a max-pooling layer are used to obtain a first feature layer of size 256×256. Then, three identical residual modules are used to obtain feature layers of sizes 128×128, 64×64, and 32×32, respectively. The final result is as follows: Figure 2 The left half shows the feature maps of the low-energy map and the subtraction map at four different scales.

[0063] The multimodal feature fusion module 103 is used to fuse the preliminary extracted features at different scales using a multimodal feature fusion network to obtain bimodal fused features at different scales.

[0064] Only by accurately locating the lesion can more precise classification be performed. Compared to low-energy images, subtraction images show the lesion outline more clearly and are more suitable for detection tasks. Low-energy images often provide richer information about breast tissue and calcifications, which is more helpful for lesion classification. Therefore, this invention develops a multimodal feature fusion network that uses subtraction images as the main fusion branch and low-energy images as an auxiliary fusion branch to fully utilize CEM image information to achieve more accurate lesion detection and classification.

[0065] Figure 3 The feature processing procedure of the multimodal feature fusion network of this invention is illustrated. Due to the large size of the bottom-level features, the fusion calculation would generate excessive computational cost; therefore, only the top three layers of features (128×128, 64×64, and 32×32) are fused. For the initial feature extraction of low-energy images and subtraction images at the same scale, the same double convolutional attention mechanism is first applied to obtain double convolutional attention features. Then, the double convolutional attention feature vector of the subtraction image is processed... The activation function is mapped to a value between 0 and 1. Then, the value is subtracted from 1, which is to invert the feature by using 1-sigmoid (double convolutional attention) to obtain its inverse feature. Then, the double convolutional attention feature of the low-energy image is multiplied element-wise to obtain a new synthetic auxiliary feature. Finally, the initial extracted features of the original subtraction image are added element-wise to the synthetic auxiliary feature to obtain the dual-modality fusion feature.

[0066] Figure 4The feature processing procedure of the dual convolutional attention mechanism of this invention is illustrated. This mechanism adaptively integrates local features and their global dependencies, capturing rich contextual dependencies based on a self-constraint mechanism. For example... Figure 4 As shown, before the low-energy map and subtraction map features of each layer are fused, they are first subjected to a double convolutional attention operation, which superimposes the convolutional attention idea on both spatial and channel dimensions. Given an intermediate feature map, attention maps are inferred sequentially along the two independent dimensions (channel and space). Then, the attention map is multiplied with the input feature map and another attention operation is performed. Finally, the features are further integrated by addition to obtain the final double convolutional attention features.

[0067] Figure 5 The feature processing procedure of channel attention operation in the dual convolutional attention mechanism of this invention is illustrated. For example... Figure 5 As shown, the input feature map is first reshaped twice to obtain the features. Figure 1 and 2 Then, the input feature maps were reshaped to the same scale and channels as the input maps and then transposed to obtain the features. Figure 3 Next, the features Figure 3 and the features after reshaping one of the branches Figure 2 The feature vector is multiplied and then passed through the softmax function. The resulting feature vector is multiplied by feature 1 and then multiplied by the scaling factor B. It is then reshaped back to its original shape (where B is initialized to 0 and gradually learned to be assigned a larger weight). Finally, it is added element-wise with the input feature to obtain the final feature with channel attention weights, i.e., the channel attention feature.

[0068] Figure 6 The feature processing procedure for spatial attention operation in the dual convolutional attention mechanism of this invention is illustrated. For example... Figure 6 As shown, the input feature map first passes through three convolutional layers to obtain three feature maps A, B, and C. Then, each feature map undergoes the same reshaping process. Next, the transpose of the reshaped C is multiplied by the reshaped B, and then passed through a softmax layer. The resulting feature vector is then multiplied by the reshaped feature map A, multiplied by a scaling factor, and reshaped back to its original shape. Finally, this is added to the input feature vector to obtain the final output feature. The value at each position of this output feature is a weighted sum of the values ​​at each position of the original features. The scaling factor is initialized to 0 and its weights are gradually adjusted as the learning process progresses.

[0069] The multi-scale pyramid feature fusion module 104 is used to input dual-modal fusion features of different scales into the multi-scale pyramid feature fusion network to obtain a multi-scale pyramid feature layer.

[0070] Specifically, dual-modal fusion features of different scales are input into a multi-scale pyramid feature fusion network. The fused features of each layer are generated from top to bottom using a neighbor interpolation 2x upsampling algorithm to produce feature maps of the same size as the next level of fusion features. Then, a horizontal addition fusion operation is performed. After fusion, a 3*3 convolution kernel is used to convolve each fusion result to eliminate the aliasing effect of upsampling. Finally, multi-scale pyramid features of 32×32, 64×64 and 128×128 from top to bottom are obtained.

[0071] The detection head lesion identification module 105 is used to input the multi-scale pyramid feature layer into the corresponding detection head and output the breast lesion identification result.

[0072] Figure 7 The feature processing procedure of the detection head of the present invention is shown. For example... Figure 2 As shown, each pyramid feature layer is followed by a detector head, and the detector heads after the three pyramid feature layers share the same weight.

[0073] like Figure 7 As shown, each detection head contains a regression sub-network and a classification sub-network. In the regression sub-network, the input pyramid features are first assigned different spatial attention weights through a spatial attention mechanism. Then, they pass through four 3x3 convolutional layers, and finally through another 3x3 convolution for feature prediction, outputting 4*k channels of features, where k is the number of anchors. The final output of 4*k feature vectors represents the detection box location information and confidence score. The role of the regression sub-network in the detection head is to locate the coordinates of the breast tumor, outputting the detection box coordinates. Here, the anchor can be understood as the anchor box, which is the initially predicted detection box information.

[0074] like Figure 7 As shown, in the classification subnetwork, the input features are first assigned different channel attention weights to all features through a channel attention mechanism. Then, similar to the regression subnetwork, it goes through four 3*3 convolutional layers, and finally through a 3*3 convolution and a softmax function to predict the output feature vector of a*k channels. Here, a is the number of categories, which includes benign and malignant tumors, so the value of a should be 2, and k is the number of anchor boxes.

[0075] In addition, when determining the final detection box, each anchor is compared with a pre-labeled ground truth box (GT Box), using a formula... Calculate the IoU (Intersection over Union) value between them. If the IoU is greater than 0.5, it is considered a positive sample, and if it is less than 0.5, it is considered a negative sample. Finally, select the anchor with the highest confidence score from the remaining positive sample anchors as the final detection box, that is, the lesion location and lesion category are identified.

[0076] This invention provides a breast lesion recognition system based on multimodal contrast-enhanced mammography images. It combines a dual-convolutional attention mechanism to perform multimodal auxiliary feature fusion, so as to make full use of the multimodal and contextual information of low-energy images and subtraction images, and quickly locate breast lesions on CEM images and make accurate classifications, which greatly improves the efficiency and accuracy of breast lesion recognition.

[0077] Furthermore, embodiments of the present invention also provide an electronic device, which may include: a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The processor can call a computer program stored in the memory to execute the aforementioned deep learning-based breast CEM image lesion recognition method, including:

[0078] S1: Acquire CEM images and perform image preprocessing to obtain preprocessed sample images; the CEM images include low-energy images and subtraction images of bilateral breast head-to-tail and internal-external oblique views; the preprocessed sample images include preprocessed low-energy images and subtraction images.

[0079] Specifically, regions of interest (ROIs) of lesions are marked in the acquired CEM images to generate lesion location labels, and the marked lesions are labeled as benign or malignant as classification labels; the labeled CEM images are binarized to obtain binarized images; the smallest bounding rectangle containing the breast foreground region is cropped from the binarized images and scaled to a uniform size to obtain scaled images; the scaled images are normalized to obtain normalized images; three identical single-channel normalized images are superimposed to obtain preprocessed sample images.

[0080] S2: Input the preprocessed low-energy image and subtraction image into two identical ResNet50 deep learning branch networks for preliminary feature extraction to obtain preliminary extracted features at different scales.

[0081] Specifically, the preprocessed low-energy map and subtraction map are input into two identical ResNet50 deep learning branch networks. First, they pass through a 3×3 convolutional layer and a max pooling layer to obtain a feature map of size 256×256. Then, they pass through three residual modules with the same structure to obtain feature maps of size 128×128, 64×64 and 32×32 respectively, which together constitute the preliminary extracted features of four different scales.

[0082] S3: Use a multimodal feature fusion network to fuse the preliminary extracted features at different scales to obtain bimodal fused features at different scales.

[0083] Specifically, for feature maps with feature sizes of 128×128, 64×64, and 32×32, the feature maps corresponding to the low-energy map and the subtraction map of the same scale are first processed by the same dual convolutional attention mechanism to obtain dual convolutional attention features. Then, the dual convolutional attention features of the subtraction map are inverted by using the 1-sigmoid method to obtain its inverse features. Then, the inverse features of the subtraction map and the dual convolutional attention features of the low-energy map are multiplied element-wise to obtain new synthetic auxiliary features. Finally, the feature map corresponding to the original subtraction map is added element-wise to the synthetic auxiliary features to obtain dual-modal fusion features of different scales.

[0084] S4: Input the dual-modal fusion features of different scales into the multi-scale pyramid feature fusion network to obtain the multi-scale pyramid feature layer.

[0085] Specifically, the dual-modal fusion features of different scales are generated from top to bottom using a neighbor interpolation 2x upsampling algorithm to create feature maps of the same size as the next level fusion features. Then, a horizontal addition fusion operation is performed. After fusion, a 3*3 convolution kernel is used to convolve each fusion result to eliminate the aliasing effect of upsampling. Finally, multi-scale pyramid features of size 32×32, 64×64 and 128×128 are obtained from top to bottom.

[0086] S5: Input the multi-scale pyramid feature layer into the corresponding detection head and output the breast lesion identification result; the detection head includes a regression sub-network and a classification sub-network; the breast lesion identification result includes the lesion location and category.

[0087] Specifically, the multi-scale pyramid features are input into the corresponding regression sub-network to output the lesion location; the multi-scale pyramid features are input into the corresponding classification sub-network to output the lesion category.

[0088] Furthermore, when the computer program in the aforementioned memory is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-transitory computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0089] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0090] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A deep learning-based breast CEM image lesion recognition system, characterized in that, include: The image acquisition and preprocessing module is used to acquire CEM images and perform image preprocessing to obtain preprocessed sample images. The CEM images include low-energy images and subtraction images of the bilateral breast head-to-tail and internal-external oblique views; the preprocessed sample images include preprocessed low-energy images and subtraction images. In the acquired CEM image, the region of interest of the lesion is marked to generate the location label of the lesion. At the same time, the marked lesions are labeled as benign or malignant as classification labels. The labeled CEM image is binarized to obtain a binarized image. The smallest bounding rectangle containing the foreground region of the breast is cropped from the binarized image and scaled to a uniform size to obtain a scaled image. The scaled image is normalized to obtain a normalized image; three identical single-channel normalized images are superimposed to obtain a preprocessed sample image. The preliminary feature extraction module is used to input the preprocessed low-energy image and subtraction image into two identical ResNet50 deep learning branch networks for preliminary feature extraction, so as to obtain preliminary extracted features at different scales. The preprocessed low-energy map and subtraction map are input into two identical ResNet50 deep learning branch networks. First, they pass through a 3×3 convolutional layer and a max pooling layer to obtain a feature map of size 256×256. Then, they pass through three residual modules with the same structure to obtain feature maps of size 128×128, 64×64 and 32×32 respectively, which together constitute four different scales of preliminary extracted features. The multimodal feature fusion module is used to fuse the preliminary extracted features at different scales using a multimodal feature fusion network to obtain dual-modal fused features at different scales. For feature maps with feature sizes of 128×128, 64×64 and 32×32, the feature maps corresponding to the low-energy map and the subtraction map of the same scale are first passed through the same double convolutional attention mechanism to obtain double convolutional attention features. The double convolutional attention features of the subtraction image are then processed through... The inverse feature is obtained by inverting the feature in a certain way; Then, the inverse features of the subtraction image are multiplied element-wise with the double convolutional attention features of the low-energy image to obtain new synthetic auxiliary features; finally, the feature map corresponding to the original subtraction image is added element-wise with the synthetic auxiliary features to obtain dual-modal fusion features at different scales. The feature processing of the dual convolutional attention mechanism: This mechanism adaptively integrates local features and their global dependencies, capturing rich contextual dependencies based on a self-constraint mechanism. Before fusion, the low-energy map and subtraction map features of each layer are first processed by dual convolutional attention operation, which superimposes the convolutional attention idea in both spatial and channel dimensions. Given an intermediate feature map, attention maps are inferred sequentially along two independent dimensions, namely channel and space. Then, the attention map is multiplied with the input feature map and another attention operation is performed. Finally, the features are further integrated by addition to obtain the final dual convolutional attention features. The feature processing of channel attention operation in the dual convolutional attention mechanism is as follows: First, the input feature map is reshaped twice to obtain feature map 1 and feature map 2. Then, the input feature map is reshaped to the same scale and channels as the original feature map and transposed to obtain feature map 3. Next, feature map 3 is multiplied with the reshaped feature map 2 on one of the branches. Then, the softmax function is used to multiply the resulting feature vector with feature 1 and multiply by the scale coefficient B. The vector is then reshaped to its original shape. Finally, the input feature is added element-wise to obtain the final feature with channel attention weights, i.e., the channel attention feature. B is initialized to 0 and gradually learned to be assigned a larger weight; The feature processing of spatial attention operation in the dual convolutional attention mechanism is as follows: The input feature map is first processed by three convolutional layers to obtain three feature maps A, B, and C. Then, the same reshape operation is performed. After that, the transpose of the reshaped C is multiplied by the reshaped B and then passed through softmax. The resulting feature vector is then multiplied by the reshaped feature map A, multiplied by a scale factor, and then reshaped back to its original shape. Finally, it is added to the input feature vector to obtain the final output feature. The value of each position of this feature is obtained by weighted summation of each position of the original feature. The scaling factor is initialized to 0 and the weights will gradually change as the learning process progresses. The multi-scale pyramid feature fusion module is used to input dual-modal fusion features of different scales into the multi-scale pyramid feature fusion network to obtain a multi-scale pyramid feature layer. The dual-modal fusion features at different scales are used to generate feature maps of the same size as the next-level fusion features from top to bottom using a neighbor interpolation 2x upsampling algorithm. Then, a horizontal addition fusion operation is performed, and after fusion, the following steps are taken: The convolution kernels are used to convolve each fusion result to eliminate the aliasing effect of upsampling, and finally multi-scale pyramid features with sizes of 32×32, 64×64 and 128×128 are obtained from top to bottom. The detection head lesion identification module is used to input the multi-scale pyramid feature layer into the corresponding detection head and output the breast lesion identification result; the detection head includes a regression subnetwork and a classification subnetwork; the breast lesion identification result includes the lesion location and category; Input the multi-scale pyramid features into the corresponding regression subnetwork to output the lesion location; input the multi-scale pyramid features into the corresponding classification subnetwork to output the lesion category. In the regression subnetwork, the input pyramid features are first assigned different spatial attention weights to all features through a spatial attention mechanism, and then processed through four... Convolutional layer, and finally through a... Convolution performs feature prediction and outputs... The characteristics of the channel, where k is the number of anchors, and the final output is... Each feature vector contains the location information of the detection box and a confidence score; When determining the final bounding box, each anchor is compared with a pre-labeled ground truth (GT) box, using a formula... Calculate the IoU value between them. If the IoU is greater than 0.5, it is considered a positive sample. If it is less than 0.5, it is considered a negative sample. Finally, select the anchor with the highest confidence score from the remaining positive sample anchors as the final detection box, that is, the lesion location and lesion category are identified.

2. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a deep learning-based method for identifying lesions in breast CEM images; the deep learning-based method for identifying lesions in breast CEM images includes: CEM images are acquired and preprocessed to obtain preprocessed sample images; the CEM images include low-energy images and subtraction images of bilateral breast head-to-tail and internal-external oblique views; the preprocessed sample images include preprocessed low-energy images and subtraction images. In the acquired CEM image, regions of interest for lesions are marked to generate lesion location labels. Simultaneously, the marked lesions are labeled as benign or malignant for classification purposes. The labeled CEM image is binarized to obtain a binarized image. The smallest bounding rectangle containing the foreground breast region is cropped from the binarized image and scaled to a uniform size to obtain a scaled image. The scaled image is normalized to obtain a normalized image. Three identical single-channel normalized images are superimposed to obtain a preprocessed sample image. The preprocessed low-energy image and subtraction image are respectively input into two identical ResNet50 deep learning branch networks for preliminary feature extraction to obtain preliminary extracted features at different scales; The preprocessed low-energy map and subtraction map are input into two identical ResNet50 deep learning branch networks. First, they pass through a 3×3 convolutional layer and a max pooling layer to obtain a feature map of size 256×256. Then, they pass through three residual modules with the same structure to obtain feature maps of size 128×128, 64×64 and 32×32 respectively, which together constitute four different scales of preliminary extracted features. A multimodal feature fusion network is used to fuse the preliminary extracted features at different scales to obtain bimodal fused features at different scales. For feature maps with feature sizes of 128×128, 64×64, and 32×32, the feature maps corresponding to the low-energy map and the subtraction map of the same scale are first processed through the same double convolutional attention mechanism to obtain double convolutional attention features; then the double convolutional attention features of the subtraction map are processed through... The inverse features are obtained by inverting the features in the following way: the inverse features of the subtraction image are then multiplied element-wise with the double convolutional attention features of the low-energy image to obtain new synthetic auxiliary features; finally, the feature map corresponding to the original subtraction image is added element-wise with the synthetic auxiliary features to obtain dual-modal fusion features at different scales. The feature processing of the dual convolutional attention mechanism: This mechanism adaptively integrates local features and their global dependencies, capturing rich contextual dependencies based on a self-constraint mechanism. Before fusion, the low-energy map and subtraction map features of each layer are first processed by dual convolutional attention operation, which superimposes the convolutional attention idea in both spatial and channel dimensions. Given an intermediate feature map, attention maps are inferred sequentially along two independent dimensions, namely channel and space. Then, the attention map is multiplied with the input feature map and another attention operation is performed. Finally, the features are further integrated by addition to obtain the final dual convolutional attention features. The feature processing procedure for channel attention operation in the dual convolutional attention mechanism is as follows: First, the input feature map is reshaped twice to obtain feature maps 1 and 2. Then, the input feature map is reshaped to the same scale and channels as the original feature maps and transposed to obtain feature map 3. Next, feature map 3 is multiplied by feature map 2 (reshaped from one of the branches), and then the softmax function is used to multiply the resulting feature vector with feature map 1 and multiply by the scale factor B. The vector is then reshaped back to its original shape. Finally, the feature vector is added element-wise with the input feature to obtain the final feature with channel attention weights, i.e., the channel attention feature. B is initialized to 0 and gradually learned to be assigned larger weights. The feature processing of spatial attention operation in the dual convolutional attention mechanism is as follows: The input feature map is first processed through three convolutional layers to obtain three feature maps A, B, and C. Then, the same reshape operation is performed. Next, the transpose of the reshaped C is multiplied by the reshaped B and then passed through softmax. The resulting feature vector is then multiplied by the reshaped feature map A, multiplied by a scaling factor, and reshaped back to its original shape. Finally, it is added to the input feature vector to obtain the final output feature. The value of each position in this feature is obtained by weighted summation of the values ​​of each position in the original feature. The scaling factor is initialized to 0 and the weights are gradually changed as the learning process progresses. The dual-modal fusion features at different scales are input into the multi-scale pyramid feature fusion network to obtain the multi-scale pyramid feature layer. The dual-modal fusion features at different scales are used to generate feature maps of the same size as the next-level fusion features from top to bottom using a neighbor interpolation 2x upsampling algorithm. Then, a horizontal addition fusion operation is performed, and after fusion, the following steps are taken: The convolution kernels are used to convolve each fusion result to eliminate the aliasing effect of upsampling, and finally multi-scale pyramid features with sizes of 32×32, 64×64 and 128×128 are obtained from top to bottom. The multi-scale pyramid feature layer is input into the corresponding detection head, and the breast lesion identification result is output; the detection head includes a regression subnetwork and a classification subnetwork; the breast lesion identification result includes the lesion location and category; Input the multi-scale pyramid features into the corresponding regression subnetwork to output the lesion location; input the multi-scale pyramid features into the corresponding classification subnetwork to output the lesion category. In the regression subnetwork, the input pyramid features are first assigned different spatial attention weights to all features through a spatial attention mechanism, and then processed through four... Convolutional layer, and finally through a... Convolution performs feature prediction and outputs... The characteristics of the channel, where k is the number of anchors, and the final output is... Each feature vector contains the location information of the detection box and a confidence score; When determining the final bounding box, each anchor is compared with a pre-labeled ground truth (GT) box, using a formula... Calculate the IoU value between them. If the IoU is greater than 0.5, it is considered a positive sample. If it is less than 0.5, it is considered a negative sample. Finally, select the anchor with the highest confidence score from the remaining positive sample anchors as the final detection box, that is, the lesion location and lesion category are identified.

3. The electronic device according to claim 2, characterized in that, The memory is a non-transitory computer-readable storage medium.

Images