Efficient neural network architecture method and system based on eretina net

Abstract

The application discloses a kind of efficient neural network architecture method and system based on ERetinaNet, method includes: selecting FRepVGG as the backbone network of convolutional neural network model, backbone network FRepVGG is composed of multiple FRepVGG stages;Equivalent conversion is carried out into single-path structure during inference using structure reparameterization technology when multiple branch structures of FRepVGG block are converted into during training;The features of the intermediate layer in FRepVGG stage are fused;Effective multispectral channel attention module is introduced in the last layer of FRepVGG stage;Vision Transformer module is inserted after backbone network;The detection head of original RetinaNet is appropriately simplified.The ERetinaNet convolutional neural network formed by the architecture of the application is more efficient and has better detection accuracy for breast lump detection.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a method and system for an efficient neural network architecture based on ERetinaNet. Background Technology

[0002] Among various malignant tumors that endanger women's physical and mental health, breast cancer is a typical example, and its incidence rate is increasing year by year. The International Agency for Research on Cancer (IARC) released the latest global cancer data in December 2020, showing that there were 2.26 million new cases of breast cancer worldwide in 2020, surpassing the 2.2 million cases of lung cancer, making breast cancer the most common cancer globally. Early diagnosis and timely treatment can accelerate the decline in breast cancer mortality. Mammography is widely recognized as the preferred and most effective diagnostic method for breast cancer. However, mammograms may contain noise or high-brightness interference, making it difficult even for experts to quickly and accurately analyze breast tissue lesions. Furthermore, the sheer volume of mammogram images makes it time-consuming and laborious for physicians to identify abnormal areas visually. Therefore, numerous computer-aided diagnosis (CAD) technologies for medical imaging have emerged and been put into clinical use to reduce the workload of doctors and improve diagnostic accuracy.

[0003] Traditional machine learning-based CAD systems for breast mass detection require image preprocessing. This involves selecting suspicious candidate regions for breast masses in the input mammogram image, extracting features from these regions, and finally classifying them to determine if they are indeed breast masses. Petrick et al. proposed using a novel adaptive density-weighted contrast enhancement filter combined with Gaussian-Laplacian edge detection to segment suspicious regions in digital mammography, followed by classification algorithms for identification. Kom et al. proposed a linear transform filter for image preprocessing and used local adaptive thresholding to automatically detect masses in mammogram images. Hang et al. proposed a system for breast mass detection and segmentation in mammography based on multi-scale morphological filtering and adaptive random forest cascades, demonstrating excellent performance on public datasets.

[0004] With the development of artificial intelligence, deep learning, as one of the branches of artificial intelligence that has experienced exponential growth in recent years, has been continuously applied to various fields and has achieved significant breakthroughs in the field of computer vision (CV). Deep learning in medical imaging has developed particularly rapidly, with more and more scholars using it to assist in the diagnosis of breast cancer. Dhungel et al. proposed a method for detecting breast masses using cascaded deep learning and a random forest classifier. This method first combines a multi-scale deep belief network and a Gaussian mixture model to select suspicious candidate regions for breast masses, and then uses a cascaded R-CNN network and a random forest classifier to reduce the number of false positives. Teare et al. proposed using dual-deep convolutional neural networks of different scales combined with a random forest network to detect breast masses in mammograms. The sensitivity and specificity obtained were basically consistent with the results obtained by professional physicians. Al-masni et al. proposed a CAD system based on the YOLO algorithm, which detects breast masses and simultaneously identifies their benign or malignant nature. The overall accuracy of the CAD system in detecting the location of masses reached 96.33%, and the overall accuracy in distinguishing between benign and malignant lesions reached 85.52%.

[0005] The above methods still have shortcomings in terms of detection accuracy or inference speed. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the above-mentioned background technology and provide an efficient neural network architecture method and system based on ERetinaNet.

[0007] Firstly, this application provides an efficient neural network architecture method based on ERetinaNet, including:

[0008] FRepVGG was selected as the backbone network of the convolutional neural network model. The backbone network FRepVGG consists of three FRepVGG stages.

[0009] The multi-branch structure of FRepVGG blocks is equivalently converted into a single-path structure using structural reparameterization techniques.

[0010] Feature fusion is performed on the features of the intermediate layer of the FRepVGG stage.

[0011] An efficient multispectral channel attention module is introduced in the final layer of the FRepVGG stage;

[0012] Insert the ViT module after the backbone network;

[0013] The original detection head is simplified.

[0014] According to the first aspect, in the first possible implementation of the first aspect, the basic block of the FRepVGG stage is equivalently converted into a single-path structure containing only 3×3 convolutions and ReLU. The step of using structural reparameterization technology to equivalently convert the multi-branch structure of the FRepVGG block during training into a single-path structure during inference specifically includes the following steps:

[0015] A 3×3 convolutional layer and a BN layer are merged into a single 3×3 convolutional layer with bias.

[0016] A 1×1 convolutional layer and a BN layer are merged into a biased 1×1 convolutional layer;

[0017] Transform a branch with only one BN layer into a branch containing a 3×3 convolutional layer;

[0018] The weights of the biased 3×3 convolutional layer, the biased 1×1 convolutional layer, and the branch containing the 3×3 convolutional layer are appropriately adjusted and then added together to form a biased 3×3 convolutional layer.

[0019] According to the first aspect, in the second possible implementation of the first aspect, the feature fusion step of the features of the intermediate layer of the FRepVGG stage specifically includes the following steps:

[0020] The outputs of each layer of the right branch of the FRepVGG stage are concatenated to obtain the concatenated feature layer.

[0021] The spliced ​​feature layers are aggregated into weight e;

[0022] Multiply the result of the left branch of the FRepVGG stage with the weight e to obtain the feature after multiplication;

[0023] The multiplied features are concatenated with the output of the last layer of the right branch to obtain the fused features of the two branches.

[0024] According to the first aspect, in a third possible implementation of the first aspect, the step of introducing an effective multispectral channel attention module in the last layer of the FRepVGG stage specifically includes the following steps:

[0025] The input X is decomposed into n parts along the channel dimension, and each part is assigned a corresponding two-dimensional DCT frequency component, denoted as Freq. i where i∈{1,2,…,n};

[0026] Then each Freq i The Freq result is obtained by concatenation, which replaces the GAP result in the original channel attention. The two fully connected layers in the original channel attention are replaced with a single convolutional layer to avoid loss of channel information.

[0027] According to the first aspect, in the fourth possible implementation of the first aspect, the step of inserting the ViT module after the backbone network specifically includes the following steps:

[0028] The feature map output from the backbone network is used as input to the ViT module. The feature map is flattened into a sequence of tokens. The ViT module learns the representations between tokens by using multi-head self-attention in the Transformer encoder.

[0029] According to the first aspect, in the fifth possible implementation of the first aspect, the step of simplifying the original detection head specifically includes the following steps:

[0030] Reduce the number of convolutional layers in the prediction subnetwork;

[0031] Reduce the number of channels in the convolutional layers of the classification subnetwork

[0032] Secondly, this application provides an efficient neural network architecture system based on ERetinaNet, including:

[0033] FRepVGG was selected as the backbone network of the convolutional neural network model. The backbone network FRepVGG consists of three FRepVGG stages.

[0034] The equivalent transformation module is communicatively connected to the backbone network selection module and is used to convert the multi-branch structure of the FRepVGG block during training into a single-path structure during inference using structural reparameterization technology.

[0035] The feature fusion module is used to fuse features from the intermediate layers of the FRepVGG stage.

[0036] Attention module reference module, used to introduce an effective multispectral channel attention module in the last layer of the FRepVGG stage;

[0037] The ViT module insertion module is used to insert ViT modules after the backbone network.

[0038] The detection head simplification module is used to simplify the original detection head.

[0039] According to the second aspect, in the first possible implementation of the second aspect, the FRepVGG stage basic block is equivalently transformed into a single-path structure containing only 3×3 convolutions and ReLU, and the equivalent transformation module includes:

[0040] The first fusion unit is used to fuse a 3×3 convolutional layer and a BN layer into a 3×3 convolutional layer with bias.

[0041] The second fusion unit is used to fuse a 1×1 convolutional layer and a BN layer into a biased 1×1 convolutional layer.

[0042] The third conversion unit is used to convert a branch with only one BN layer into a branch with a 3×3 convolutional layer.

[0043] The weight addition unit is communicatively connected to the first fusion unit, the second fusion unit, and the third conversion unit. It is used to add the weights of the biased 3×3 convolutional layer, the biased 1×1 convolutional layer, and the branch containing the 3×3 convolutional layer after appropriate adjustment to form a biased 3×3 convolutional layer.

[0044] According to the second aspect, in a second possible implementation of the second aspect, the feature fusion module includes:

[0045] The splicing unit is used to splice the outputs of each layer of the right branch of the FRepVGG stage to obtain the spliced ​​feature layer.

[0046] The agglomeration unit, which is communicatively connected to the splicing unit, is used to agglomerate the spliced ​​feature layers into weights e.

[0047] The multiplication unit, which is communicatively connected to the condensation unit, is used to multiply the result of the left branch of the FRepVGG stage with the weight e to obtain the multiplied feature.

[0048] The feature fusion unit, which is communicatively connected to the multiplication unit, is used to concatenate the multiplied features with the output of the last layer of the right branch to obtain the fused two-branch features.

[0049] Thirdly, this application also provides a method for detecting breast masses based on an efficient neural network called ERetinaNet, specifically including the following steps:

[0050] The architecture is based on the EREtinaNet convolutional neural network model;

[0051] Train and validate the convolutional neural network model of the architecture;

[0052] Breast lumps were detected in mammograms using a trained and validated convolutional neural network model.

[0053] Compared with the prior art, the advantages of the present invention are as follows:

[0054] This application proposes an efficient neural network architecture method based on ERetinaNet. By converting the multi-branch structure of the backbone network into a single-path structure, performing feature fusion in the intermediate layers, introducing an effective multispectral channel attention module in the last layer, inserting a ViT module after the backbone network, and simplifying the processing of the detection head, the resulting convolutional neural network has higher detection efficiency and accuracy for breast lumps. Attached Figure Description

[0055] Figure 1 This is a flowchart of the efficient neural network architecture method based on ERetinaNet according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the RetinaNet architecture;

[0057] Figure 3 This is a schematic diagram of the ERetinaNet architecture according to an embodiment of the present invention;

[0058] Figure 4(a) is a schematic diagram of the RepVGG stage in an embodiment of the present invention;

[0059] Figure 4(b) is a schematic diagram of the FRepVGG stage in an embodiment of the present invention;

[0060] Figure 5(a) is a schematic diagram of the RepBlock training process according to an embodiment of the present invention;

[0061] Figure 5(b) is a schematic diagram of the RepDownBlock training process according to an embodiment of the present invention;

[0062] Figure 5(c) is a schematic diagram of the structure during FRepVGG block inference in an embodiment of the present invention;

[0063] Figure 6 This is a schematic diagram of the intermediate feature layer feature extraction operation in the FRepVGG stage of this invention.

[0064] Figure 7(a) is a schematic diagram of the SE module structure according to an embodiment of the present invention;

[0065] Figure 7(b) is a schematic diagram of the eMCA module structure according to an embodiment of the present invention;

[0066] Figure 8(a) is a schematic diagram of the ViT module according to an embodiment of the present invention;

[0067] Figure 8(b) is a schematic diagram of the Transformer encoder according to an embodiment of the present invention;

[0068] Figure 9(a) is a schematic diagram of the original RetinaNet detection head;

[0069] Figure 9(b) is a schematic diagram of the simplified detection head in ERetinaNet provided in the embodiments of this application;

[0070] Figure 10(a) is the original image without Gamma transformation.

[0071] Figure 10(b) shows the image after Gamma transformation;

[0072] Figure 11(a) shows the image before data augmentation;

[0073] Figure 11(b) shows the image after horizontal flipping;

[0074] Figure 11(c) shows the image after vertical flipping;

[0075] Figure 12(a) shows the actual bounding box of the mass on a mammogram;

[0076] Figure 12(b) shows the detection results of the Faster R-CNN detection method on mammograms;

[0077] Figure 12(c) shows the detection results of the SSD detection method on mammograms;

[0078] Figure 12(d) shows the detection results of the YOLOv3 detection method on mammograms;

[0079] Figure 12(e) shows the detection results of the YOLOv7 detection method on mammograms;

[0080] Figure 12(f) shows the detection results of the ERetinaNet detection method on mammograms;

[0081] Figure 13(a) shows the actual bounding box of the mass on a mammogram;

[0082] Figure 13(b) shows the visualization results of the original RetinaNet detection method on mammograms.

[0083] Figure 13(c) shows the visualization results of the RetinaNet detection method using FRepVGG as the backbone network on mammograms.

[0084] Figure 13(d) shows the visualization results of the RetinaNet detection method on mammograms, which uses FRepVGG as the backbone network, adds the ViT module, and uses the concat operation for feature fusion.

[0085] Figure 13(e) shows the visualization results of the detection method based on ERetinaNet on mammograms. Detailed Implementation

[0086] Referring now to specific embodiments of the invention, examples of which are illustrated in the accompanying drawings. Although the invention will be described in conjunction with specific embodiments, it will be understood that it is not intended to limit the invention to the described embodiments. Rather, it is intended to cover variations, modifications, and equivalents included within the spirit and scope of the invention as defined by the appended claims. It should be noted that the method steps described herein can be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0087] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0088] Note: The examples described below are merely specific examples and are not intended to limit the embodiments of the present invention to the specific steps, values, conditions, data, order, etc. Those skilled in the art can utilize the concept of the present invention to construct more embodiments not mentioned herein by reading this specification.

[0089] Firstly, please refer to Figure 1 This application provides an efficient neural network architecture method based on ERetinaNet, including the following steps:

[0090] RetinaNet was chosen as the convolutional neural network model. RetinaNet is a single, unified network; please refer to [reference needed]. Figure 2 It consists of a backbone network, an FPN, and two prediction subnetworks. Figure 2 The structure of RetinaNet is demonstrated. ResNet-50 serves as the backbone network, initially extracting image features to obtain three feature layers C3, C4, and C5. These three feature layers are then each passed through a 1×1 convolution to reduce the number of channels before being fed into the Feature Fusion Network (FPN). The FPN, acting as a feature fusion network, fuses features at multiple scales, ultimately outputting five feature layers P3, P4, P5, P6, and P7 at different scales. P6 is obtained by directly passing C5 through a 3×3 convolution with a stride of 2, and similarly, P7 is obtained by passing P6 through a 3×3 convolution with a stride of 2. Next, these five feature layers are fed into the detection head, which consists of a classification subnetwork and a bounding box regression subnetwork. The classification subnetwork classifies the targets from these five feature layers, while the bounding box regression subnetwork predicts the location information of the target bounding boxes.

[0091] Even though RetinaNet outperforms some classic networks, it still struggles to accurately detect tumor regions due to the presence of numerous small masses and surrounding tissue interference in mammograms. Furthermore, for datasets with only one class, the RetinaNet architecture can be further simplified to improve detection speed. Therefore, this application proposes ERetinaNet, whose architecture is as follows: Figure 3 As shown, the FRepVGG architecture is proposed as the backbone network, and the ViT module is applied to ERetinaNet. Figure 3 As shown, all add operations in FPN are replaced by concat operations, and the Head in the figure represents a simplified detection head. It is worth noting that P6 is obtained from P5 through a 3×3 convolution with a stride of 2, not from C5. This change is mainly to facilitate subsequent improvements and reduce the number of parameters. In practice, whether P6 is obtained from P5 or C5 has little impact on detection accuracy. Experimental results are shown in Table 2. The step of selecting ERetinaNet as the convolutional neural network model is specifically implemented as follows:

[0092] Step S1: Select FRepVGG as the backbone network of the ERetinaNet convolutional neural network model;

[0093] Overall, the backbone network FRepVGG mainly consists of three FRepVGG stages, with each stage comprising 3, 5, and 15 RepBlocks, respectively. The structures of the RepVGG and FRepVGG stages are shown in Figure 4. In the FRepVGG stage, the feature layer first undergoes downsampling via a RepDownBlock, then is divided into two parts. The first part undergoes only a 1×1 convolution operation with half the number of output channels, while the other part first undergoes a 1×1 convolution to halve the number of channels, then passes through n convolutional layers. The Extraction operation is used to aggregate the features of this part into a weight. The features of the first part are then multiplied by this weight, achieving preliminary fusion of the two parts. Finally, the features of the two parts are concatenated and processed by an eMCA (effective multi-spectral channel attention) module.

[0094] Step S2: Use structural reparameterization technology to convert the multi-branch structure of the FRepVGG block into a single-path structure.

[0095] Step S3: Perform feature fusion on the features of the intermediate layers of the FRepVGG stage to obtain richer features;

[0096] Step S4: Introduce an effective multispectral channel attention module in the last layer of the FRepVGG stage;

[0097] Step S5: Insert the ViT module after the backbone network of ERetinaNet so that the model can learn global representations and improve robustness.

[0098] Step S6: Simplify the original detection head to make the model more efficient.

[0099] The efficient neural network architecture method based on ERetinaNet provided in this application has higher detection accuracy and better detection efficiency for breast lumps.

[0100] This application proposes an improved backbone network, Faster RepVGG (FRepVGG). In each FRepVGG stage, the channel features are divided into two parts, each with half the original number of channels. The first part undergoes only one convolutional operation, while the other part is processed by the original convolutional layer. To obtain richer features, an Extraction operation is proposed to aggregate intermediate layer features, and an effective multi-spectral channel attention (eMCA) module is added to the last layer of each stage.

[0101] Inserting a ViT (Vision Transformer) module after the backbone network can improve the robustness of ERetinaNet. Then, all addition operations in the feature pyramid network are replaced with concatenation operations.

[0102] The detection head is made more efficient by reducing the number of convolutional layers in the two prediction subnetworks and reducing the number of channels in the convolutional layers in the classification subnetwork.

[0103] Experiments verified the effectiveness and efficiency of ERetinaNet in the detection of breast lumps.

[0104] In one embodiment, the FRepVGG stage basic blocks are equivalently transformed into a single-path structure containing only 3×3 convolutions and ReLU. RepDownBlock and RepBlock are both FRepVGG basic blocks. RepDownBlock is located in the first layer of each stage and is used to change the size of the feature map, while RepBlock is located in the middle layer of each stage and mainly extracts feature information. During training, the FRepVGG stage basic blocks (i.e., RepDownBlock and RepBlock) are a multi-branch structure, while during inference, they are equivalently transformed into a single-path structure containing only 3×3 convolutions and ReLU. As shown in Figure 5, RepBlock (see Figure 5(b)) has one more identity branch than RepDownBlock (see Figure 5(a)), and the convolution stride in RepBlock is 1, while the convolution stride in RepDownBlock is 2. The structural reparameterization technique from RepVGG can decouple the structure during training and inference (see Figure 5(c)). We can use the structural reparameterization technique to convert a multi-branch structure into a single-path structure.

[0105] In one embodiment, the step of using structural reparameterization technology to equivalently convert the multi-branch structure of the FRepVGG block into a single-path structure specifically includes the following steps:

[0106] A 3×3 convolutional layer and a BN layer are merged into a single 3×3 convolutional layer with bias.

[0107] A 1×1 convolutional layer and a BN layer are merged into a biased 1×1 convolutional layer;

[0108] Transform a branch with only one BN layer into a branch containing a 3×3 convolutional layer;

[0109] The weights of the biased 3×3 convolutional layer, the biased 1×1 convolutional layer, and the branch containing the 3×3 convolutional layer are summed to form a biased 3×3 convolutional layer. Specifically, this is implemented as follows:

[0110] First, a 3×3 convolutional layer and a BN (Batch Normalization) layer are fused into a single 3×3 convolutional layer with bias. Assume... This represents a 3×3 convolution kernel with C1 input channels and C2 output channels. Indicates input, Indicates the output, * indicates the convolution operation. Let μ represent the input to the BN layer following the convolutional layer, and let μ, σ, γ, and β represent the mean, standard deviation, weights, and bias of the BN layer, respectively. For the i-th channel, we can derive the formula for the BN layer as follows:

[0111]

[0112] sti∈{1,2,…,C2},

[0113] Where ∈ is a very small number to prevent the denominator from being 0. Transforming equation (1) yields:

[0114]

[0115] sti∈{1,2,…,C2}.

[0116] Clearly, M here is the result of the convolution, i.e., M = W * X, assuming W ′ b and b′ represent the new convolutional kernel weights and biases, respectively. For the i-th convolutional kernel, W ′ b′ is defined as follows:

[0117]

[0118] sti∈{1,2,…,C2}.

[0119] Therefore, a 3×3 convolutional layer and a BN layer can be fused into a single 3×3 convolutional layer with bias:

[0120] Y :,i,:,: =BN(W*X) :,i,:,: =(W′*X) :,i,:,: +b′ i (4)

[0121] sti∈{1,2,…,C2}.

[0122] Second, a 1×1 convolutional layer and a Batch Normalization (BN) layer are fused into a biased 1×1 convolutional layer in the same way. However, to achieve weight addition with the 3×3 convolutional layer, this 1×1 convolutional kernel is equivalently transformed into a 3×3 convolutional kernel using zero-padding. Third, to transform a branch with only a BN layer into a branch containing a 3×3 convolutional layer, a 3×3 convolutional layer that can achieve the identity mapping needs to be constructed, and then this 3×3 convolutional layer is fused with the BN layer. Finally, the weights of these three layers are added to form a biased 3×3 convolutional layer, thus achieving the equivalent transformation from a multi-branch structure to a single-path structure.

[0123] According to the first aspect, in the second possible implementation of the first aspect, the feature fusion step of the features of the intermediate layer of the FRepVGG stage specifically includes the following steps:

[0124] The outputs of each layer of the right branch of the FRepVGG stage are concatenated to obtain the concatenated feature layer.

[0125] The spliced ​​feature layers are aggregated into weight e;

[0126] Multiply the result of the left branch of the FRepVGG stage with the weight e to obtain the feature after multiplication;

[0127] The multiplied features are concatenated with the output of the last layer of the right branch to obtain the fused features from the two branches. The specific implementation is as follows:

[0128] To effectively aggregate information from intermediate layers, an Extraction operation is proposed. This operation condenses the features of intermediate layers into a single weight. Its principle is similar to the channel attention mechanism, both obtaining weights through squeezing and excitation. Assume the feature layer generated by RepDownBlock uses... As shown in Figure 4(b), in each FRepVGG stage, there is only one convolutional layer on the left branch. After this 1×1 convolution operation, the number of channels of X is halved, and the generated result is denoted as... The right branch contains a 1×1 convolutional layer and n RepBlocks. The output of each layer or each block is denoted as... Where i∈{1,2,…,n+1}. To merge the features of the two branches, firstly... splicing:

[0129]

[0130] This gives us the number of channels as C. ′ Feature layer Y R C ′ = (n+1)×C. At this point, to better utilize the information from this branch, the Extraction operation is used to extract Y. R The process of condensing into weight e is as follows: Figure 6 As shown, the object of the Extraction operation is Y. R First, Y is processed using global average pooling (GAP). R The weights are obtained by compression, then by reducing the number of channels through 1×1 convolution, and finally by activation through ReLU and Sigmoid.

[0131] Then the result of the left branch, Y L Multiplying by the weight e yields richer features. Finally, Y and The two branches are then spliced ​​together to achieve the final fusion of their features.

[0132] In one embodiment, the step of introducing an effective multispectral channel attention module in the final layer of the FRepVGG stage specifically includes the following steps:

[0133] The input X is decomposed into n parts along the channel dimension, and each part is assigned a corresponding two-dimensional DCT frequency component, denoted as Freq. i where i∈{1,2,…,n};

[0134] Then each Freq i The Freq result is obtained by concatenation, which replaces the GAP result in the original channel attention. The two fully connected layers in the original channel attention are replaced with a single convolutional layer to avoid loss of channel information.

[0135] In one embodiment, the step of introducing an effective multispectral channel attention module in the last layer of the FRepVGG stage is specifically implemented as follows:

[0136] In recent years, attention mechanisms have been widely used in deep neural networks and have played a significant role in medical image processing. Attention mechanisms are similar to humans focusing their attention on observing important information, thus quickly acquiring valuable information. The essence of attention mechanisms is that they autonomously learn weights representing importance, and then combine these weights with features to make the network pay more attention to key information. Many studies have shown that introducing attention mechanisms into neural networks can significantly improve performance. Therefore, this application introduces an eMCA module in the last layer of each FRepVGG stage. The eMCA module is an improved Squeeze-and-Excitation (SE) module, and its structure is shown in Figure 7.

[0137] Channel attention mechanisms use a scalar for computation per channel, while SENet uses GAP. However, from a frequency perspective, Qin et al. mathematically proved that GAP is only equivalent to the lowest frequency component of the Discrete Cosine Transform (DCT). The proof is as follows:

[0138] Typically, the basis functions of a two-dimensional DCT can be expressed as:

[0139]

[0140] Two-dimensional DCT can be represented as:

[0141]

[0142] sth∈{0,1,…,H-1},w∈{0,1,…,W-1},

[0143] Where f 2d ∈R H×WThe spectrum of a two-dimensional DCT, x 2d ∈R H×W Let H represent the input image, and W represent the height and width of the input image, respectively. When both h and w are 0, we can obtain the following by combining formulas (6) and (7):

[0144]

[0145] at this time The lowest frequency component of the two-dimensional DCT is represented by Equation (8), which is proportional to GAP. GAP is a special case of two-dimensional DCT, meaning that the use of GAP in the channel attention mechanism only retains the lowest frequency information. Therefore, the MCA module was proposed to utilize more frequency information. First, the input X is decomposed into n parts according to the channel dimension, and each part is assigned a corresponding two-dimensional DCT frequency component. The result of the two-dimensional DCT is denoted as Freq. i , where i∈{1,2,…,n}, then for each Freq i The Freq result is obtained by splicing, and it is used to replace the GAP result in the original channel attention.

[0146] To limit model complexity and reduce computation, the two fully-connected (FC) layers of the SE module need to reduce the channel dimension. The first fully-connected layer reduces the number of channels of the input features from C. (where r is a scaling factor), the second fully connected layer recovers the feature channels, which leads to the loss of channel information. Therefore, this application replaces the two fully connected layers with a 1×1 convolutional layer without changing the number of channels.

[0147] By combining the two methods described above, this application proposes an eMCA module. The input is decomposed into multiple parts along the channel dimension, and each part is assigned a corresponding two-dimensional DCT frequency component. All results are then concatenated, thus compressing the features. The features are then activated through 1×1 convolution, ReLU, and sigmoid to generate channel attention weights. Finally, the input is multiplied by the channel attention weights to obtain the output.

[0148] In one embodiment, the step of inserting the ViT module after the ERetinaNet backbone network specifically includes the following steps:

[0149] The feature map output from the backbone network is used as input to the ViT module. The feature map is flattened into a sequence of tokens. The ViT module learns the representations between tokens by using multi-head self-attention in the Transformer encoder. Specifically, the implementation is as follows:

[0150] Since the introduction of ViT, an increasing number of studies have applied Transformers to computer vision (CV) with excellent results. Typically, convolutional neural networks (CNNs) can only learn local semantic information in image processing, while ViT can learn global semantic information because it divides the image into a series of image patches and then uses multi-head self-attention (MSA) to learn representations between patches, rather than learning local features. ViT also exhibits strong scalability in large models and with large amounts of data. However, considering the two drawbacks of the ViT model: (1) huge training cost; (2) the large amount of data required for pre-training, this application only applies the ViT module to ERetinaNet. This eliminates the need for large amounts of data for pre-training while still enabling ERetinaNet to learn global representations.

[0151] The original ViT model first divides the image into a series of 16×16 image blocks. This lack of image division leads to higher training costs. Since this application only uses the ViT module, the original image is not used as input; instead, the feature maps from the intermediate layers are used. Because the intermediate feature maps are small, they do not need to be divided. The feature maps are flattened into a sequence of tokens. The ViT module learns the representations between tokens by using multi-head self-attention in the Transformer encoder. The structure of the ViT module is shown in Figure 8(a), and its process is as follows:

[0152] The input is first flattened into a sequence of tokens. Then the position embedding is added to S. t In this context, position embedding is essentially learnable parameters encoding position, which is related to S... t They have the same dimension, so they can be added together, then S t The tokens are fed into a Transformer encoder (see Figure 7(b)), then pass through an LN (Layer Normalization) layer, and finally the resulting token sequence is reshaped. As output, the key to the Transformer encoder lies in the multi-head self-attention mechanism, which is an extension of self-attention (SA).

[0153] The inputs for self-attention include dimension D k q, k, and dimension D vv, assuming Represents the input sequence, Representing three linear transformation matrices, we can obtain Q, K, and V:

[0154] Q = ZW q K = ZW k V=ZW v (9)

[0155] The output matrix can then be calculated using the following formula:

[0156]

[0157] Multi-head self-attention can be viewed as a combination of self-attention structures, where h self-attention operations are first performed, the results are concatenated, and then a learnable parameter matrix is ​​used. To perform the conversion, follow these steps:

[0158]

[0159] This application attempts to add a ViT module after the backbone network. Since the size of C3 is too large, ViT blocks are only inserted after C4 and C5 to achieve better results.

[0160] Furthermore, feature fusion is generally achieved through concatenation or addition. For example, ResNet and FPN use the add operation, while DenseNet uses the concat operation, each with its own advantages. This application demonstrates through experiments that using the concat operation after the ViT module yields better feature fusion results.

[0161] In one embodiment, the step of simplifying the original detection head specifically includes the following steps:

[0162] Reduce the number of convolutional layers in the prediction subnetwork;

[0163] Reduce the number of channels in the convolutional layers of the classification subnetwork.

[0164] In one embodiment, the step of simplifying the original detection head is specifically implemented as follows:

[0165] Generally, the best model varies depending on the dataset. Models with simple network structures may perform better on simple datasets. Since the dataset in this application has only one class, the detection head of RetinaNet is simplified. As shown in Figure 9, in the original RetinaNet, the classification subnetwork has four 3×3 convolutional layers with 256 output channels, followed by one 3×3 convolutional layer with KA output channels, where K=1 represents the number of classes and A=9 represents the number of anchor boxes per layer. The structure of the bounding box regression subnetwork is similar to that of the classification subnetwork, except that the last layer is a 3×3 convolutional layer with 4A output channels. In ERetinaNet, fewer convolutional layers are used and the number of channels in the convolutional layers is appropriately reduced. The four sequential convolutions are replaced with two, and the number of output channels in each convolutional layer in the classification subnetwork is adjusted to 128 (except for the last layer).

[0166] Based on the same inventive concept, this application also provides a method for detecting breast lumps using a high-efficiency neural network based on ERetinaNet, specifically including the following steps:

[0167] The architecture of the ERetinaNet convolutional neural network model;

[0168] Train and validate the convolutional neural network model of the architecture;

[0169] Breast lumps were detected in mammograms using a trained and validated convolutional neural network model.

[0170] In one embodiment, the training and validation steps for the convolutional neural network model of the architecture specifically include the following steps:

[0171] A portion of the experimental dataset was randomly selected as the test set.

[0172] Data augmentation was performed on another part of the experimental dataset to obtain the augmented data, which was then divided into a training set and a validation set.

[0173] The training set and validation set are input into the established convolutional neural network model for training and validation, and the converged convolutional neural network model is obtained.

[0174] The test set is input into the converged convolutional neural network for processing to obtain the breast lump detection results.

[0175] In a more specific embodiment, the experimental data of this application comes from two publicly available datasets: DDSM and MIASS. The DDSM dataset contains 2620 cases, including 695 normal cases, 1011 benign cases, and 914 malignant cases. Each case includes axial (Craniocaudal, CC) and lateral oblique (MLO) views of the patient's left and right breasts, and the annotation file for each view records information such as lesion type, lesion region, and lesion description. The MIASS dataset contains 322 mammograms, and its annotation file also contains information such as the type of abnormality, the severity of the abnormality, and the region of the abnormality. The types of lesions typically include calcification, masses, structural distortion, or asymmetry. In this application, 1930 samples belonging to the mass type were selected from the two datasets as experimental data.

[0176] In the experimental data of this application, a large number of mammograms showed low contrast and brightness, which made it difficult to distinguish the mass from the surrounding glandular tissue. Therefore, this application adopted Gamma transformation to enhance the image, thereby improving the overall brightness of the image and enhancing the contrast of the low grayscale areas of the image, as shown in Figure 10, (a) is the original image, and (b) is the image after Gamma transformation.

[0177] Due to the small size of the experimental dataset, this application employs horizontal and vertical flipping techniques for data augmentation, as shown in Figure 10. First, one-fifth of the data is randomly selected as the test set. Then, data augmentation is performed on the remaining four-fifths of the data. Finally, the augmented data is divided into training and validation sets in an 8:2 ratio.

[0178] All experiments in this application were implemented using the deep learning framework PyTorch, with a 64-bit Windows 10 operating system, an Intel i5-9400F CPU, 16GB of RAM, an NVIDIA GeForce RTX 3070 GPU with 8GB of GPU memory, and a software environment running Python 3.8, CUDA 11.3, and cuDNN 8.2. Training and testing were both accelerated using the GPU. The initial learning rate was set to 0.0001, using the Adam optimizer with a fixed step size for the learning rate decay. In each iteration, the learning rate was reduced to 0.96, for a total of 200 iterations.

[0179] This application uses mean average precision (mAP), recall, and inference time to evaluate detection performance. mAP is the average AP (Average Precision) for each class. AP can be obtained from precision and recall. The formulas for calculating precision and recall are as follows:

[0180]

[0181]

[0182] Where TP represents the number of true positives, FP represents the number of false positives, and FN represents the number of false negatives. The PR curve is obtained by plotting Recall on the horizontal axis and Precision on the vertical axis, and AP is the area under the PR curve. The formula for calculating AP is as follows:

[0183]

[0184] Inference speed is crucial in practical applications, so inference time is used as one of the evaluation metrics. Recall can directly describe the number of missed detections, and in the field of medical imaging, it is important to minimize missed detections. Therefore, this application chooses recall as one of the evaluation metrics. In addition, the mAP in this application is calculated with the IoU threshold set to 0.5.

[0185] Results and Discussion

[0186] Comparative experiment of different detection methods

[0187] To verify the ability of the proposed method to detect masses in mammograms, ERetinaNet was compared with other detection methods, and the results are shown in Table 1. Compared with the original RetinaNet, ERetinaNet not only demonstrated high detection accuracy but also significantly improved detection speed, with mAP increasing from 79.16% to 85.01%, recall from 68.21% to 74.62%, and inference time decreasing from 26.5 ms to 23.4 ms. ERetinaNet also achieved the best results compared with other classic methods and state-of-the-art (SOTA) methods.

[0188] Table 1 Comparison of different detection methods

[0189]

[0190] Figure 12 shows the detection results of different detection methods on mammograms. As can be seen from the figure, Faster R-CNN is more likely to detect more false positive masses, while the YOLO series of methods have relatively good detection results. However, overall, ERetinaNet has the best performance in both small target detection and recognition accuracy.

[0191] ablation experiment

[0192] To facilitate subsequent improvements, this application changed the method of obtaining P6, which is now obtained from P5 (not C5). To test the impact of this change on the tumor detection results, relevant experiments were conducted, and the results are shown in Table 2. RetinaNet* represents the modified RetinaNet. From the detection results, there is no significant difference between the two, so changing the method of obtaining P6 has almost no impact on the experimental results.

[0193] Table 2 shows the experiments on different methods of obtaining P6.

[0194]

[0195] As shown in Table 3, the model performs better when using RepVGG or FRepVGG as the backbone network. Furthermore, when ResNet-50 is replaced with FRepVGG as the backbone network, mAP increases from 79.16% to 81.75%, Recall increases from 68.21% to 70.77%, and inference speed is also significantly improved. This is because in FRepVGG, features are divided into two parts, with only one part undergoing multiple convolutional layers and the other part undergoing only one convolutional operation. This allows the model using FRepVGG to detect features faster.

[0196] Table 3 shows the detection results of RetinaNet using different backbone networks.

[0197]

[0198] To verify the effectiveness of the Extraction operation and the eMCA module, relevant experiments were conducted, and the results are shown in Table 4. The symbol * indicates that the Extraction operation was not used. This indicates that the eMCA module is not added. As shown in the table, although using the Extraction operation and adding the eMCA module will slightly decrease the inference speed, mAP and Recall will be significantly improved.

[0199] Table 4. Experiments on FRepVGG

[0200]

[0201] Table 5 shows the detection results after adding the ViT module and simplifying the detection head. The first row represents the original RetinaNet, the second row represents RetinaNet after adding the ViT block, the third row represents RetinaNet that not only introduces the ViT module but also replaces all add operations in FPN with concat operations, and the fourth row simplifies the detection head based on the former. All models in Table 5 use FRepVGG as their backbone network. As shown in the table, adding the ViT block significantly improves recall, indicating a substantial reduction in missed detections. Furthermore, using the concat operation for feature fusion in FPN yields even better results. Finally, by appropriately simplifying the detection head, the model achieves the best results, with an mAP of 85.01%.

[0202] Table 5 shows the experiments on adding the ViT module and simplifying the detection head. RetinaNet(ViT): Add a ViT block after the backbone network; RetinaNet(ViT+cat): Add a ViT block after the backbone network and replace all add operations in FPN with concat operations; RetinaNet(ViT+cat+shead): Add a ViT block after the backbone network, use concat operations in FPN, and appropriately simplify the detection head.

[0203] Table 5 shows the experiments related to adding the ViT module and simplifying the detection head.

[0204]

[0205] Figure 13 shows the results of mass detection in mammograms using the improved RetinaNet and the original RetinaNet. When FRepVGG is used instead of ResNet-50 as the backbone network, the model detects fewer false positives because the eMCA module in the last layer of each stage allows the model to focus more on the mass region and extract more effective feature information. When the ViT module is introduced and feature fusion is performed using the concat operation, the number of missed detections is significantly reduced because the ViT module can learn global semantic information well through multi-head self-attention, which also allows the model to extract more comprehensive feature information. Finally, the number of false positives is further reduced when the detection head is simplified because the experimental data in this application has only one class, so it is necessary to reduce the number of channels in the convolutional layers of the detection head, and a simpler detection head is more suitable for the dataset in this application. However, ERetinaNet also has difficulty accurately detecting some masses with unclear boundaries, as shown in the samples in the last row of Figure 13.

[0206] This application proposes an efficient neural network based on ERetinaNet for breast mass detection. ERetinaNet utilizes an improved backbone network, FRepVGG, and appropriately simplifies the detection head, thereby improving inference speed and detection performance. Furthermore, ERetinaNet applies ViT to the CNN, inserting the ViT module after the backbone network to enable ERetinaNet to learn global representations. Experiments compare ERetinaNet with other detection methods, demonstrating that ERetinaNet exhibits the best mass detection capability and significantly improves the false negative rate. In addition, the proposed FRepVGG has broader applications and can serve as a general backbone network for object detection. However, ERetinaNet's performance in detecting small masses with indistinct boundaries needs further improvement, and more in-depth research will be conducted on these issues in future studies.

[0207] Secondly, based on the same inventive concept, this application provides a high-efficiency neural network architecture system based on ERetinaNet, comprising:

[0208] The backbone network selection module is used to select FRepVGG as the convolutional neural network model. The backbone network of ERetinaNet consists of 3 FRepVGG stages.

[0209] The equivalent conversion module is communicatively connected to the backbone network selection module and is used to convert the multi-branch structure of the FRepVGG block into a single-path structure using structural reparameterization technology.

[0210] The feature fusion module is used to fuse features from the intermediate layers of the FRepVGG stage.

[0211] Attention module reference module, used to introduce an effective multispectral channel attention module in the last layer of the FRepVGG stage;

[0212] The ViT module insertion module is used to insert ViT modules after the ERetinaNet backbone network.

[0213] The detection head simplification module is used to simplify the original detection head.

[0214] In one embodiment, the FRepVGG stage basic block is equivalently transformed into a single-path structure containing only 3×3 convolutions and ReLU. The equivalent transformation module includes:

[0215] The first fusion unit is used to fuse a 3×3 convolutional layer and a BN layer into a 3×3 convolutional layer with bias.

[0216] The second fusion unit is used to fuse a 1×1 convolutional layer and a BN layer into a biased 1×1 convolutional layer.

[0217] The third conversion unit is used to convert a branch with only one BN layer into a branch with a 3×3 convolutional layer.

[0218] The weight addition unit is communicatively connected to the first fusion unit, the second fusion unit, and the third conversion unit. It is used to add the weights of the biased 3×3 convolutional layer, the biased 1×1 convolutional layer, and the branch containing the 3×3 convolutional layer to form a biased 3×3 convolutional layer.

[0219] In one embodiment, the feature fusion module includes:

[0220] The splicing unit is used to splice the outputs of each layer of the right branch of the FRepVGG stage to obtain the spliced ​​feature layer.

[0221] The agglomeration unit, which is communicatively connected to the splicing unit, is used to agglomerate the spliced ​​feature layers into weights e.

[0222] The multiplication unit, which is communicatively connected to the condensation unit, is used to multiply the result of the left branch of the FRepVGG stage with the weight e to obtain the multiplied feature.

[0223] The feature fusion unit, which is communicatively connected to the multiplication unit, is used to concatenate the multiplied features with the output of the last layer of the right branch to obtain the fused two-branch features.

[0224] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for constructing an efficient neural network architecture based on ERetinaNet, wherein the efficient neural network based on ERetinaNet is used to detect breast lumps, characterized in that, include: FRepVGG was selected as the backbone network of the convolutional neural network model. The backbone network FRepVGG consists of three FRepVGG stages. The multi-branch structure of the basic blocks in the FRepVGG stage is equivalently converted into a single-path structure during inference using the structural reparameterization technique. Feature fusion is performed on the features of the intermediate layer of the FRepVGG stage. An efficient multispectral channel attention module is introduced in the final layer of the FRepVGG stage; Insert the Vision Transformer (ViT) module after the backbone network; The original detection head is simplified. Specifically, the detection of breast lumps includes: The architecture of the ERetinaNet convolutional neural network model; Train and validate the convolutional neural network model of the architecture; Breast mass detection was performed on mammogram images using a trained and validated convolutional neural network model. The feature fusion step of the intermediate layer of the FRepVGG stage specifically includes the following steps: The outputs of each layer of the right branch of the FRepVGG stage are concatenated to obtain the concatenated feature layer. The spliced ​​feature layers are aggregated into weights. ; The results and weights of the left branch of the FRepVGG stage Multiply to obtain the features after multiplication; The multiplied features are concatenated with the output of the last layer of the right branch to obtain the fused features of the two branches; The step of introducing an effective multispectral channel attention module in the final layer of the FRepVGG stage specifically includes the following steps: Input Decomposed by channel dimension Each part is divided into several sections, and each section is assigned a corresponding two-dimensional DCT frequency component, denoted as . ,in ; Then each spliced ​​together This replaces the GAP result in the original channel attention and changes the two fully connected layers in the original channel attention to a single convolutional layer to avoid the loss of channel information.

2. The efficient neural network architecture method based on ERetinaNet as described in claim 1, characterized in that, The basic blocks of the FRepVGG stage are equivalently transformed into a single-path structure containing only 3×3 convolutions and ReLU. The step of using structural reparameterization to equivalently transform the multi-branch structure of the FRepVGG block during training into a single-path structure during inference specifically includes the following steps: A 3×3 convolutional layer and a BN layer are merged into a single 3×3 convolutional layer with bias. A 1×1 convolutional layer and a BN layer are merged into a biased 1×1 convolutional layer; Transform a branch with only one BN layer into a branch containing a 3×3 convolutional layer; The weights of the biased 3×3 convolutional layer, the biased 1×1 convolutional layer, and the branch containing the 3×3 convolutional layer are appropriately adjusted and then added together to form a biased 3×3 convolutional layer.

3. The efficient neural network architecture method based on ERetinaNet as described in claim 1, characterized in that, The step of inserting the ViT module after the backbone network specifically includes the following steps: The feature map output from the backbone network is used as input to the ViT module. The feature map is flattened into a sequence of tokens. The ViT module learns the representations between tokens by using multi-head self-attention in the Transformer encoder.

4. The efficient neural network architecture method based on ERetinaNet as described in claim 1, characterized in that, The steps for simplifying the original detection head are as follows: Includes the following steps: Reduce the number of convolutional layers in the prediction subnetwork; Reduce the number of channels in the convolutional layers of the classification subnetwork.

5. A high-efficiency neural network architecture system based on ERetinaNet, wherein the high-efficiency neural network based on ERetinaNet is used to realize breast lump detection, characterized in that, include: The backbone network selection module is used to select FRepVGG as the backbone network of the convolutional neural network model. The backbone network FRepVGG consists of 3 FRepVGG stages. The equivalent transformation module is communicatively connected to the backbone network selection module and is used to convert the multi-branch structure of the FRepVGG block during training into a single-path structure during inference using structural reparameterization technology. The feature fusion module is used to fuse features from the intermediate layers of the FRepVGG stage. Attention module reference module, used to introduce an effective multispectral channel attention module in the last layer of the FRepVGG stage; The ViT module insertion module is used to insert ViT modules after the ERetinaNet backbone network. The detection head simplification module is used to simplify the original detection head; Specifically, the detection of breast lumps includes: The architecture of the ERetinaNet convolutional neural network model; Train and validate the convolutional neural network model of the architecture; Breast mass detection was performed on mammogram images using a trained and validated convolutional neural network model. The feature fusion module includes: The splicing unit is used to splice the outputs of each layer of the right branch of the FRepVGG stage to obtain the spliced ​​feature layer. The agglomeration unit, which is communicatively connected to the splicing unit, is used to agglomerate the spliced ​​feature layers into weights. ; The multiplication unit, communicatively connected to the condensation unit, is used to combine the results and weights of the left branch of the FRepVGG stage. Multiply to obtain the features after multiplication; The feature fusion unit, which is communicatively connected to the multiplication unit, is used to concatenate the multiplied features with the output of the last layer of the right branch to obtain the fused two-branch features. The step of introducing an effective multispectral channel attention module in the final layer of the FRepVGG stage specifically includes the following steps: Input Decomposed by channel dimension Each part is divided into several sections, and each section is assigned a corresponding two-dimensional DCT frequency component, denoted as . ,in ; Then each spliced ​​together This replaces the GAP result in the original channel attention and changes the two fully connected layers in the original channel attention to a single convolutional layer to avoid the loss of channel information.

6. The high-efficiency neural network architecture system based on ERetinaNet as described in claim 5, characterized in that, The FRepVGG stage basic blocks are equivalently transformed into a single-path structure containing only 3×3 convolutions and ReLU. The equivalent transformation module includes: The first fusion unit is used to fuse a 3×3 convolutional layer and a BN layer into a 3×3 convolutional layer with bias. The second fusion unit is used to fuse a 1×1 convolutional layer and a BN layer into a biased 1×1 convolutional layer. The third conversion unit is used to convert a branch with only one BN layer into a branch with a 3×3 convolutional layer. The weight addition unit is communicatively connected to the first fusion unit, the second fusion unit, and the third conversion unit. It is used to add the weights of the biased 3×3 convolutional layer, the biased 1×1 convolutional layer, and the branch containing the 3×3 convolutional layer after appropriate adjustment to form a biased 3×3 convolutional layer.

Images