Construction method of bidirectional adaptive reweighted neural network for image processing

Abstract

The invention discloses a construction method of a bidirectional self-adaptive reweighted neural network for image processing, belongs to the technical field of neural networks, is used for image classification, and comprises the following steps: inputting an image to be processed into a convolutional neural network feature extractor to obtain image features, inputting the image features into the bidirectional self-adaptive reweighted neural network, and extracting the image features from the bidirectional self-adaptive reweighted neural network; image features in the bidirectional adaptive reweighting neural network are divided into two branches, one branch is input into the cross-region reweighting module to obtain a first output, the other branch is input into the adaptive multi-scale global-to-local reweighting module to obtain a second output, the two outputs are reduced into vectors with fixed sizes in combination with an attention mechanism, and the two vectors are output; and obtaining an image processing result through the full connection layer. The method can well balance global and local information, does not need to depend on ROI (Region of Interest) labeling, and does not need to calibrate hyper-parameters; and large-scale information is integrated to evaluate a small-scale area, and detail information is effectively highlighted, so that the classification effect is improved.

Description

Technical Field

[0001] The invention discloses a method for constructing a bidirectional adaptive reweighted neural network for image processing, and belongs to the technical field of neural networks. Background Art

[0002] In the field of computer vision, traditional image processing methods may not capture the subtle but critical details required for accuracy, and the emergence of deep learning technology has opened up new ways to improve the accuracy of image classification. Deep neural networks (DNNs) have demonstrated excellent ability to learn hierarchical features directly from data, reducing the reliance on manual feature engineering. However, these methods require a large amount of labeled data, including bounding boxes or segmentation information, and the creation of such datasets usually requires a lot of manpower. Advances in imaging technology have led to the generation of a large number of datasets based on film and digitized images, which vary significantly between different devices. These differences pose challenges to training high-quality and versatile models. In addition, most existing models use rigid segmentation in multi-instance learning, which may damage and fragment information in image classification. These models also rely heavily on hyperparameters, and the optimal hyperparameters for different datasets vary, which further hinders the generalization process of the model. Summary of the invention

[0003] The purpose of the present invention is to provide a method for constructing a bidirectional adaptive reweighted neural network for image processing, so as to solve the problem in the prior art that image classification tasks are overly dependent on a large amount of standard data, resulting in a large amount of time spent on data set construction, thereby reducing the efficiency of image classification tasks.

[0004] A method for constructing a bidirectional adaptive reweighted neural network for image processing, wherein the image to be processed is input into a convolutional neural network feature extractor to obtain image features. ,Will Input bidirectional adaptive reweighted neural network, bidirectional adaptive reweighted neural network It is divided into two branches, one of which inputs the cross-region reweighting module to obtain the output , an input adaptive multi-scale global to local reweighting module to get the output , combined with the attention mechanism and Reduced to a vector of fixed size, the image processing result is obtained through a fully connected layer.

[0005] The cross-region reweighting module includes a regional attention module and a cross-region attention module.

[0006] The regional attention module will The dimension is compressed to 512 to obtain image features ,Will Divided into 64 blocks, each block uses the two-dimensional selective state space model SS2D to perform local reweighting to obtain image features ,Will As the output of the region attention module .

[0007] SS2D includes:

[0008] ;

[0009] In the formula, represents the fusion operation, represents the two-dimensional selectivity state space model Module operation, , , , Respectively represent flattening along four directions.

[0010] Modules include, The input of the module is linearly transformed to generate the matrix and the time step parameter , the hidden state is updated to:

[0011] ;

[0012] In the formula, is a natural constant, yes The module input, yes Always hide the status;

[0013] Output for:

[0014] ;

[0015] All time steps Sequence sorting Module output.

[0016] The cross-region attention module divides the input into two branches, one of which combines the adjustable parameters Perform matrix multiplication and then split it into three inputs , and calculate the weight:

[0017] ;

[0018] ;

[0019] ;

[0020] In the formula, is the number of unidimensional cuts, is the total side length of the block, It is The side length of a block, , , There are three inputs The corresponding output weight, Indicates keys, Represents the total number of keys, , is the activation function.

[0021] The second branch of the cross-region attention module After matrix multiplication, the output is input into the SS2D module. ,Will , , Perform matrix multiplication to get .

[0022] The adaptive multi-scale global to local reweighting module Perform maximum pooling, and then The global scale module, the quarter scale module, and the sixteenth scale module are input simultaneously, and weighted processing is performed in combination with different scale modules:

[0023] ;

[0024] In the formula, , It is a scale module, including the global scale module , quarter-scale module , 1 / 16 scale module , include The final result of the weighting is include The final result of the weighting is include The final result of the weighting is Right of expression, represents the final result of weighting, Represents element-by-element multiplication, including the weight information The output information of is unified in size, and then embedded respectively, and finally feature fusion is performed to obtain ;

[0025] right Combined with 1×1 convolution, , , As three branches, 1×1 convolution processing is performed respectively, two of the three convolution processing results are matrix added, and then the addition result and the third convolution processing result are matrix added to obtain :

[0026] ;

[0027] In the formula, It is an adjustable parameter.

[0028] Combined with the attention mechanism and Reduction to a fixed-size vector includes:

[0029] ;

[0030] In the formula, express or , is a fully connected layer, are the fully connected layer parameters, represents the product of length and width, Indicates the number of channels, Time Representative , Time Representative , represents matrix multiplication, represents the output of the attention mechanism;

[0031] Will Input fully connected layer:

[0032] ;

[0033] In the formula, yes or The predicted probability of is the activation function, and are the weights and biases of the linear regression layer respectively;

[0034] Get the image processing results :

[0035] ;

[0036] In the formula, yes The predicted probability of yes The predicted probability of .

[0037] Loss function of bidirectional adaptive reweighted neural network for:

[0038] ;

[0039] In the formula, is the binary result of image processing, , The two values ​​of correspond to two binary results, is the predicted probability corresponding to the binary outcome, =1, is the architectural parameter of the bidirectional adaptive reweighted neural network, , represents the regularization term, is the weight used to alleviate sample imbalance.

[0040] Compared with the prior art, the present invention has the following beneficial effects: the present invention can well balance global and local information, does not rely on ROI annotation, and does not require calibration of hyperparameters; integrates large-scale information to evaluate small-scale areas, and effectively highlights detail information, thereby improving classification results; uses a cross-regional module to integrate information from different regions, reduces image information loss and dependence on hyperparameters due to rigid segmentation, and avoids the high-value deviation that may be caused by global attention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] A method for constructing a bidirectional adaptive reweighted neural network for image processing, wherein the image to be processed is input into a convolutional neural network feature extractor to obtain image features. ,Will Input bidirectional adaptive reweighted neural network, bidirectional adaptive reweighted neural network It is divided into two branches, one of which inputs the cross-region reweighting module to obtain the output , an input adaptive multi-scale global to local reweighting module to get the output , combined with the attention mechanism and Reduced to a vector of fixed size, the image processing result is obtained through a fully connected layer.

[0043] The cross-region reweighting module includes a regional attention module and a cross-region attention module.

[0044] The regional attention module will The dimension is compressed to 512 to obtain image features ,Will Divided into 64 blocks, each block uses the two-dimensional selective state space model SS2D to perform local reweighting to obtain image features ,Will As the output of the region attention module .

[0045] SS2D includes:

[0046] ;

[0047] In the formula, represents the fusion operation, represents the two-dimensional selectivity state space model Module operation, , , , Respectively represent flattening along four directions.

[0048] Modules include, The input of the module is linearly transformed to generate the matrix and the time step parameter , the hidden state is updated to:

[0049] ;

[0050] In the formula, is a natural constant, yes The module input, yes Always hide the status;

[0051] Output for:

[0052] ;

[0053] All time steps Sequence sorting Module output.

[0054] The cross-region attention module divides the input into two branches, one of which combines the adjustable parameters Perform matrix multiplication and then split it into three inputs , and calculate the weight:

[0055] ;

[0056] ;

[0057] ;

[0058] In the formula, is the number of unidimensional cuts, is the total side length of the block, It is The side length of a block, , , There are three inputs The corresponding output weight, Indicates keys, Represents the total number of keys, , is the activation function.

[0059] The second branch of the cross-region attention module After matrix multiplication, the output is input into the SS2D module. ,Will , , Perform matrix multiplication to get .

[0060] The adaptive multi-scale global to local reweighting module Perform maximum pooling, and then The global scale module, the quarter scale module, and the sixteenth scale module are input simultaneously, and weighted processing is performed in combination with different scale modules:

[0061] ;

[0062] In the formula, , It is a scale module, including the global scale module , quarter-scale module , 1 / 16 scale module , include The final result of the weighting is include The final result of the weighting is include The final result of the weighting is Right of expression, represents the final result of weighting, Represents element-by-element multiplication, including the weight information The output information of is unified in size, and then embedded respectively, and finally feature fusion is performed to obtain ;

[0063] right Combined with 1×1 convolution, , , As three branches, 1×1 convolution processing is performed respectively, two of the three convolution processing results are matrix added, and then the addition result and the third convolution processing result are matrix added to obtain :

[0064] ;

[0065] In the formula, It is an adjustable parameter.

[0066] Combined with the attention mechanism and Reduction to a fixed-size vector includes:

[0067] ;

[0068] In the formula, express or , is a fully connected layer, are the fully connected layer parameters, represents the product of length and width, Indicates the number of channels, Time Representative , Time Representative , represents matrix multiplication, represents the output of the attention mechanism;

[0069] Will Input fully connected layer:

[0070] ;

[0071] In the formula, yes or The predicted probability of is the activation function, and are the weights and biases of the linear regression layer respectively;

[0072] Get the image processing results :

[0073] ;

[0074] In the formula, yes The predicted probability of yes The predicted probability of .

[0075] Loss function of bidirectional adaptive reweighted neural network for:

[0076] ;

[0077] In the formula, is the binary result of image processing, , The two values ​​of correspond to two binary results, is the predicted probability corresponding to the binary outcome, =1, is the architectural parameter of the bidirectional adaptive reweighted neural network, , represents the regularization term, is the weight used to alleviate sample imbalance.

[0078] The present invention uses two well-recognized datasets, CBIS-DDSM and INbreast, and an internal dataset In-house to evaluate the performance of the BARNet (model of the present invention) method. The CBIS-DDSM dataset contains 3071 mammograms, which have been carefully annotated for classification and segmentation, covering a variety of pathological types such as masses and calcifications. The INbreast dataset contains 410 digitized mammograms from multiple angles of 115 different cases, which have been carefully classified and segmented. It not only covers masses and calcifications, but also includes various lesion types such as asymmetry and distortion, with comprehensive annotation information. A detailed evaluation of medical records from February 2021 to October 2022 was conducted to identify patients with histologically confirmed breast cancer and benign tumors. The dataset contains 406 full-field mammography images of 214 patients, including 113 benign tumors and 293 malignant tumors.

[0079] Table 1 provides the data distribution of the CBIS-DDSM mammogram image set and the internal dataset. Regarding the CBIS-DDSM dataset, referring to its original data source, it was found that some images contained both tumors and calcifications, causing them to appear in both the tumor training set and the calcification dataset. Therefore, all duplicate images were deleted from the database, and the dataset was randomly divided into five equal parts, with a training set to test set ratio of 4:1, and a cross-validation technique was used. The ratio of positive and negative samples in the INbreast dataset is 3.1:1, which is significantly imbalanced. To address this problem, the INbreast dataset was also randomly divided into five equal parts, with a training set to test set ratio of 4:1, and a cross-validation method was used. For the internal dataset, the same processing strategy as the INbreast dataset was adopted.

[0080] Table 1. Data distribution of CBIS-DDSM mammography image set and internal dataset

[0081] ;

[0082] The present invention uses deep learning technology and Py-Torch framework to process medical imaging data. For the INbreast dataset, the Otsu segmentation algorithm is used to identify and crop the breast area, remove unnecessary background, and resize the image to 800×800. The internal dataset was also processed similarly and resized to 448×448. For the CBIS-DDSM dataset containing artifacts, the image was resized to 448×448, binarized, and masked with the maximum connected area to retain the maximum connected area. All images were normalized by dividing the pixel value by 255 to map it to the [0,1] interval. To prevent overfitting, the images were randomly adjusted and rotated in each training cycle with a rotation angle between -25 and +25 degrees.

[0083] The present invention is implemented in PyTorch, and the experiments are performed on a Tesla V100 GPU. The DenseNet-169 architecture based on the code provided by the torchvision library is used. Some minor adjustments were made to replace the final classification layer with a multi-scale module that can extract features from a global to local perspective, and a position embedding module from a local to a global perspective, and finally a fixed-length linear regression layer is formed for output. The backbone network weights were initialized using pre-trained weights from ImageNet to ensure a fair comparison with existing studies. In experiments using the INbreast and CBIS-DDSM datasets, the CNN was also initialized using pre-trained weights from ImageNet and updated with a small learning rate. These pre-trained weights can be obtained from the public software package provided by the torchvision library. Model training is performed using the Adam optimization algorithm. The learning rate of the logistic regression layer is initialized as , while the learning rate of the convolutional neural network (CNN) layer is set to A learning rate decay strategy of 0.98 every 10 cycles was also implemented, and the loss function hyperparameters were set to Model training was stopped after 150 epochs, and the weights were retained for testing. Given that the number of training epochs required for different datasets varies, the model with the highest accuracy was saved during training to prevent overfitting due to excessive training.

[0084] Table 2 shows the details of the experimental data obtained by integrating the adaptive multi-scale global to local reweighting module AMG2L and the cross-regional reweighting module L2GCR into the baseline convolutional neural network (CNN) for standard mammogram classification. This table evaluates the impact of these two modules on the model performance from a quantitative perspective. Specifically, the accuracy and AUC values ​​listed in Table 2 are the average results of five independent tests, providing statistical robustness for the experimental results. These values ​​not only reflect the overall performance of the model in the classification task, but also reveal the specific contributions of AMG2L and L2GCR to the model performance by comparing the performance before and after the introduction of these two modules.

[0085] Table 2. Accuracy and AUC values ​​of L2GCR of different baseline models combined with AMG2L on the INBREAST dataset

[0086] ;

[0087] In Table 2, Alexnet, Resnet-34, Resnet-50, VGG19, and DenseNet-169 are state-of-the-art deep learning models. In these experiments, the average improvement value measures the performance difference between the baseline model with AMG2L and L2GCR introduced and the original model. This metric reflects the improvement of model performance from a macro perspective, indicating the positive impact of integrating AMG2L and L2GCR on model performance. At the same time, the maximum improvement value represents the maximum performance improvement recorded in these five tests, revealing the potential performance gain that AMG2L and L2GCR can bring to the model under optimal conditions. The experimental results show that integrating AMG2L and L2GCR into the baseline CNN model significantly improves the AUC and accuracy. This finding is significant because it not only confirms the effectiveness of AMG2L and L2GCR in improving model performance, but also shows that their addition helps the model identify lesion areas more accurately. This is crucial to improve the accuracy and reliability of mammogram classification, especially in the detection and diagnosis of early lesions. Furthermore, the increase in parameters and computational cost brought about by these improvements is negligible. This is particularly noteworthy because in practical applications, computational resources and model complexity are often important considerations. The high efficiency of AMG2L and L2GCR means that they can bring significant performance improvements to the model without significantly increasing the computational burden.

[0088] In this paper, the performance of BARNet is compared with existing baseline methods on three datasets. Traditional models usually rely on a multi-stage training process or require manual annotation of data, and edge information or regions of interest (ROI) need to be annotated during training and testing, which leads to high training costs. Recent studies have explored the application of convolutional neural networks (CNNs) in the field of mammography classification, which have shown effectiveness in other fields. However, these methods often ignore the specific features of mammography images or rely on hyperparameters. Experimental results show that BARNet's accuracy and AUC are close to the optimal results and have extremely high specificity, which means that BARNet performs well in reducing false positives (incorrectly identifying non-lesion areas as lesion areas). BARNet is fully automated, without manual annotation or hyperparameter adjustment, and can directly extract classification labels from case results for training, thus realizing an end-to-end automated training process. To ensure the fairness of the experiment, all conditions are strictly kept consistent except for the model itself, and the model is the only variable modified. This univariate variation experimental design is applied in experiments on both CBIS-DDSM and internal datasets. The results show that BARNet performs well in multiple evaluation indicators without the need for manual annotation and dual-view information, especially in key indicators such as accuracy and precision, which are significantly improved compared with the baseline network. By integrating global and local information and fusing multi-scale features, BARNet can more accurately identify and classify lesions. The model not only takes into account the small size and irregular shape characteristics of breast cancer lesions, but also excludes irrelevant areas and integrates information at different scales for comprehensive judgment.

[0089] In the experiments conducted on the internal dataset, the same preprocessing method as the INbreast dataset was used, and the results showed that BARNet performed extremely well on various evaluation indicators including AUC, accuracy, specificity, and precision. This shows that BARNet is very effective in reducing false positives, identifying lesions, and distinguishing positive and negative samples. The comprehensive experimental results show that BAR-Net performs well in the breast cancer classification task, which is mainly attributed to the following points: First, the feature extractor fully utilizes the advantages of convolutional neural networks (CNNs), and the feature reuse characteristics of the dense network (DenseNet) feature extractor help to reduce overfitting to a certain extent. Secondly, the AMG2L module uses the attention mechanism to reweight the features within and between regions to evaluate the lesion situation from local to global. The L2GCR module uses the spatial state through multi-scale input and the forward attention characteristics of SSM to judge the local lesion situation from global to local. Finally, the two results are fused through a fully connected layer to obtain the final classification decision, which effectively balances the information at different scales. It is worth noting that the attention mechanism of SSM has linear complexity, which significantly reduces the training cost compared to the quadratic complexity of the traditional Transformer, and does not require additional manual annotation, which is a clear advantage.

[0090] The effects of different numbers of partitions in the L2GCR module on the model accuracy (Acc) and area under the curve (AUC) were studied in detail. The results show that when the number of partitions exceeds 20, the AUC value of the model tends to stabilize and is no longer significantly affected by the number of partitions. At the same time, the change in the accuracy value is small, and its standard deviation is controlled within 4% as the number of partitions increases. This result strongly proves that the L2GCR module can effectively alleviate the adverse effects of rigid segmentation on model performance. In addition, the impact of different segmentation methods on the model is relatively small. However, when the number of partitions decreases, the L2GCR module becomes a simple superposition of two global attention modules. At this time, the AUC value is significantly reduced and is more susceptible to high value deviations.

[0091] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a bidirectional adaptive reweighted neural network for image processing, characterized in that: Input the image to be processed into the convolutional neural network feature extractor to obtain image features ,Will Input bidirectional adaptive reweighted neural network, bidirectional adaptive reweighted neural network It is divided into two branches, one of which inputs the cross-region reweighting module to obtain the output , an input adaptive multi-scale global to local reweighting module to get the output , combined with the attention mechanism and Reduced to a vector of fixed size, the image processing result is obtained through a fully connected layer.

2. The method for constructing a bidirectional adaptive reweighted neural network for image processing according to claim 1, characterized in that: The cross-region reweighting module includes a regional attention module and a cross-region attention module.

3. The method for constructing a bidirectional adaptive reweighted neural network for image processing according to claim 2, characterized in that: The regional attention module will The dimension is compressed to 512 to obtain image features ,Will Divided into 64 blocks, each block uses the two-dimensional selective state space model SS2D to perform local reweighting to obtain image features ,Will As the output of the region attention module .

4. The method for constructing a bidirectional adaptive reweighted neural network for image processing according to claim 3, characterized in that: SS2D includes: ； In the formula, represents the fusion operation, represents the two-dimensional selectivity state space model Module operation, , , , Respectively represent flattening along four directions.

5. The method for constructing a bidirectional adaptive reweighted neural network for image processing according to claim 4, characterized in that: Modules include, The input of the module is linearly transformed to generate the matrix and the time step parameter , the hidden state is updated to: ； In the formula, is a natural constant, yes The module input, yes Always hide the status; Output for: ； All time steps Sequence sorting Module output.

6. The method for constructing a bidirectional adaptive reweighted neural network for image processing according to claim 5, characterized in that: The cross-region attention module divides the input into two branches, one of which combines the adjustable parameters Perform matrix multiplication and then split into three inputs , and calculate the weight: ； ； ； In the formula, is the number of one-dimensional cuts, is the total side length of the block, It is The side length of a block, , , There are three inputs The corresponding output weight, Indicates keys, Represents the total number of keys, , is the activation function.

7. The method for constructing a bidirectional adaptive reweighted neural network for image processing according to claim 6, characterized in that: The second branch of the cross-region attention module After matrix multiplication, the output is input into the SS2D module. ,Will , , Perform matrix multiplication to get .

8. The method for constructing a bidirectional adaptive reweighted neural network for image processing according to claim 7, characterized in that: The adaptive multi-scale global to local reweighting module Perform maximum pooling, and then The global scale module, the quarter scale module, and the sixteenth scale module are input simultaneously, and weighted processing is performed in combination with different scale modules: ； In the formula, , It is a scale module, including the global scale module , quarter-scale module , 1 / 16 scale module , include The final result of the weighting is include The final result of the weighting is include The final result of the weighting is Right of expression, represents the final result of weighting, Represents element-by-element multiplication, including the weight information The output information of is unified in size, and then embedded respectively, and finally feature fusion is performed to obtain ; right Combined with 1×1 convolution, , , As three branches, 1×1 convolution processing is performed respectively, two of the three convolution processing results are matrix added, and then the addition result and the third convolution processing result are matrix added to obtain : ； In the formula, It is an adjustable parameter.

9. The method for constructing a bidirectional adaptive reweighted neural network for image processing according to claim 8, characterized in that: Combined with the attention mechanism and Reduction to a fixed-size vector includes: ； In the formula, express or , is a fully connected layer, are the fully connected layer parameters, represents the product of length and width, Indicates the number of channels, Time Representative , Time Representative , represents matrix multiplication, represents the output of the attention mechanism; Will Input fully connected layer: ； In the formula, yes or The predicted probability of is the activation function, and are the weights and biases of the linear regression layer respectively; Get the image processing results : ； In the formula, yes The predicted probability of yes The predicted probability of .

10. The method for constructing a bidirectional adaptive reweighted neural network for image processing according to claim 9, characterized in that: Loss function of bidirectional adaptive reweighted neural network for: ； In the formula, is the binary result of image processing, , The two values ​​of correspond to two binary results, is the predicted probability corresponding to the binary outcome, =1, is the architectural parameter of the bidirectional adaptive reweighted neural network, , represents the regularization term, is the weight used to alleviate sample imbalance.

Images