Neural network with dynamic multi-scale feature fusion and channel scoring capabilities

Abstract

The invention discloses a neural network with dynamic multi-scale feature fusion and channel scoring capabilities, belongs to the field of neural network technology, is used for image classification, comprises a feature extraction module, a dynamic multi-scale feature fusion module and a channel scoring module, the output of the feature extraction module is an input feature, the input feature is input into the dynamic multi-scale feature fusion module, the dynamic multi-scale feature fusion module outputs a multi-scale fusion feature, the multi-scale fusion feature is input into the channel scoring module, the channel scoring module outputs a robust feature; the image to be processed is initially segmented using the maximum inter-class variance segmentation method, the image after the initial segmentation is input into the feature extraction module, and the robust feature is processed using a fully connected layer to obtain the classification result of the image to be processed. The invention has good performance in classification accuracy, is superior to some classification networks in the prior art, and improves the generalization of the model.

Description

Technical Field

[0001] The invention discloses a neural network with dynamic multi-scale feature fusion and channel scoring capabilities, belonging to the technical field of neural networks. Background Art

[0002] Due to differences in image acquisition protocols and scanner models, there are differences in the images captured by different X-ray examination devices. These image datasets from different devices represent different domains, and there are differences between each domain. Models trained in one domain usually perform poorly in other domains, and the lack of generalization ability is a major obstacle to deploying models in changing clinical environments. Therefore, deep learning models are required to comprehensively consider features at different scales, so that the learned features have stronger robustness. In addition, another challenge in the generalization of image classification domains is how to make the model pay more attention to domain features. Deep learning models can easily learn false correlations between different environments. This vulnerability to environmental changes may cause serious problems for deep learning systems deployed in the real world, as their reliability will be questioned. Summary of the invention

[0003] The purpose of the present invention is to provide a neural network with dynamic multi-scale feature fusion and channel scoring capabilities to solve the problem of inaccurate image classification results in the prior art.

[0004] A neural network with dynamic multi-scale feature fusion and channel scoring capabilities includes a feature extraction module, a dynamic multi-scale feature fusion module and a channel scoring module, wherein the output of the feature extraction module is an input feature, the input feature is input into the dynamic multi-scale feature fusion module, the dynamic multi-scale feature fusion module outputs a multi-scale fusion feature, the multi-scale fusion feature is input into the channel scoring module, and the channel scoring module outputs a robust feature;

[0005] The image to be processed is initially segmented using the maximum inter-class variance segmentation method Otsu, the image after initial segmentation is input into the feature extraction module, and the fully connected layer is used to process the robust features to obtain the classification result of the image to be processed.

[0006] Initial segmentation includes performing edge detection to distinguish foreground and background and removing irrelevant background.

[0007] The domain discriminator is embedded to distinguish and remove domain sensitive channels, and the dual consistency loss is used to enhance the neural network's ability to extract domain invariant features.

[0008] The dynamic multi-scale feature fusion module includes using dynamic convolution modules DCM to process input features at multiple scales, then fusing the output results of multiple DCMs with the input features, inputting the feature fusion results into a 1×1 convolution layer, and outputting multi-scale fusion features.

[0009] In the DCM with kernel size k, there are two branches, one of which includes a 1×1 convolutional layer and applies feature reduction operations to derive a reduced feature map. , and the other is a context-aware filter, which includes adaptive pooling, 1×1 convolutional layer and kernel generator. , the results of the two branches are deeply convolved and then input into a 1×1 convolution layer to form the output of a DCM with a kernel size of k .

[0010] The channel scoring module divides the multi-scale fusion features into sub-features and then splits them into three branches for processing. The first branch and the second branch perform average pooling and sub-feature division respectively, and then perform feature fusion. Then, they input the 1×1 convolution layer and the separation function Split to divide the features into two vectors. The two vectors are respectively imported into the activation function sigmoid, and then re-weighted together with the multi-scale fusion features, and the re-weighted results are group normalized; the third branch is split into two branches, and after group normalization, it is split into two branches. The first branch performs average pooling, sub-feature division and activation function softmax, and then performs matrix multiplication with the first branch after the sub-feature division. The second branch performs average pooling, sub-feature division and activation function softmax, and then performs matrix multiplication with the second branch after the sub-feature division. The results of the two matrix multiplications pass through the activation function sigmoid, and then are fused with the input features to form robust features.

[0011] Dividing the sub-features includes dividing the feature into g sub-features along the channel dimension.

[0012] Compared with the prior art, the present invention has the following beneficial effects: the present invention has better performance in classification accuracy, is superior to some classification networks in the prior art, and improves the generalization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a technical flow chart of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention is described clearly and completely below. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] A neural network with dynamic multi-scale feature fusion and channel scoring capabilities includes a feature extraction module, a dynamic multi-scale feature fusion module and a channel scoring module, wherein the output of the feature extraction module is an input feature, the input feature is input into the dynamic multi-scale feature fusion module, the dynamic multi-scale feature fusion module outputs a multi-scale fusion feature, the multi-scale fusion feature is input into the channel scoring module, and the channel scoring module outputs a robust feature;

[0016] The image to be processed is initially segmented using the maximum inter-class variance segmentation method Otsu, the image after initial segmentation is input into the feature extraction module, and the fully connected layer is used to process the robust features to obtain the classification result of the image to be processed.

[0017] Initial segmentation includes performing edge detection to distinguish foreground and background and removing irrelevant background.

[0018] The domain discriminator is embedded to distinguish and remove domain sensitive channels, and the dual consistency loss is used to enhance the neural network's ability to extract domain invariant features.

[0019] The dynamic multi-scale feature fusion module includes using dynamic convolution modules DCM to process input features at multiple scales, then fusing the output results of multiple DCMs with the input features, inputting the feature fusion results into a 1×1 convolution layer, and outputting multi-scale fusion features.

[0020] In the DCM with kernel size k, there are two branches, one of which includes a 1×1 convolutional layer and applies feature reduction operations to derive a reduced feature map. , and the other is a context-aware filter, which includes adaptive pooling, 1×1 convolutional layer and kernel generator. , the results of the two branches are deeply convolved and then input into a 1×1 convolution layer to form the output of a DCM with a kernel size of k .

[0021] The channel scoring module divides the multi-scale fusion features into sub-features and then splits them into three branches for processing. The first branch and the second branch perform average pooling and sub-feature division respectively, and then perform feature fusion. Then, they input the 1×1 convolution layer and the separation function Split to divide the features into two vectors. The two vectors are respectively imported into the activation function sigmoid, and then re-weighted together with the multi-scale fusion features, and the re-weighted results are group normalized; the third branch is split into two branches, and after group normalization, it is split into two branches. The first branch performs average pooling, sub-feature division and activation function softmax, and then performs matrix multiplication with the first branch after the sub-feature division. The second branch performs average pooling, sub-feature division and activation function softmax, and then performs matrix multiplication with the second branch after the sub-feature division. The results of the two matrix multiplications pass through the activation function sigmoid, and then are fused with the input features to form robust features.

[0022] Performing the sub-feature division includes dividing the feature into g sub-features along the channel dimension.

[0023] The technical process of the present invention is as follows Figure 1 As shown in the figure, the breast cancer X-ray image to be processed is initially segmented using the Otsu segmentation algorithm, irrelevant background is removed and the size is standardized, and features are extracted through a deep convolutional neural network, in which a domain discriminator is embedded and a double consistency loss is used. The dynamic multi-scale feature fusion module integrates information of different scales to obtain more robust features, and the channel scoring module adjusts the importance of different channels to promote the model to learn domain-invariant features related to the disease, and the final classification result is generated through the fully connected layer.

[0024] The embodiment selected breast cancer image classification for verification, using a public dataset (INbreast) and two internal datasets (InH1 and InH2) from different mammography devices. Different mammography devices can lead to inter-domain differences. For each dataset, we randomly divided it into a training set and a validation set with a ratio of 9:1. The test set is all the data of a single dataset.

[0025] The INbreast dataset consists of fully labeled full-field digital mammograms, with a total of 115 different cases. Each case shows multi-view images of the left and right breasts, resulting in a dataset of 410 mammograms. The dataset features various lesion types such as masses, calcifications, asymmetries, and deformities. Both the InH1 and InH2 datasets are derived from medical record databases. It is worth noting that the InH1 dataset has a total of 513 images. The InH2 dataset has a total of 956 images.

[0026] The code of the present invention is implemented in the PyTorch framework, and the RTX 3090 GPU is used in the experiment. The pre-trained ResNet-50 is used as the backbone, and the pre-trained weights obtained from the public package provided by the library are used. The network is optimized for 50 epochs using stochastic gradient descent (SGD), with a momentum factor of 0.9 and a weight decay of 0.0005. The batch size for training is 64, the initial learning rate is 0.001, and it is decreased by 10% at the 80% mark of the total training epochs. Model performance is evaluated using accuracy (Acc). The best model is selected on the validation set of all source domains and the accuracy is reported. The reported results are the average accuracy of five independent trials.

[0027] The classification network of the present invention and various classification networks were tested and evaluated on three datasets: INbreast, InH1 and InH2. When one dataset was used as the test set, the other two datasets were used for training. The final results are shown in Table 1.

[0028] Table 1 Classification accuracy of different classification methods on three datasets;

[0029] .

[0030] It can be seen from Table 1 that the method proposed in the present invention can obtain the best results in the unknown domain under all settings. On the public dataset INbreast, the present invention achieved the highest average value compared with other models. It is 1.52% higher than the second-ranked model. The present invention also achieved the highest average value on the private datasets InH1 and InH2. For InH1, the present invention is 0.83% higher than the second-best model. For InH2, the present invention is 1.27% higher than the second-ranked model. The average accuracy of the three datasets is also the highest, which is 1.78% higher than the second-ranked model.

[0031] This study conducted a series of ablation experiments to verify the effectiveness of the dynamic multi-scale feature fusion module and the channel scoring module in the generalization task of breast X-ray image classification domain. First, the settings of each experiment are introduced: the first experiment is to delete the dynamic multi-scale feature fusion module and the channel scoring module from the present invention, the second experiment is to delete the channel scoring module from the present invention, and the third experiment is to delete the dynamic multi-scale feature fusion module from the present invention. The experimental results are shown in Table 2.

[0032] Table 2 Ablation experiment results;

[0033] ;

[0034] The above results show that the introduction of the dynamic multi-scale feature fusion module alone can improve the accuracy of the model. The results show that the features obtained from multi-scale information have strong robustness. The introduction of the channel scoring module alone can improve the accuracy of the model. Adjusting the importance of different channels can promote the model to learn domain-invariant features and improve the generalization performance. In addition, the combined effect of the dynamic multi-scale feature fusion module and the channel scoring module is also studied. After using superposition, the performance of the model is improved and the accuracy of the model is also higher. The comparative experimental results on 3 data sets show that the present invention has better performance in classification accuracy and is better than several other classification networks. Improve the generalization of the model.

[0035] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features may be replaced by equivalents, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A neural network with dynamic multi-scale feature fusion and channel scoring capabilities, characterized in that: It includes a feature extraction module, a dynamic multi-scale feature fusion module and a channel scoring module. The output of the feature extraction module is the input feature. The input feature is input into the dynamic multi-scale feature fusion module. The dynamic multi-scale feature fusion module outputs the multi-scale fusion feature. The multi-scale fusion feature is input into the channel scoring module. The channel scoring module outputs the robust feature. The image to be processed is initially segmented using the maximum inter-class variance segmentation method Otsu, the image after initial segmentation is input into the feature extraction module, and the robust features are processed using the fully connected layer to obtain the classification result of the image to be processed; Initial segmentation includes performing edge detection to distinguish foreground and background and removing irrelevant background; The dynamic multi-scale feature fusion module includes using dynamic convolution modules DCM to process input features at multiple scales, then fusing the output results of multiple DCMs with the input features, inputting the feature fusion results into a 1×1 convolution layer, and outputting multi-scale fusion features; In the DCM with kernel size k, there are two branches, one of which includes a 1×1 convolutional layer and applies feature reduction operations to derive a reduced feature map. , and the other is a context-aware filter, which includes adaptive pooling, 1×1 convolutional layer and kernel generator. , the results of the two branches are deeply convolved and then input into a 1×1 convolution layer to form the output of a DCM with a kernel size of k ; The channel scoring module divides the multi-scale fusion features into sub-features and then splits them into three branches for processing. The first branch and the second branch perform average pooling and sub-feature division respectively, and then perform feature fusion. Then, they input the 1×1 convolution layer and the separation function Split to divide the features into two vectors. The two vectors are respectively imported into the activation function sigmoid, and then re-weighted together with the multi-scale fusion features, and the re-weighted results are group normalized; the third branch is split into two branches, and after group normalization, it is split into two branches. The first branch performs average pooling, sub-feature division and activation function softmax, and then performs matrix multiplication with the first branch after the sub-feature division. The second branch performs average pooling, sub-feature division and activation function softmax, and then performs matrix multiplication with the second branch after the sub-feature division. The results of the two matrix multiplications pass through the activation function sigmoid, and then are fused with the input features to form robust features.

2. The neural network with dynamic multi-scale feature fusion and channel scoring capabilities according to claim 1, characterized in that: The domain discriminator is embedded to distinguish and remove domain sensitive channels, and the dual consistency loss is used to enhance the neural network's ability to extract domain invariant features.

3. The neural network with dynamic multi-scale feature fusion and channel scoring capabilities according to claim 2, characterized in that: Dividing the sub-features includes dividing the feature into g sub-features along the channel dimension.

Images