Multi-scale image processing method integrated with attention mechanism

Abstract

The invention discloses a multi-scale image processing method integrated with an attention mechanism, and belongs to the technical field of image processing. The method comprises the following steps: firstly, inputting an original image, and generating a hierarchical feature map by using an encoder so as to extract corresponding feature map information; secondly, a feature set generated by the encoder is transmitted to a multi-scale feature fusion (SDI) module for extraction, and the feature expression ability is enhanced; then, inputting the extracted feature map into an attention (ResGAM) module to further improve the feature expression ability and decision accuracy of the model, thereby extracting brand new feature map information; and finally, the extracted feature information is transmitted to a decoder to obtain an output graph. According to the method, the generalization ability and robustness of the model are effectively improved, and an innovative solution is provided for extraction of image feature information.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and specifically relates to a multi-scale image processing method integrating an attention mechanism. Background Art

[0002] In the field of image processing technology, with the rapid development of deep learning, more and more researchers are paying attention to how to effectively extract and utilize feature information in images. Traditional image processing methods often rely on manually designed features and are difficult to cope with complex scenes and diverse input data. To solve this problem, researchers introduced a multi-scale feature fusion module to obtain more comprehensive image information by simultaneously processing features of different scales. This method can not only capture the detailed features in the image, but also effectively integrate global context information, significantly improving the performance of the model.

[0003] At the same time, the introduction of the attention mechanism further enhances the ability of feature extraction. By dynamically adjusting the feature weights, the attention module can automatically focus on the most important parts of the image and suppress irrelevant information. This mechanism enables the model to better focus on key features when processing complex scenes, improving the accuracy of tasks such as object detection and image segmentation. Combining the advantages of multi-scale feature fusion and attention mechanism, modern image processing systems have significantly improved in accuracy, robustness and efficiency, providing strong technical support for various application scenarios (such as medical image analysis, autonomous driving vision systems, etc.).

[0004] The present invention proposes a new feature extraction method that can capture features of different scales at the same time, ensuring the coexistence of details and global information, and can dynamically focus on key information, solving the problem of incomplete feature extraction caused by information loss. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-scale image processing method integrating an attention mechanism, which generates a hierarchical feature map through an encoder and transmits it to a multi-scale feature fusion (SDI) module and an attention (ResGAM) module in sequence for refinement, thereby enhancing the feature expression capability and extracting new feature information.

[0006] The present invention is achieved through the following technical solutions:

[0007] Step 1: Input the original image and use the encoder to generate a hierarchical feature map to extract the corresponding feature map information;

[0008] Step 2: The feature set generated by the encoder is transferred to the multi-scale feature fusion (SDI) module for refinement to enhance the feature expression capability;

[0009] Step 3: Input the refined feature map into the attention (ResGAM) module to further improve the model's feature expression ability and decision accuracy, thereby extracting new feature map information;

[0010] Step 4: Transmit the extracted feature information to the decoder to obtain the output graph.

[0011] Furthermore, the step 1 is specifically as follows: 70% of the ISIC 2017 data set is divided into a training set and 30% is divided into a test set, and U-Netv2 is used as a feature extraction network to train the training images to generate a hierarchical feature map and extract feature information.

[0012] Furthermore, the step 2 is specifically as follows: the extracted hierarchical features are respectively input into the multi-scale feature fusion (SDI) module for semantic and detail injection. It first performs weighted processing of local and global information on each level so that the feature map can integrate local spatial information and global channel information, and then uses point-by-point convolution to reduce the number of channels of the feature map to adapt to subsequent processing of the network.

[0013] Furthermore, the multi-scale feature fusion (SDI) module is specifically as follows: the SDI module processes the feature information in five steps. First, the SDI module receives the M-level features generated by the encoder, and the feature of the i-th level is represented as The feature set is represented as For each level i feature The local and global information are weighted to obtain The specific calculation expression is as follows:

[0014]

[0015] in represents the feature map after processing of the i-th layer, and Represent the weighted parameters of global and local information of the i-th layer respectively;

[0016] Then use point-by-point convolution to reduce The channel of in Among them, H i , W i and c represent Width, height and channels;

[0017] Next, use As a target reference, the size of each j-level feature map is adjusted to the same The specific calculation expression is as follows:

[0018]

[0019] Where D, I and U represent adaptive average pooling, identity mapping and bilinear interpolation respectively. Adjust to H i ×W i The size of , where 1≤i, j≤M;

[0020] Then, the resized feature map Applying smoothing convolution, we get The specific calculation expression is as follows:

[0021]

[0022] where θ ij represents the parameters of smoothed convolution, represents the j-th smoothed feature map of the i-th layer;

[0023] Finally, after all feature maps of the i-th layer are resized to the same size, the element-wise Hadamard product is applied to all resized feature maps. The specific calculation expression is as follows:

[0024]

[0025] Where H(·) represents the Hadamard product.

[0026] Furthermore, the step three is specifically as follows: the refined feature map is input into the attention (ResGAM) module, and the local and global information of the feature map are weighted, aiming to reduce information loss, make it difficult to lose details, and enhance the interactive features of the global dimension.

[0027] Furthermore, the attention (ResGAM) module is specifically as follows: the feature map is input into the ResGAM module, and skip connections are added between the layers inside the ResGAM so that the input can propagate forward faster. The specific formula is as follows:

[0028]

[0029] Among them, M s , M c Respectively represent the weighted processing of global and local information;

[0030] For global information, ResGAM initially uses 3D arrangement to preserve three-dimensional information. Subsequently, it uses a two-layer feedforward neural network to amplify the channel-space dependency across dimensions. The specific formula is as follows:

[0031] M c(F)=σ[ReversePermutation(MLP(Permutation(F)))]

[0032] Among them, M c (F) represents global information weighting, Permutation represents 3D permutation, ReversePermutation represents reverse 3D permutation, and σ represents Sigmoid function;

[0033] For local information, ResGAM uses two large convolutional layers to integrate spatial information. The specific formula is as follows:

[0034] M s (F) = σ[BN(f 7×7 (BN+ReLU(f 7×7 (F)))))]

[0035] Among them, M s (F) represents local information weighting, BN represents batch normalization, ReLU represents an activation function, and f 7×7 Represents a large convolutional layer.

[0036] Furthermore, the step four is specifically as follows: the input feature information is carefully refined through the SDI module, which aims to capture rich information from different features. Then, the information enters ResGAM for further processing to generate a processed output image, forming a complete image processing chain to ensure the quality and accuracy of the final result.

[0037] The beneficial effects of the present invention are as follows:

[0038] The present invention combines the multi-scale feature fusion (SDI) module and the attention (ResGAM) module to enhance the feature expression ability of the model, improve decision accuracy and model robustness, reduce information loss and optimize computational efficiency, providing an innovative solution for the extraction of image feature information. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0040] Figure 1 It is a schematic diagram of the process of the present invention;

[0041] Figure 2 It is a schematic diagram of the structure of the model of the present invention;

[0042] Figure 3 It is a schematic diagram of the structure of SDI;

[0043] Figure 4 Schematic diagram of the structure of ResGAM; DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] See also Figure 1-4 As shown, the present invention is a multi-scale image processing method integrating an attention mechanism, comprising the following steps:

[0046] S101: input the original image, generate a hierarchical feature map using an encoder, and extract corresponding feature map information;

[0047] S102: The feature set generated by the encoder is transmitted to the multi-scale feature fusion (SDI) module for refinement to enhance the feature expression capability;

[0048] S103: Input the refined feature map into the attention (ResGAM) module to further improve the model's feature expression ability and decision accuracy, thereby extracting new feature map information;

[0049] S104: Transmit the extracted feature information to the decoder to obtain an output graph.

[0050] As an optimization scheme of the above embodiment, the step 1 is specifically as follows: 70% of the ISIC 2017 data set is divided into a training set and 30% is divided into a test set, and U-Netv2 is used as a feature extraction network to train the training images to generate a hierarchical feature map and extract feature information.

[0051] As an optimization scheme for the above embodiment, the step 2 is specifically as follows: the extracted hierarchical features are respectively input into the multi-scale feature fusion (SDI) module for semantic and detail injection. It first performs weighted processing of local and global information on each level so that the feature map can integrate local spatial information and global channel information, and then uses point-by-point convolution to reduce the number of channels of the feature map to adapt to subsequent processing of the network.

[0052] As an optimization scheme of the above embodiment, the multi-scale feature fusion (SDI) module is specifically as follows: the SDI module processes the feature information in five steps. First, the SDI module receives the M-level features generated by the encoder, and the features of the i-th level are represented as The feature set is represented as For each level i feature The local and global information are weighted to obtain The specific calculation expression is as follows:

[0053]

[0054] in represents the feature map after processing of the i-th layer, and Represent the weighted parameters of global and local information of the i-th layer respectively;

[0055] Then use point-by-point convolution to reduce The channel of in Among them, H i , W i and c represent Width, height and channels;

[0056] Next, use As a target reference, the size of each j-level feature map is adjusted to the same The specific calculation expression is as follows:

[0057]

[0058] Where D, I and U represent adaptive average pooling, identity mapping and bilinear interpolation respectively. Adjust to H i ×W i The size of , where 1≤i, j≤M;

[0059] Then, the resized feature map Applying smoothing convolution, we get The specific calculation expression is as follows:

[0060]

[0061] where θ ij represents the parameters of smoothed convolution, represents the j-th smoothed feature map of the i-th layer;

[0062] Finally, after all feature maps of the i-th layer are resized to the same size, the element-wise Hadamard product is applied to all resized feature maps. The specific calculation expression is as follows:

[0063]

[0064] Where H(·) represents the Hadamard product.

[0065] As an optimization scheme for the above embodiment, the step three is specifically as follows: the refined feature map is input into the attention (ResGAM) module, and the local and global information of the feature map is weighted, aiming to reduce information loss, make it difficult to lose details, and enhance the interactive features of the global dimension.

[0066] As an optimization solution of the above embodiment, the attention (ResGAM) module is specifically as follows: the feature map is input into the ResGAM module, and skip connections are added between the layers inside the ResGAM so that the input can propagate forward faster. The specific formula is as follows:

[0067]

[0068] Among them, M s , M c Respectively represent the weighted processing of global and local information;

[0069] For global information, ResGAM initially uses 3D arrangement to preserve three-dimensional information. Subsequently, it uses a two-layer feedforward neural network to amplify the channel-space dependency across dimensions. The specific formula is as follows:

[0070] M c (F)=σ[ReversePermutation(MLP(Permutation(F)))]

[0071] Among them, M c (F) represents global information weighting, Permutation represents 3D permutation, ReversePermutation represents reverse 3D permutation, and σ represents Sigmoid function;

[0072] For local information, ResGAM uses two large convolutional layers to integrate spatial information. The specific formula is as follows:

[0073] M s (F) = σ[BN(f 7×7 (BN+ReLU(f 7×7 (F)))))]

[0074] Among them, M s(F) represents local information weighting, BN represents batch normalization, ReLU represents an activation function, and f 7×7 Represents a large convolutional layer.

[0075] As an optimization solution for the above embodiment, the step four is specifically as follows: the input feature information is carefully refined through the SDI module, which aims to capture rich information from different features. Then, the information enters ResGAM for further processing to generate a processed output image, forming a complete image processing chain to ensure the quality and accuracy of the final result.

[0076] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative labor are still within the scope of protection of the present invention.

Claims

1. A multi-scale image processing method integrating attention mechanism, characterized in that: The multi-scale image processing method integrating the attention mechanism comprises the following steps: Step 1: Input the original image and use the encoder to generate a hierarchical feature map to extract the corresponding feature map information; Step 2: The feature set generated by the encoder is transferred to the multi-scale feature fusion (SDI) module for refinement to enhance the feature expression capability; Step 3: Input the refined feature map into the attention (ResGAM) module to further improve the model's feature expression ability and decision accuracy, thereby extracting new feature map information; Step 4: Transmit the extracted feature information to the decoder to obtain the output graph.

2. The method according to claim 1, characterized in that The step 1 is specifically as follows: 70% of the ISIC 2017 data set is divided into a training set and 30% is divided into a test set, and U-Netv2 is used as a feature extraction network to train the training images to generate a hierarchical feature map and extract feature information.

3. The method according to claim 1, characterized in that The step 2 is specifically as follows: the extracted hierarchical features are respectively input into the multi-scale feature fusion (SDI) module for semantic and detail injection. It first performs weighted processing of local and global information on each level so that the feature map can integrate local spatial information and global channel information, and then uses point-by-point convolution to reduce the number of channels of the feature map to adapt to subsequent processing of the network.

4. The method as claimed in claim 3, characterized in that The multi-scale feature fusion (SDI) module is specifically as follows: the SDI module processes the feature information in five steps. First, the SDI module receives the M-level features generated by the encoder, and the feature of the i-th level is represented as f i 0 , the feature set is represented as For each level i feature The local and global information are weighted to obtain The specific calculation expression is as follows: in represents the feature map after processing of the i-th layer, and Represent the weighted parameters of global and local information of the i-th layer respectively; Then use point-by-point convolution to reduce The channel of in Among them, H i , W i and c represent Width, height and channels; Next, use As a target reference, the size of each j-level feature map is adjusted to the same The specific calculation expression is as follows: Where D, I and U represent adaptive average pooling, identity mapping and bilinear interpolation respectively. Adjust to H i ×W i The size of , where 1≤i, j≤M; Then, the resized feature map Applying smoothing convolution, we get The specific calculation expression is as follows: where θ ij represents the parameters of smoothed convolution, represents the j-th smoothed feature map of the i-th layer; Finally, after all feature maps of the i-th layer are resized to the same size, the element-wise Hadamard product is applied to all resized feature maps. The specific calculation expression is as follows: Where H(·) represents the Hadamard product.

5. The method according to claim 1, characterized in that The step three is specifically as follows: the refined feature map is input into the attention (ResGAM) module, and the local and global information of the feature map are weighted, aiming to reduce information loss, make it difficult to lose details, and enhance the interactive features of the global dimension.

6. The method as claimed in claim 5, characterized in that The attention (ResGAM) module is specifically as follows: the feature map is input into the ResGAM module, and skip connections are added between the layers inside the ResGAM so that the input can propagate forward faster. The specific formula is as follows: Among them, M s , M c Respectively represent the weighted processing of global and local information; For global information, ResGAM initially uses 3D arrangement to preserve three-dimensional information. Subsequently, it uses a two-layer feedforward neural network to amplify the channel-space dependency across dimensions. The specific formula is as follows: M c (F)=σ[ReversePermutation(MLP(Permutation(F)))] Among them, M c (F) represents global information weighting, Permutation represents 3D permutation, ReversePermutation represents reverse 3D permutation, and σ represents Sigmoid function; For local information, ResGAM uses two large convolutional layers to integrate spatial information. The specific formula is as follows: M s (F)=σ[BN(f 7×7 (BN+ReLU(f 7×7 (F))))] Among them, M s (F) represents local information weighting, BN represents batch normalization, ReLU represents an activation function, and f 7×7 Represents a large convolutional layer.

7. The method according to claim 1, characterized in that The step four is specifically as follows: the input feature information is carefully refined through the SDI module, which aims to capture rich information from different features. Then, this information enters ResGAM for further processing to generate a processed output image, forming a complete image processing chain to ensure the quality and accuracy of the final result.

Images