A salient object detection method based on attention mechanism and multi-scale feature fusion

Abstract

The application discloses a salient object detection method based on an attention mechanism and multi-scale feature fusion, and comprises the following steps: S1, data preprocessing is performed, and a data set and a label required for deep learning model training and testing are constructed; S2, a salient object detection network based on the attention mechanism and the multi-scale feature fusion is constructed; S3, the training data set in S1 is input into the salient object detection network constructed in S2 for training, and a salient object detection model is obtained; and S4, a picture is randomly selected and input into the model, and a detection result is obtained. According to the application, the features of the salient region are effectively extracted, and the redundant features are effectively filtered.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a salient target detection method based on attention mechanism and multi-scale feature fusion. Background Technology

[0002] Saliency object detection aims to identify the most visually appealing objects in an image and has become an indispensable part of computer vision as a preprocessing step in many tasks. For example, saliency object detection is used in downstream tasks such as weakly supervised semantic segmentation, visual tracking, image retrieval, video segmentation, and content-aware image cropping and editing. However, many current saliency object detection networks do not pay enough attention to salient regions and extract redundant features, affecting detection accuracy. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide a salient target detection method based on an attention mechanism and multi-scale feature fusion, which effectively extracts features from salient regions and effectively filters redundant features. To achieve the above-mentioned objectives and other advantages of the present invention, a salient target detection method based on an attention mechanism and multi-scale feature fusion is provided, comprising:

[0004] S1. Perform data preprocessing to construct the dataset and labels required for training and testing deep learning models;

[0005] S2. Construct a salient object detection network based on attention mechanism and multi-scale feature fusion;

[0006] S3. Input the training dataset from S1 into the salient object detection network constructed in S2 for training to obtain the salient object detection model;

[0007] S4. Randomly select an image and input it into the model to obtain the detection results.

[0008] Preferably, the construction of the salient object detection network in step S2 includes constructing a feature extraction module, convolving the input original image, normalizing the output after convolution, and linearly correcting the normalized result to obtain a feature map with dimensions C×H×W.

[0009] Construct a channel attention module;

[0010] An encoder and decoder based on an attention mechanism are constructed, and the encoder and decoder have the same structure;

[0011] Construct a U-shaped structure based on the attention gating mechanism, and build encoders and decoders with depths of 7, 6, 5, 4 and 3 respectively.

[0012] Preferably, the feature map with dimension C×H×W extracted after the image is processed by convolution is named A. The construction of the channel attention module includes the following steps: A is reshaped into two C×N (N=H×W) dimensional matrices, denoted by B and C respectively. D is the same as A. D is reshaped and transposed into an N×C dimensional matrix, named D'.

[0013] Multiplying matrices C and D', followed by a softmax function, yields a C×C matrix X. The formula for generating X is: Where, x ji It measures the impact of the i-th channel on the j-th channel;

[0014] X is then multiplied by B to obtain a C×N matrix, which is then reshaped into a C×H×W matrix.

[0015] The matrix is ​​added to the original feature map A element by element to obtain a new feature map with channel attention weights. The final feature of each channel is the weighted sum of all channel features and the original feature.

[0016] Preferably, constructing a U-shaped structure based on an attention gating mechanism includes arranging encoders of layers 7, 6, 5, 4, and 3 in a top-down path and linking them in the middle through downsampling; arranging decoders of layers 3, 4, 5, 6, and 7 in a bottom-up path; connecting the two paths through the encoders of layer 3; the input of each decoder is the feature that has been filtered for redundant features by the attention gating mechanism and the feature that has been upsampled from the previous layer; the attention gating mechanism receives the feature output from the previous layer and the feature output from the encoder of the corresponding layer.

[0017] Preferably, the U-shaped structure contains encoders and decoders of different depths, corresponding to feature information of different scales. The output of the encoder of each layer is convolved and then bilinearly interpolated to obtain the saliency map of each layer. The U-shaped structure has 6 layers, so 6 different saliency maps are obtained. The final result is obtained by fusing these 6 saliency maps.

[0018] Preferably, in step S1, the publicly available dataset DUTS-TR in the field of salient object detection is downloaded. The DUTS-TR dataset includes 10,553 PNG images and corresponding JPG salient maps. The PNG images are used as data, and the JPG salient maps are used as labels. Data augmentation is performed on these images by horizontal flipping and other operations, expanding the dataset to 21,106 images. The 21,106 images and their corresponding salient maps are used as the training dataset.

[0019] Compared with existing technologies, the advantages of this invention are: through the encoder and decoder structure, this module adaptively allocates the weights of channel information, making features more concentrated in salient regions. The U-shaped structure based on the attention gating mechanism can effectively filter redundant features. Attached Figure Description

[0020] Figure 1 A flowchart of the salient target detection method based on attention mechanism and multi-scale feature fusion according to the present invention;

[0021] Figure 2 This is a diagram of the training dataset for constructing the salient object detection method based on attention mechanism and multi-scale feature fusion according to the present invention.

[0022] Figure 3 A diagram illustrating the channel attention mechanism of the salient target detection method based on attention mechanism and multi-scale feature fusion according to the present invention;

[0023] Figure 4 The diagram shows the encoder and decoder structure of the attention-based mechanism-based salient target detection method based on attention mechanism and multi-scale feature fusion according to the present invention.

[0024] Figure 5 A diagram illustrating the attention gating mechanism of the salient target detection method based on attention mechanism and multi-scale feature fusion according to the present invention;

[0025] Figure 6 The diagram shows the U-shaped structure of the attention gating mechanism in the salient target detection method based on attention mechanism and multi-scale feature fusion according to the present invention.

[0026] Figure 7 The image shows the detection results of the salient target detection method based on attention mechanism and multi-scale feature fusion according to the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Reference Figure 1-7A salient object detection method based on attention mechanism and multi-scale feature fusion includes: Step 1, downloading the publicly available dataset DUTS-TR dataset in the field of salient object detection, wherein the DUTS-TR dataset includes 10553 PNG format images and corresponding JPG format salient maps of the images, using the PNG format images as data and the JPG format salient maps as labels; performing data augmentation operations such as horizontal flipping on these images to expand the dataset to 21106 images; using these 21106 images and the corresponding salient maps as training data and labels as input to the salient object detection network constructed by S2 for training.

[0029] Step 2: Construct an attention-based encoder and decoder. First, build a feature extraction module. The original input image is convolved, and the output is normalized. The normalized result is then linearly corrected to obtain a feature map of dimension C×H×W. Second, build a channel attention module. The C×H×W feature map extracted after convolution is named A. Step 1: Reshape A into two C×N (N=H×W) dimensional matrices, denoted as B and C. D is the same as A; it is reshaped and transposed to form an N×C dimensional matrix, named D'. Overall, as follows... Figure 2 As shown. The second step: Multiply matrices C and D', followed by a softmax function, to obtain a C×C dimensional matrix X. The formula for generating X is: Where, x ji This measures the influence of the i-th channel on the j-th channel. The third step: Multiply X by B to obtain a C×N matrix, then reshape this matrix into a C×H×W matrix. The fourth step: Finally, add the matrix obtained in the third step to the original feature map A element-wise to obtain a new feature map with channel attention weights, namely E. The formula for generating E is: Thus, the final feature of each channel is the weighted sum of all channel features and the original feature. Finally, an encoder and decoder based on an attention mechanism are constructed. The encoder and decoder have the same structure, but their functions in the network structure are different. The input original image undergoes multiple feature extractions and downsampling, followed by feature fusion. The input to the feature fusion module is the feature from the previous stage and the feature output from the feature extraction at the corresponding scale. After processing by the feature fusion module, it is input to the feature extraction module again. The output from the feature extraction module is then input to the channel attention module, and finally, upsampling is performed. This process is repeated until the feature dimension is the same as the input feature dimension. Constructing this module at different depths changes the number of times feature extraction and downsampling are repeated. Because different depths result in different receptive fields, feature information at different scales can be extracted.

[0030] Step 3: Construct a U-shaped structure based on an attention gating mechanism. Construct encoders and decoders with depths of 7, 6, 5, 4, and 3 layers, respectively, based on the attention mechanism. The encoders of layers 7, 6, 5, 4, and 3 are arranged in a top-down path and linked by downsampling. The decoders of layers 3, 4, 5, 6, and 7 are arranged in a bottom-up path. The two paths are connected by the 3-layer encoder. The input to each decoder consists of features filtered for redundancy by the attention gating mechanism and features upsampled from the previous layer. The attention gating mechanism receives features from the previous layer and features from the corresponding encoder layer. The working principle of the attention gating mechanism is as follows: First, convolve the corresponding output of the encoder to obtain A; then convolve the output from the previous layer to obtain B; add A and B to obtain C. Adding is to highlight features; if a point exists in both images, adding them together will make it more prominent; linearly correct C to obtain D; convolve D to reduce the dimensionality to 1 channel to obtain E; apply a sigmoid function to E so that the value falls within the 0-1 range, with larger values ​​indicating greater importance. The result is that the attention weight is multiplied by the output of the previous layer. Through this series of operations, we can suppress irrelevant areas in the input image while highlighting the salient features of specific local areas.

[0031] Step 4: Construct a U-shaped structure based on the attention gating mechanism. Since the U-shaped structure contains encoders and decoders of different depths, corresponding to feature information of different scales, the output of the encoder of each layer is convolved and then bilinearly interpolated to obtain the saliency map of each layer. Since there are 6 layers in the U-shaped structure, 6 different saliency maps are obtained. The final result is obtained by fusing these 6 saliency maps.

[0032] Step 5: Input the training dataset in S1 into the salient object detection network constructed in S2 for training to obtain the salient object detection model. Randomly select images and input them into the model to obtain the detection results.

[0033] The number of devices and processing scale described herein are for the purpose of simplifying the description of the invention, and applications, modifications and variations thereof will be apparent to those skilled in the art.

[0034] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A salient target detection method based on attention mechanism and multi-scale feature fusion, characterized in that, Includes the following steps: S1. Perform data preprocessing to construct the dataset and labels required for training and testing deep learning models; S2. Construct a salient object detection network based on attention mechanism and multi-scale feature fusion; The construction of the salient object detection network includes building a feature extraction module, which convolves the input original image, normalizes the output after convolution, and performs linear correction on the normalized result to obtain a feature map with dimensions C×H×W. Construct a channel attention module; An encoder and decoder based on an attention mechanism are constructed, and the encoder and decoder have the same structure; A U-shaped structure based on an attention gating mechanism is constructed, with encoders and decoders of depths of 7, 6, 5, 4, and 3 layers respectively. The construction of the U-shaped structure involves arranging the encoders of layers 7, 6, 5, 4, and 3 in a top-down path and linking them via downsampling, and arranging the decoders of layers 3, 4, 5, 6, and 7 in a bottom-up path. The two paths are connected by a 3-layer encoder. The input to each decoder consists of features filtered for redundancy by the attention gating mechanism and features upsampled from the previous layer. The attention gating mechanism receives features output from the previous layer and features output from the encoder of the corresponding layer. S3. Input the training dataset from S1 into the salient object detection network constructed in S2 for training to obtain the salient object detection model; S4. Randomly select an image and input it into the model to obtain the detection results.

2. The salient target detection method based on attention mechanism and multi-scale feature fusion as described in claim 1, characterized in that, The feature map with dimensions C×H×W extracted after the image is processed by convolution is named A. The channel attention module includes the following steps: A is reshaped into two C×N (N = H×W) dimensional matrices, denoted by B and C respectively. D is the same as A. D is reshaped and transposed into an N×C dimensional matrix, named D'. Multiplying matrices C and D', followed by a softmax function, yields a C×C matrix X. The formula for generating X is: ; in, It is a measure of the first i The first channel is for the first j The impact of each channel; X is then multiplied by B to obtain a C×N matrix, which is then reshaped into a C×H×W matrix. The matrix is ​​added to the original feature map A element by element to obtain a new feature map with channel attention weights. The final feature of each channel is the weighted sum of all channel features and the original feature.

3. The salient target detection method based on attention mechanism and multi-scale feature fusion as described in claim 1, characterized in that, The U-shaped structure contains encoders and decoders of different depths, corresponding to feature information of different scales. The output of the encoder of each layer is convolved and then bilinearly interpolated to obtain the saliency map of each layer. The U-shaped structure has 6 layers, so 6 different saliency maps are obtained. The final result is obtained by fusing these 6 saliency maps.

4. The salient target detection method based on attention mechanism and multi-scale feature fusion as described in claim 1, characterized in that, In step S1, download the DUTS-TR dataset, a public dataset in the field of salient object detection. The DUTS-TR dataset includes 10,553 PNG images and corresponding JPG salient maps. The PNG images are used as data, and the JPG salient maps are used as labels. Data augmentation was performed by horizontally flipping these images, expanding the dataset to 21,106 images. These 21,106 images and their corresponding saliency maps were then used as the training dataset.

Images