A salient object detection method based on adaptive feature fusion

Abstract

The application discloses a salient object detection method based on adaptive feature fusion. First, a channel-space attention module (CSA) is added to each side output of the residual backbone network to solve the problem that the feature extraction network is difficult to focus on the salient target area. Second, an adaptive feature module (AFFM) is used to construct a new feature fusion strategy, which connects the residual learning strategy and the multi-scale unified fusion strategy, and solves the problem that it is difficult to clearly predict the salient target boundary after multi-scale feature fusion. Then, a new fusion loss is adopted, which combines binary cross entropy (BCE), intersection over union (IOU), structural similarity (SSIM) and dice loss (Dice), so that the salient target can be more accurately detected and segmented. Finally, an end-to-end deep learning method is used for model training, which reduces the training time and improves the training effect. Through experiments on the data set, the effectiveness of the method is verified, the ability to solve the salient object detection task is improved, and excellent experimental results are obtained.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and image processing, and in particular, relates to a salient object detection method based on adaptive feature fusion. BACKGROUND

[0002] The statements in this section merely refer to the background art of the present application and do not necessarily constitute the prior art.

[0003] Generally, visual saliency can be defined as the region or object that can attract people's interest and attention most in human vision; and the detection and segmentation of salient objects in a natural scene can be referred to as salient object detection.

[0004] Salient object detection can be used as an important image preprocessing process for many applications, such as image auxiliary description, scene classification of high-resolution satellite images under unsupervised feature learning, unsupervised video object segmentation, image and video compression, video summary, photo collage / video repositioning / cropping / thumbnail, image quality evaluation, image segmentation, content-based image retrieval and image set browsing, image editing and operation, visual tracking, object discovery, and human-computer interaction, etc.

[0005] However, detecting salient objects requires semantic understanding of the entire image and detailed structure of the object. Therefore, salient detection is still an important and challenging basic problem in computer vision.

[0006] Current deep convolutional neural networks, such as VGG and ResNet, have shown application prospects in the field of computer vision. However, when ResNet is applied to medical image segmentation, scene segmentation and other dense prediction tasks, there are difficult problems to reconcile. As the depth of the network deepens, when the features are output from the last convolutional layer, the spatial size of the feature map will be reduced by multiple times due to the step in multiple convolution operations. Therefore, the salient object detection method based on early neural network learning method uses superpixels or image patches, and such a learning framework cannot fully utilize high-level semantic information, and spatial information cannot be propagated to the last fully connected layer, resulting in loss of global information.

[0007] These problems have prompted people to introduce fully convolutional networks to dense prediction tasks, but at the same time we also face the problem of difficulty in recovering the target boundary for the up-sampling operation of the feature map. SUMMARY

[0008] In order to alleviate the above problems, the present application provides a salient object detection method based on adaptive feature fusion.

[0009] The present application provides the following technical solutions:

[0010] A salient object detection method based on adaptive feature fusion, characterized by comprising the following steps:

[0011] Step 1: image preprocessing process; using random horizontal or vertical flip operation, then randomly rotating all images, and finally randomly changing the brightness, contrast, saturation and hue of the image, so as to overcome the overfitting of the model and enhance the generalization prediction ability of the model.

[0012] Step 2: feature extraction; first scale the image to be fed to a uniform scale, then feed it into a deep residual backbone network, and after passing through multiple layers of residual backbone network calculation and processing, five scale feature maps are formed, which are then sent into a channel-space attention module (CSA).

[0013] Step 3: apply attention mechanism; at each scale, apply an attention module to each of the five feature maps, and the attention module calculates the input feature map in the channel and spatial dimensions, so that the network can focus more on the salient object region, and then use the DCCP, ARM to realize the step-by-step residual learning and fusion process of the five scale feature maps, and finally feed them into the AFFM.

[0014] Step 4: adaptive feature fusion; after the attention calculation of the five feature maps, the corresponding weights at the five scales are generated by 1x1 convolution in the adaptive feature fusion module, then the scales are unified, and the final prediction output feature map is obtained by element-wise addition.

[0015] The salient object detection method based on adaptive feature fusion according to claim 1, wherein the specific method of step 1) is: first, organize the dataset for salient object detection, and in this method, the ECSSD dataset, DUT-OMRON dataset, DUT-S dataset, HKU-IS dataset and PASCAL-S dataset are used for subsequent implementation and data processing; then use opencv and the image processing library PIL of Python to perform image preprocessing.

[0016] The salient object detection method based on adaptive feature fusion according to claim 1, wherein the residual backbone network uses ResNet-101 network, which uses deep residual network to extract rich feature information of the image, including high-level semantic information and low-level target boundary information, and then generates multi-scale feature maps for subsequent neural network learning of the model.

[0017] According to claim 1, a salient target detection method based on adaptive feature fusion is characterized in that the specific method of step 3) is: first, the five-scale feature maps generated by the backbone network are fed into the CSA module for calculation, and then the DCPP module is used to perform a dilated convolution calculation on the feature map of the fifth scale to generate a rough global feature map, and then the ARM module is used to perform residual learning on the five feature maps step by step, and gradually restore them to the scale of the original input from the bottom up, and finally feed them into the adaptive feature fusion module (AFFM).

[0018] The CSA module can be defined as the following formula:

[0019] (1)

[0020] (2)

[0021] (3)

[0022] (4)

[0023] (5)

[0024] where s c represents the spatial attention mechanism, s s Represents the channel attention mechanism, W represents the learnable kernel weight, GAP represents the global average pooling operation, here we use the nn.AdaptiveAvgPool2d method for global average pooling operation, f represents the fusion function, here we use the element-by-element addition operation; σ represents the sigmoid activation function, and δ represents the ReLU activation function.

[0025] According to claim 1, a salient target detection method based on adaptive feature fusion is characterized in that the specific method of step 4) is: using 1×1 convolution to generate corresponding weights for the feature maps of five scales, then multiplying the original feature maps by the corresponding weights, and finally unifying the scales and adding element by element to obtain the final predicted output map.

[0026] We use The feature tensor representing the position (i, j) of the feature map is scaled from scale n to S. Then the adaptive feature fusion process can be defined as:

[0027] (6)

[0028] in Represents the output feature map The (i, j)th tensor feature of ; Refers to the weight values ​​of the spatial dimensions of the feature map at five scales. These weights can be adaptively learned by the network. The entire network uses a hybrid loss function for supervised learning. This loss function calculates and compares the features at five scales, such as output, and supervises the predicted output after the adaptive feature fusion module.

[0029] The loss function can be defined as:

[0030] (7)

[0031] in represents the cross entropy loss function, represents the structural similarity loss function, represents the dice loss function, Represents the intersection loss function. These loss functions can be defined as follows:

[0032] (8)

[0033] where y i refers to the label value, x i Refers to the predicted value, y i ,x i ∈ [0,1];

[0034] (9)

[0035] in TP represents the proportion of true positive samples, FN represents the proportion of false negative samples, FP represents the proportion of false positive samples;

[0036] (10)

[0037] in μ x , μ y , σ x , σ y They are x, y The mean and standard deviation of σ xy for x, y Covariance, C1=0.01 2 and C2=0.03 2 Used to avoid division by zero in formulas;

[0038] (11)

[0039] inTP a proportion of true positive samples, FN a proportion of false negative samples, FP a proportion of false positive samples.

[0040] Compared with the prior art, the present application has the following advantages and characteristics.

[0041] 1. The method of the present application proposes a channel-space attention module to help the model focus on the salient target area, which is conducive to the model learning of multi-scale salient features and improves the detection performance.

[0042] 2. The present application designs an adaptive feature fusion module to alleviate the loss of detailed information when the multi-scale features are up-sampled and restored, and at the same time improves the utilization effect of multi-scale feature information, so that the model can learn more fully the feature information and make better salient target prediction.

[0043] 3. The present application designs a hybrid loss function, which combines the characteristics of four loss functions, and gives the model supervised learning at global and local scales, which improves the ability of the model to predict the salient target boundary and locate the salient target position. DETAILED DESCRIPTION

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can be obtained without creative labor on the basis of these drawings.

[0045] Figure 1 Overall architecture diagram of network model.

[0046] Figure 2 Channel-space attention module.

[0047] Figure 3 Cave convolution module.

[0048] Figure 4 Attention residual module.

[0049] Figure 5 Adaptive feature fusion module.

[0050] Figure 6 Comparison chart of prediction results.

[0051] Figure 7 Data comparison table. DETAILED DESCRIPTION

[0052] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the examples of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0053] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0054] The overall framework of the model of the present application is shown in Figure 1, which is a salient object detection method based on adaptive feature fusion. The specific operation is described as follows.

[0055] Step 1: image preprocessing process; using random horizontal or vertical flip operation, then randomly rotating all images, and finally randomly changing the brightness, contrast, saturation and hue of the image, so as to overcome the overfitting of the model and enhance the generalization prediction ability of the model.

[0056] Step 2: feature extraction; first scale the image to be fed into a uniform scale, then feed into a deep residual backbone network, form five scale feature maps after multi-layer residual backbone network calculation and processing, then respectively send into a channel-space attention module (CSA).

[0057] Step 3: applying attention mechanism; on each scale, five feature maps are respectively applied to the attention module, which respectively calculates the input feature map in the channel and spatial dimension, so that the network can focus more on the salient target area, and then uses the DCCP (Dilated Convolution Pyramid Pooling) module, the ARM (Attention Residual Module) to realize the step-by-step residual learning and fusion process of the five scale feature maps, and finally feed them into the AFFM (Adaptive Feature Fusion Module).

[0058] Step 4: adaptive feature fusion; after the attention calculation of the five feature maps, the corresponding weights on the five scales are generated by 1x1 convolution in the adaptive feature fusion module, then the scales are unified, and the final prediction output feature map is obtained by element-wise addition.

[0059] The specific process of step 1 is as follows:

[0060] Firstly, the data set for salient object detection is sorted, and the ECSSD data set, DUT-OMRON data set, DUT-S data set, HKU-IS data set and PASCAL-S data set are used for subsequent implementation and data processing; then the image preprocessing work is carried out by using opencv and the image processing library PIL of Python.

[0061] The specific process of step 2 is:

[0062] The residual backbone network adopts ResNet-101 network, and the deep residual network is used to extract rich feature information of the image, including high-level semantic information and low-level target boundary information, and then a multi-scale feature map is generated for neural network learning of the subsequent model.

[0063] The specific process of step 3 is:

[0064] As shown in Figure 1 , first, the five scale feature maps generated by the backbone network are fed into the CSA module for calculation, then the DCPP module is used for the fifth scale feature map to perform the calculation of the hollow convolution, to generate the rough global feature map, and then the ARM module is used to gradually learn the residual of the five feature maps, from bottom to top, gradually recovering to the size of the original input, and finally fed into the adaptive feature fusion module (AFFM), the framework of the three types of CSA, DCPP and ARM modules is as shown in Figure 2 、 Figure 3 、 Figure 4 .

[0065] The CSA module can be defined as the following formula:

[0066] (1)

[0067] (2)

[0068] (3)

[0069] (4)

[0070] (5)

[0071] Where s c represents the spatial attention mechanism, s s represents the channel attention mechanism, W represents the learnable kernel weight, GAP represents the global average pooling operation, here we use the nn.AdaptiveAvgPool2d method for global average pooling operation, f represents the fusion function, here we use the element-wise addition operation; sigma represents the sigmoid activation function, and delta represents the ReLU activation function.

[0072] The specific process of step 4 is as follows:

[0073] like Figure 5 As shown in , 1×1 convolution is used to generate the corresponding weights for the feature maps of the five scales, and then the original feature map is multiplied by the corresponding weights. Finally, the scale is unified and element-by-element addition is performed to obtain the final prediction output map. We use The feature tensor representing the position (i, j) of the feature map is scaled from scale n to S. Then the adaptive feature fusion process can be defined as:

[0074] (6)

[0075] in Represents the output feature map The (i, j)th tensor feature of ; Refers to the weight values ​​of the spatial dimensions of the feature map at five scales. These weights can be adaptively learned by the network. The entire network uses a hybrid loss function for supervised learning. This loss function calculates and compares the features at five scales, such as output, and supervises the prediction output of the adaptive feature fusion module. The loss function is defined as:

[0076] (7)

[0077] in represents the cross entropy loss function, represents the structural similarity loss function, represents the dice loss function, Represents the intersection loss function. These loss functions can be defined as follows:

[0078] (8)

[0079] where y i refers to the label value, x i Refers to the predicted value, y i ,x i ∈ [0,1];

[0080] (9)

[0081] in TP represents the proportion of true positive samples, FN represents the proportion of false negative samples, FP represents the proportion of false positive samples;

[0082] (10)

[0083] wherein μ x , μ y , σ x , σ y respectively are x, y the average and standard deviation, σ xy is x, y the covariance, C1=0.01 2 and C2=0.03 2 for avoiding formula division by zero;

[0084] (11)

[0085] wherein TP represents the true positive sample proportion, FN represents the false negative sample proportion, FP represents the false positive sample proportion.

[0086] In order to verify the high efficiency and model generalization ability of the present application, a series of experiments are also carried out, including qualitative and quantitative comparison with other related methods; as shown in Figure 6 , the present application obtains the best prediction result compared with other methods, and can obtain more fine target boundary result. As shown in Figure 7 , the present application is obviously higher than other methods in most indexes on multiple data sets, and excellent prediction result is obtained.

[0087] The above only describes the preferred embodiments of the present application and is not used to limit the present application, and the present application can have various changes and variations for those skilled in the art; any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0088] The contents not described in detail in the specification of the present application belong to the prior art known by those skilled in the art.

Claims

1. A salient object detection method based on adaptive feature fusion, characterized in that: The following steps are involved: Step 1: Image preprocessing process; Use random horizontal or vertical flip operations, then randomly rotate all images, and finally randomly change the brightness, contrast, saturation, and hue of the images to overcome the overfitting of the model and enhance the generalization prediction ability of the model; Step 2: Feature extraction: First, the image to be fed is scaled to a uniform scale and then fed into a deep residual backbone network. After multi-layer residual backbone network calculations and processing, feature maps of five scales are formed and then fed into the channel-spatial attention module (CSA). Step 3: Apply attention mechanism; At each scale, an attention module is applied to the five feature maps. The attention module calculates the fed feature maps in the channel and spatial dimensions, allowing the network to focus more on the salient target area. Subsequently, the dilated convolutional pyramid pooling module (DCPP) and the attention residual module (ARM) are used to implement the step-by-step residual learning fusion process for the five scale feature maps, and finally they are fed into the adaptive feature fusion module (AFFM). Step 4 : Feature adaptive fusion; After the five feature maps are calculated through attention, the corresponding weights on five scales are generated through 1×1 convolution in the adaptive feature fusion module, and then the scales are unified and element-by-element addition is performed to obtain the final predicted output feature map.

2. The method for salient object detection based on adaptive feature fusion according to claim 1, characterized in that: The specific method of step 1) is: First, the datasets for salient object detection are organized. This method uses the ECSSD dataset, DUT-OMRON dataset, DUT-S dataset, HKU-IS dataset, and PASCAL-S dataset for subsequent implementation and data processing; then, OpenCV and Python's image processing library PIL are used for image preprocessing.

3. The method for salient object detection based on adaptive feature fusion according to claim 1, characterized in that: The specific method of step 2) is: The residual backbone network adopts the ResNet-101 network and uses a deep residual network to extract rich feature information of the image, including high-level semantic information and low-level object boundary information, and then generates multi-scale feature maps for neural network learning of subsequent models.

4. The method for salient object detection based on adaptive feature fusion according to claim 1, characterized in that: The specific method of step 3) is: First, the five-scale feature maps generated by the backbone network are fed into the CSA module for calculation. Then, the DCPP module is used to perform dilated convolution calculation on the fifth-scale feature map to generate a rough global feature map. The ARM module is then used to perform residual learning on the five feature maps step by step, gradually restoring them to the scale of the original input from the bottom up, and finally feeding them into the adaptive feature fusion module (AFFM). The CSA module can be defined as the following formula: (1) (2) (3) (4) (5) where s c represents the spatial attention mechanism, s s represents the channel attention mechanism, W represents the learnable kernel weight, GAP represents the global average pooling operation, here we use the nn.AdaptiveAvgPool2d method for global average pooling operation, f represents the fusion function, here we use the element-by-element addition operation; σ represents the sigmoid activation function, and δ represents the ReLU activation function.

5. The method for salient object detection based on adaptive feature fusion according to claim 1, characterized in that: The specific method of step 4) is: use 1×1 convolution to generate the corresponding weights for the feature maps of the five scales, then multiply the original feature map with the corresponding weights, and finally unify the scale and add element by element to obtain the final prediction output map; we use The feature tensor representing the position (i, j) of the feature map is scaled from scale n to S. Then the adaptive feature fusion process can be defined as: (6) in Represents the output feature map The (i, j)th tensor feature of ; Refers to the weight values ​​of the spatial dimensions of the feature map at five scales. These weights can be adaptively learned by the network. The entire network uses a hybrid loss function for supervised learning. This loss function calculates and compares the features at five scales, such as output, and supervises the prediction output of the adaptive feature fusion module. The loss function is defined as: (7) in represents the cross entropy loss function, represents the structural similarity loss function, represents the dice loss function, Represents the intersection loss function. These loss functions can be defined as follows: (8) where y i refers to the label value, x i Refers to the predicted value, y i ,x i ∈ [0,1]; (9) in TP represents the proportion of true positive samples, FN represents the proportion of false negative samples, FP represents the proportion of false positive samples; (10) in μ x ,μ y ,σ x ,σ y They are x, y The mean and standard deviation of σ xy for x, y Covariance, C1=0.01 2 and C2=0.03 2 Used to avoid division by zero in formulas; (11) in TP represents the proportion of true positive samples, FN represents the proportion of false negative samples, FP Represents the proportion of false positive samples.

Images