A deep learning method for salient region detection

Abstract

The present invention provides a deep learning detection method for salient regions, comprising constructing a multi-scale deep network, and comprising the following steps: training the multi-scale deep network; saliency prediction and fusion. The present invention proposes a deep learning detection method for salient regions, using a multi-scale deep network for salient region detection, utilizing the inherent hierarchical features of the deep network, inputting RGB images into a deep convolutional neural network, and predicting feature maps under different scale spaces through bottom-up feature extraction and top-down feature integration.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a deep learning detection method for salient areas. Background Art

[0002] The goal of saliency detection is to find salient regions in an image, which is a fundamental problem in image understanding and analysis, and is often used as a preprocessing for other computer vision problems, such as image classification, image segmentation, and relocalization. Although saliency detection has made great progress in recent years, saliency remains a challenging task.

[0003] Traditional saliency detection methods usually use a variety of saliency cues or prior information, such as local or global contrast, boundary priors. These methods cannot recognize and understand the semantic object concepts in images because they use low-level artificially designed features and models. Recently, deep convolutional neural networks have achieved remarkable results in visual pattern recognition methods, such as semantic segmentation, image classification, object tracking, etc. As long as enough training data is provided, deep convolutional neural networks can still accurately identify salient objects in complex images, and their performance exceeds that of most traditional methods based on artificially designed features. Despite the good detection results, there are still problems that deserve further study.

[0004] Multi-scale saliency features have been successfully used in traditional salient region detection based on artificially designed features. However, in CNN-based methods, multiple deep networks need to be trained to obtain multi-scale feature extraction. Although good performance has been achieved, the computational and storage costs are huge, and it is not very practical in practical applications.

[0005] The invention patent application with application number CN104408711A discloses a salient region detection method based on multi-scale region fusion, and its technical features are: firstly, the optimal color gradient and normalized gradient of the image are calculated; then the gradient index is used to traverse the image, and the pixel pairs that meet the fusion threshold are merged to obtain the label image of the first scale; then the edge gradient map of the label image is calculated; then the multi-scale label map is iteratively calculated at each scale; then the multi-scale candidate salient map is obtained by using mutual information; finally, the optimal scale is found by using spatial geometric information entropy, and the candidate salient map at this scale is the final salient map. Although this method takes multi-scale factors into consideration in the extraction of salient regions, this method ultimately selects a salient map at one scale as the final salient map, and does not actually fuse salient maps at multiple scales. Summary of the invention

[0006] In order to solve the above technical problems, the present invention proposes a deep learning detection method for salient areas, and proposes to use a multi-scale deep network for salient area detection. The inherent hierarchical features of the deep network are utilized, and the RGB image is input into a deep convolutional neural network. Through bottom-up feature extraction and top-down feature integration, the feature maps under different scale spaces are predicted and fused.

[0007] The present invention provides a deep learning detection method for salient regions, comprising constructing a multi-scale deep network, characterized in that the method comprises the following steps:

[0008] Step 1: training the multi-scale deep network;

[0009] Step 2: Saliency prediction and fusion.

[0010] Preferably, the constructing of a multi-scale deep network comprises the following sub-steps:

[0011] Step 01: Bottom-up feature extraction;

[0012] Step 02: Top-down feature connection;

[0013] Step 03: Weight prediction of salient region size.

[0014] In any of the above solutions, preferably, step 01 includes inputting an image into the network, and the feature extraction module performs feature extraction from bottom to top to obtain multi-scale features.

[0015] In any of the above schemes, preferably, step 01 also includes using a 16-layer VGG network, the feature extraction module includes 13 convolutional layers, and nonlinear mapping and maximum pooling operations are performed through a ReLU linear correction unit to obtain features of four scales represented as {F2; F3; F4; F5}.

[0016] In any of the above solutions, preferably, step 02 includes connecting features at different levels, as shown below:

[0017]

[0018] Among them, d k represents a deconvolution operation with a kernel size of 4x4 and a stride of 2, f k Represents a 1x1 convolution operation. represents the feature map, F k-1 Represents the features of the k-1th layer of the VGG deep network structure, F k Represents the features of the kth layer in the VGG network.

[0019] In any of the above solutions, preferably, step 03 includes connecting the output of relu5_3 of VGG16 with a global average pooling layer to transform F5 into a 512-dimensional vector.

[0020] In any of the above solutions, preferably, step 03 further includes generating two full convolutional layers, each having 256 and 3 units, to generate a 3-dimensional weight vector in, and Indicates the weights corresponding to salient regions of different sizes.

[0021] In any of the above solutions, preferably, step 03 also includes performing a dropout operation after the first fully connected layer to prevent overfitting.

[0022] In any of the above solutions, preferably, step 03 also includes the output layer further normalizing the result to between [0, 1] through a sigmoid operation.

[0023] In any of the above schemes, preferably, step 1 includes the following sub-steps:

[0024] Step 11: Design the objective function of the saliency map deep learning prediction network training;

[0025] Step 12: Design the objective function for training the salient region size weight network;

[0026] Step 13: Design the objective function for the entire deep network training.

[0027] In any of the above solutions, preferably, step 11 includes training the image set X and the corresponding reference binary annotation set S, by minimizing the objective function L S To train the network, the formula is:

[0028]

[0029] Where (x, y) represents the coordinates of the input image.

[0030] In any of the above solutions, preferably, step 12 includes finding all non-connected salient regions in the benchmark binary annotation set S by using the Moore-Neighbor tracking method, and representing these regions as Where N represents the number of detected salient regions.

[0031] In any of the above schemes, preferably, the step 12 further comprises for each significant O (i) ∈O area, calculate the ratio r of its area to the entire image (i) .

[0032] In any of the above solutions, preferably, the step 12 further includes obtaining after the calculation is completed Vector ω i Each component of is set to

[0033]

[0034] in, and They represent the minimum and maximum thresholds of the ratio of the salient area to the entire image.

[0035] In any of the above solutions, preferably, the step 12 further includes defining a loss function of weight prediction according to the labeling result ω of the size of the salient region as follows:

[0036]

[0037] in, is the predicted value.

[0038] In any of the above solutions, preferably, step 13 includes setting the weighted predicted loss value L S and L ω The significant prediction loss values ​​are integrated together as the loss function of the entire deep network, and the formula is:

[0039] L=L S +αL ω

[0040] Among them, the α parameter is used to balance the two loss values.

[0041] In any of the above schemes, preferably, step 04 includes using enhanced features of different scales and Each of the three scales is independently predicted to obtain saliency maps.

[0042] In any of the above solutions, preferably, the calculation formula of the saliency map is:

[0043]

[0044] Among them, f max It is the maxout function, which performs the maximum pooling operation on each position of the multi-scale feature map.

[0045] The present invention proposes a deep learning detection method for salient areas. On the one hand, it utilizes the inherent hierarchical characteristics of the deep network and obtains multi-scale features through only one deep network, without increasing the complexity of the network and having a simple network structure. On the other hand, this method takes into account that salient areas of different sizes should have different weights when fused, and the network obtains a weight vector through training, so that the fused salient map is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The flowchart is a preferred embodiment of the deep learning detection method for salient areas according to the present invention.

[0047] Figure 2 For the deep learning detection method of the salient area according to the present invention, Figure 1 Flowchart of the training steps of the illustrated embodiment.

[0048] Figure 3 The flowchart is another preferred embodiment of the deep learning detection method for salient areas according to the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0050] Embodiment 1

[0051] like Figure 1 As shown, execute step 100 to construct a multi-scale deep network. Including, execute step 101, bottom-up feature extraction. An image is input into the network, and the feature extraction module extracts features from the bottom to the top to obtain multi-scale features. A 16-layer VGG network is used, and the feature extraction module includes 13 convolutional layers. Nonlinear mapping and maximum pooling operations are performed through ReLU linear correction units. The features of the four scales are represented as {F2; F3; F4; F5}.

[0052] Execute step 102 to connect features from top to bottom. Connect features at different levels, as shown below: Among them, d k represents a deconvolution operation with a kernel size of 4x4 and a stride of 2, f k Represents a 1x1 convolution operation. represents the feature map, F k-1 Represents the features of the k-1th layer of the VGG deep network structure, F k Represents the features of the kth layer in the VGG network. Execute step 103 to predict the weight of the salient region size. Connect the output of relu5_3 of VGG16 with a global average pooling layer to transform F5 into a 512-dimensional vector. Generate two full convolutional layers with 256 and 3 units respectively, generating a 3-dimensional weight vector in, and Indicates the weights corresponding to salient regions of different sizes. To prevent overfitting, a dropout operation is performed after the first fully connected layer. The output layer further normalizes the result to [0,1] through a sigmoid operation.

[0053] Execute step 110 to train the multi-scale deep network. Figure 2 As shown, this step includes the following steps:

[0054] Execute step 111 to design the objective function of the saliency map deep learning prediction network training. The training image set X and the corresponding benchmark binary annotation set S are trained by minimizing the objective function L S To train the network, the formula is:

[0055]

[0056] Where (x, y) represents the coordinates of the input image. Execute step 112 to design the objective function of the salient region size weight network training. Use the Moore-Neighbor tracking method to find all non-connected salient regions in the benchmark binary annotation set S, and represent these regions as Where N represents the number of detected salient regions. (i) ∈O, calculate the ratio r of its area to the entire image (i) After the calculation is completed, we get Vector ω i Each component of is set to

[0057]

[0058] in, and The points represent the minimum and maximum thresholds of the proportion of the salient area to the entire image. According to the annotation result ω of the salient area size, the loss function of weight prediction is defined as follows:

[0059]

[0060] in, is the predicted value. Execute step 113 to design the objective function of the entire deep network training. Set the predicted loss value L of the weight S and L ω The significant prediction loss values ​​are integrated together as the loss function of the entire deep network, and the formula is: L = L S +αL ω , where the α parameter is used to balance the two loss values.

[0061] Execute step 120, saliency prediction and fusion. Use enhanced features of different scales and Each of the three scales is independently predicted to obtain saliency maps. The calculation formula of the saliency map is: Among them, f max It is the maxout function, which performs the maximum pooling operation on each position of the multi-scale feature map.

[0062] Embodiment 2

[0063] Multi-scale saliency features have been successfully used in traditional salient region detection based on artificially designed features. However, in CNN-based methods, multiple deep networks need to be trained for multi-scale feature extraction. Although good performance has been achieved, the computational and storage costs are huge, and the practicality is not strong in practical applications. However, recent work has shown that deep convolutional neural networks themselves have a hierarchical structure of multi-scale features. Therefore, the feature maps of sub-sampling convolutional layers in different spaces can themselves be regarded as multi-scale features, and do not need to be constructed through additional network structures. Influenced by this idea, the present invention proposes a multi-scale deep network for salient region detection, which utilizes the inherent hierarchical features of the deep network. The present invention inputs an RGB image into a deep convolutional neural network, and predicts feature maps under different scale spaces through bottom-up feature extraction and top-down feature integration. In addition, the network can also predict the weight values ​​of different salient region sizes, fuse the prediction results of different branches, and obtain the final salient map.

[0064] Recently, saliency detection methods based on deep learning have proven the effectiveness of deep learning features. At present, multiple complex network structures are usually designed to detect salient areas, which results in very high computational and storage costs. However, the network structure of deep learning itself has multi-scale characteristics, and using the multi-scale characteristics of deep learning itself to detect salient areas does not bring additional storage and computational costs. Based on this, the present invention proposes a deep learning detection method for salient areas. In addition, the salient areas in the image have different sizes, so the salient maps of images with different salient area sizes should have different weights when fused, which is also a problem to be solved by the present invention.

[0065] 1. Method flow.

[0066] like Figure 3 As shown, the method proposed in the present invention consists of four parts: bottom-up feature extraction, top-down feature connection, weight prediction of salient region size, and saliency prediction and fusion.

[0067] Bottom-up feature extraction

[0068] An image is input into the network, and the feature extraction module extracts features from the bottom up to obtain multi-scale features. A 16-layer VGG network is used, and the feature extraction module includes 13 convolutional layers, which perform nonlinear mapping and maximum pooling operations through ReLU linear correction units. The features of the four scales are represented as {F2; F3; F4; F5}. The closer to the bottom, the higher the resolution, and the better the detail information and global structure are maintained. On the contrary, the closer to the upper layer of the deep learning network structure, the higher the feature level and the stronger the semantics, but the detail information of the image is also lost. Since the features of the bottom and high layers have their own advantages and disadvantages, they are very important for saliency detection.

[0069] Top-down feature connection

[0070] In a deep learning network, different layers show different saliency clues and have complementary characteristics to a certain extent. In order to make the multi-scale saliency features have stronger representation capabilities, the features at different levels are connected, as shown below:

[0071]

[0072] d k (.) is the deconvolution layer operation, which includes 20 filters, the convolution kernel size is 4x4, and the stride is 2. The feature map is obtained by upsampling and feature Figure 1 The same size. k (.) represents a 1x1 convolution operation, which converts F k-1 As input, the output is a feature map with 20 feature channels.

[0073] As shown in the figure, this process starts with the top feature map F5 of the deep network structure and iterates until the feature map F2 with the finest resolution at the bottom. In this way, the new feature map and It is generated sequentially, not only containing all the information of the original network structure features, but also having more robust saliency clues.

[0074] Prediction of salient region size weights

[0075] Different images have different sizes of salient regions, so it is necessary to design the most suitable scale for salient region detection. The present invention sets the salient region size to be divided into 3 levels. The designed network can predict 3 weights of different salient region sizes based on the input image and assign them to salient maps of 3 scales respectively. First, the output of relu5_3 of VGG16 is connected to a global average pooling layer to transform F5 into a 512-dimensional vector. Next are two full convolutional layers with 256 and 3 units respectively, generating a 3-dimensional weight vector in, and Indicates the weights corresponding to salient regions of different sizes. To prevent overfitting, a dropout operation is performed after the first fully connected layer. The output layer further normalizes the result to [0,1] through a sigmoid operation.

[0076] Prediction and fusion of saliency maps

[0077] The prediction module is responsible for multi-scale saliency prediction. With the enhanced features of different scales and Use these feature maps to independently predict saliency. Obtain saliency maps at three scales: The final saliency map is calculated as follows:

[0078]

[0079] Among them, f max (.) is the maxout function, which performs the maximum pooling operation on each position of the multi-scale feature map.

[0080] 2) Training of multi-scale deep networks

[0081] (1) Designing the objective function for saliency map deep learning prediction network training

[0082] The training image set X and the corresponding benchmark binary annotation set S are used to train the network by minimizing the following objective function:

[0083]

[0084] Where (x, y) represents the coordinates of the input image.

[0085] (2) Design the objective function for training the salient region size weight network

[0086] 1. The present invention sets the benchmark weights of the training image set X by the following method. First, all non-connected salient regions in the benchmark binary annotation set S are found by the Moore-Neighbor tracking method, and these regions are represented as N represents the number of salient regions detected. (i) ∈O, calculate the ratio r of its area to the entire image (i) After the calculation is completed, we get Vector ω i Each component of is set to

[0087]

[0088] in, and They represent the minimum and maximum thresholds of the ratio of the salient area to the entire image, respectively. The threshold settings are shown in Table 1.

[0089]

[0090]

[0091] Table 12. According to the annotation results ω of the salient region size, the loss function of weight prediction is defined as follows:

[0092]

[0093] in, is the predicted value.

[0094] (3) Design the objective function for the entire deep network training

[0095] The weighted predicted loss value L S and L ω The significant prediction loss values ​​are integrated together as the loss function of the entire deep network

[0096] L=L S +αL ω

[0097] Among them, the α parameter is used to balance the two loss values.

[0098] (4) Specific implementation process of deep network training

[0099] The stochastic gradient descent method was used when training the network, with batch size = 256, moment = 0.9, and the initial learning rate was 0.01. When the loss function stabilized, it was decreased at a rate of 0.1. The dropout was set to 0.5 to prevent overfitting. The parameter α in Formula 6 and the parameter λ in Formula 9 were empirically set to 0.1 and 0.01. The caffe framework was used for training and testing, and the computer had an i7-4790 CPU and a TITAN-X GPU. The training process was 80 rounds in total, and each image was tested for 3 seconds.

[0100] In order to better understand the present invention, the above is described in detail in conjunction with the specific embodiments of the present invention, but it is not intended to limit the present invention. Any simple modifications made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention. Each embodiment in this specification focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

Claims

1. A deep learning detection method for salient regions, comprising constructing a multi-scale deep network, characterized in that: The following steps are involved: Step 1: Training the multi-scale deep network, including the following sub-steps: Step 11: Design the objective function of the saliency map deep learning prediction network training, training image set X and corresponding benchmark binary annotation set S, by minimizing the objective function L S To train the network, the formula is: Where (x, y) represents the coordinates of the input image; Step 12: Design the objective function of the salient region size weight network training, find all non-connected salient regions in the benchmark binary annotation set S through the Moore-Neighbor tracking method, and represent these regions as Where N represents the number of detected salient regions; For each salient region O (i) ∈O, calculate the ratio r of its area to the entire image (i) ; After the calculation is completed, we get Vector ω i Each component of is set to in, and They represent the minimum and maximum thresholds of the ratio of the salient area to the entire image; According to the annotation result ω of the salient region size, the loss function of weight prediction is defined as follows: in, is the predicted value, k∈{2,3,4}; Step 13: Design the objective function for the entire deep network training and set the predicted loss value L of the weight S and L ω The significant prediction loss values ​​are integrated together as the loss function of the entire deep network, and the formula is: L=L S +αL ω Among them, the α parameter is used to balance the two loss values; Step 2: Saliency prediction and fusion.

2. The deep learning detection method for salient areas according to claim 1, characterized in that: The construction of a multi-scale deep network includes the following sub-steps: Step 01: Bottom-up feature extraction; Step 02: Top-down feature connection; Step 03: Weight prediction of salient region size.

3. The deep learning detection method for salient areas according to claim 2, characterized in that: The step 01 includes inputting an image into the network, and the feature extraction module extracts features from the bottom up to obtain multi-scale features.

4. The deep learning detection method for salient areas according to claim 3, characterized in that: The step 01 also includes using a 16-layer VGG network, the feature extraction module includes 13 convolutional layers, and nonlinear mapping and maximum pooling operations are performed through the ReLU linear correction unit to obtain features of four scales represented as {F2; F3; F4; F5}.

5. The deep learning detection method for salient areas according to claim 4, characterized in that: The step 02 includes connecting features at different levels, as shown below: Among them, d k represents a deconvolution operation with a kernel size of 4x4 and a stride of 2, f k Represents a 1x1 convolution operation. represents the feature map, F k-1 Represents the features of the k-1th layer of the VGG deep network structure, F k Represents the features of the kth layer in the VGG network.

6. The deep learning detection method for salient areas according to claim 5, characterized in that: The step 03 includes connecting the output of relu5_3 of VGG16 with a global average pooling layer to transform F5 into a 512-dimensional vector.

7. The deep learning detection method for salient areas according to claim 6, characterized in that: The step 03 also includes generating two full convolutional layers, with 256 and 3 units respectively, to generate a 3-dimensional weight vector in, and Indicates the weights corresponding to salient regions of different sizes.

8. The deep learning detection method for salient areas according to claim 7, characterized in that: The step 03 also includes performing a dropout operation after the first fully connected layer to prevent overfitting.

9. The deep learning detection method for salient areas according to claim 8, characterized in that: The step 03 also includes the output layer further normalizing the result to between [0, 1] through a sigmoid operation.

Images