A fully supervised salient target detection method

Abstract

The present application relates to a kind of full supervision's salient object detection method, constructs complete multi-branch feature fusion refinement network MFFRNet as salient object detection model;Again training set in data set is input to the proposed MFFRNet model training, every time completing a round will be back propagated once, to optimize MFFRNet model parameter;With data set test set, the performance of model is evaluated;Finally, the model after evaluation is used for salient object detection.The model effectively fuses the detail information of low-level feature and the semantic information of high-level feature.The module designed for low-level feature utilizes asymmetric convolution to reduce background noise and other interference factors, and a module designed for high-level feature obtains rich semantic information.Meanwhile, aliasing effects caused by frequent up-sampling are effectively handled.The method effectively captures salient objects and obtains saliency prediction map, and has strong robustness.

Description

Technical Field

[0001] This invention relates to a detection technology, and more particularly to a fully supervised method for detecting salient targets. Background Technology

[0002] Salient object detection aims to identify important, salient parts of natural images or videos that contain significant salient information. It lays the foundation for other related applications such as image classification, video compression, and object tracking. Convolutional neural networks (CNNs), with their powerful feature extraction capabilities, have been increasingly repurposed for various visual tasks, and CNN-based salient object detection methods are among the best.

[0003] CNN-based methods for salient object detection have been widely proposed. Many methods have achieved good results through clever network architecture design. However, these methods treat shallow and deep features indiscriminately, leading to unsatisfactory detection results in complex environments. Specifically, shallow features contain more detailed information and can better delineate object boundaries, but are filled with background noise; deep features contain more semantic information and can better locate salient objects, but the object boundaries are coarser. Simply put, shallow and deep features have different distributions and should be treated differently. Furthermore, existing SOD (Salient Object Detection) models have other major problems, such as not considering the negative impact of multiple upsampling on the saliency map. Summary of the Invention

[0004] To address the above problems, a fully supervised salient target detection method is proposed.

[0005] The technical solution of this invention is as follows: a fully supervised salient object detection method, using the publicly available DUTS-TR dataset as the dataset. First, the acquired data is preprocessed and augmented. Then, a complete multi-branch feature fusion refinement network (MFFRNet) is constructed as the salient object detection model. Next, the training set in the dataset is input into the proposed MFFRNet model for training, and backpropagation is performed once after each round to optimize the MFFRNet model parameters. The model performance is evaluated using the test set in the dataset. Finally, the evaluated model is used for salient object detection.

[0006] Furthermore, the method for establishing the Multi-Branch Feature Fusion Refinement Network (MFFRNet) is as follows: MFFRNet includes a Detail Optimization Module (DOM), a Pyramid Feature Extraction Module (PFEM), a Multi-Scale Feature Fusion Module (MFFM), and a Feature Optimization Module (FOM). Dataset images are input into the ResNet-50 backbone network, where they are encoded to generate five sets of feature maps of different sizes. These five sets of feature maps undergo convolution operations of varying numbers to extract features. The second and third sets of feature maps are considered low-level features, while the fourth and fifth sets are considered high-level features. Low-level features are processed by the Detail Optimization Module (DOM) to suppress noise. High-level features are processed by the Pyramid Feature Extraction Module (PFEM) to obtain multi-scale features and rich contextual information. The outputs of the DOM and PFEM modules are fed into the Multi-Scale Feature Fusion Module (MFFM) to aggregate features. The output features of the MFFM module are fed into the FOM module to optimize multi-scale feature fusion. Finally, a 3×3 convolution + batch normalization + ReLU is used to extract features, and a sigmoid activation function is applied to obtain the final prediction map.

[0007] Furthermore, in the Detailed Optimization (DOM) module, firstly, each group of low-level features is duplicated into five identical feature copies, forming five branches. Four of these branches extract features using combinations of 1×1 convolutions, asymmetric convolutions consisting of 1×n and n×1 convolutions, and dilated convolutions with a dilation rate of 3, where n = 1, 3, 5, and 7. Then, the features extracted from the four branches are concatenated and combined, followed by channel compression and further feature extraction using 3×3 convolutions. Finally, residual connections are used to superimpose the input and input processing features to supplement missing information during feature extraction, and the ReLU activation function is used to increase non-linearity and improve the neural network's expressive power over the model.

[0008] Furthermore, in the pyramid feature extraction module PFEM, firstly, each set of high-level features is duplicated into six identical features, forming six branches respectively; the first branch undergoes a 3×3 dilated convolution and a 3×3 ordinary convolution, and the extracted features are used as input, concatenated with the channels of the next branch, and then the same convolution operation is performed again, and so on, each branch is concatenated with the features extracted from all the previous branches, and then convolved; finally, the feature channels extracted from the five branches are concatenated, and then residual connections are performed.

[0009] Furthermore, the features processed by the detail optimization module DOM and the pyramid feature extraction module PFEM are first concatenated by channels, and then further enhanced by spatial attention and channel attention mechanisms respectively. High-level features are upsampled to the same size as low-level features by bilinear interpolation.

[0010] Furthermore, in the multi-scale feature fusion module MFFM, the aliasing effect caused by upsampling is fused and optimized. First, the pixel values ​​are added together, and then the image is divided into four branches, three of which are downsampled to 1 / 2, 1 / 4, and 1 / 8 of the original size, respectively. The image sampled to 1 / 2 of the original size is then convolved with a 3×3 convolution, and then downsampled by 1 / 2 again, and concatenated with the feature map channel downsampled by 1 / 4. This process is repeated until the resulting 1 / 2, 1 / 4, and 1 / 8 feature maps are uniformly upsampled to the size before downsampling, and then added to the pixel values ​​before downsampling. Finally, a 3×3 convolution is used to extract the features once.

[0011] Furthermore, in the Feature Optimization Module (FOM), the input is copied into two branches. One branch directly extracts features using a 3×3 convolution, while the other branch first undergoes average pooling followed by a 3×3 convolution to extract features. Next, one of these branches is further divided into two sub-branches. One sub-branch directly extracts features using a 3×3 convolution + batch normalization + ReLU + 3×3 convolution + batch normalization, while the other sub-branch uses average pooling + 3×3 convolution + batch normalization + ReLU + 3×3 convolution + batch normalization. Then, the output pixel values ​​of the two sub-branches are summed, and a ReLU activation function is used to increase non-linearity. The resulting pixel values ​​of the two branches are then summed again, and features are extracted using a 3×3 convolution + batch normalization + ReLU. Finally, residual connections and a 3×3 convolution + batch normalization + ReLU operation are performed with the input features to obtain the output.

[0012] Furthermore, the preprocessing involves smoothing and denoising the DUTS-TR dataset, as well as random cropping and flipping, to increase the diversity of the images.

[0013] The beneficial effects of this invention are as follows: This invention proposes a fully supervised salient object detection method, which utilizes a novel salient object detection model, MFFRNet (multibranch feature fusion and refinement network), to detect salient objects. This model effectively fuses the detailed information of low-level features and the semantic information of high-level features. A module designed for low-level features employs asymmetric convolution to reduce background noise and other interfering factors, while a module designed for high-level features obtains rich semantic information. Simultaneously, it effectively addresses the aliasing effect caused by frequent upsampling. Finally, the fused model is optimized. This method effectively captures salient objects and obtains salientity prediction maps, exhibiting strong robustness. Attached Figure Description

[0014] Figure 1 Flowchart of the fully supervised salient target detection method of this invention;

[0015] Figure 2 A schematic diagram of the network model structure constructed in this invention;

[0016] Figure 3 A schematic diagram of the DOM structure in the salient target detection model of this invention;

[0017] Figure 4 A schematic diagram of the PFEM structure in the salient target detection model of this invention;

[0018] Figure 5 A schematic diagram of the MFFM structure in the salient target detection model of this invention.

[0019] Figure 6 A schematic diagram of the FOM structure in the salient target detection model of this invention. Detailed Implementation

[0020] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0021] Salient object detection (SOD) aims to identify important salient parts of natural images or videos containing significant salient information, laying the foundation for other related applications such as image classification, video compression, and object tracking. After image processing by a backbone network, five sets of feature maps are obtained. Each set of feature maps undergoes more convolutional kernels to extract features than the previous set, possessing more semantic information. Therefore, the feature maps can be divided into two categories: low-level features and high-level features. Low-level features contain more detailed information and better delineate object boundaries, but are filled with background noise during the fusion process. High-level features, on the other hand, contain more semantic information and better help locate salient objects, but the object boundaries are relatively coarse. In short, low-level and high-level features have different characteristics and should be treated differently during processing. Furthermore, existing SOD models have other major problems, such as not considering the negative impact of multiple upsampling on the saliency map. To address these issues, a novel SOD model, the Multi-Branch Feature Fusion Refinement Network (MFFRNet), is proposed, which can obtain high-precision saliency maps. Since the first set of feature maps extracted from the backbone network has a small performance improvement on the overall network but introduces a large computational burden, we chose to discard the first set of feature maps and retain the other four sets. The second and third sets of feature maps are considered low-level features, while the fourth and fifth sets are considered high-level features. We first use two modules, DOM (Detail Optimization Module) and PFEM (Pyramid Feature Extraction Module), to process the features extracted from the backbone network. The DOM module processes low-level features, which can better suppress noise; the PFEM module processes high-level features, which can obtain multi-scale features and rich contextual information, thereby improving accuracy. Subsequently, MFFM (Multi-scale Feature Fusion Module) is used to aggregate the multi-scale feature maps from DOM and PFEM to better reduce the aliasing effect caused by frequent upsampling. Finally, a FOM (Feature Optimization Module) is designed to optimize multi-scale feature fusion.

[0022] like Figure 1 The flowchart shown is for a fully supervised salient object detection method. The dataset used in this invention is the publicly available DUTS-TR dataset. First, the acquired data is preprocessed and augmented. Then, a complete model is constructed. Next, the dataset is input into the proposed model for training, and backpropagation is performed once after each round to optimize the model parameters. The model's performance is evaluated using a test set. Finally, the evaluated model is used for salient object detection.

[0023] like Figure 2 The diagram shows the network model structure. This embodiment provides a more detailed description of the salient target detection method based on the above embodiments.

[0024] We obtained the DUTS-TR dataset and smoothed and denoised it, as well as randomly cropped and flipped it to increase the diversity of the images.

[0025] The image is input into the ResNet-50 backbone network, where it is encoded to generate five sets of feature maps of different sizes. These five sets of feature maps undergo varying numbers of convolutional operations to extract features, resulting in feature maps containing information with distinct characteristics. Based on this, a Detail Optimization (DOM) module and a Pyramid Feature Extraction (PFEM) module are designed to process the second and third sets of low-level features and the fourth and fifth sets of high-level features, respectively.

[0026] like Figure 2 , 3 As shown, each set of low-level features is fed into the detail optimization module (DOM), which duplicates the features into five identical copies, forming five branches. Four of these branches extract features using combinations of 1×1 convolutions, asymmetric convolutions consisting of 1×n and n×1 (n = 1, 3, 5, 7) layers, and dilated convolutions with a dilation rate of 3. Asymmetric convolutions reduce computation without affecting feature extraction, while dilated convolutions provide a larger receptive field. The features extracted from the four branches are then concatenated and combined, followed by channel compression and further feature extraction using 3×3 convolutions. Finally, residual connections are used to superimpose the input and processed features to supplement missing information from the feature extraction process, and the ReLU activation function is used to add non-linearity and improve the neural network's expressive power.

[0027] like Figure 2 , 4 As shown, each set of high-level features is fed into the Pyramid Feature Extraction (PFEM) module, which, as described above, is divided into six branches. The first branch undergoes a 3×3 dilated convolution and a 3×3 regular convolution, extracting features which are then used as input and concatenated with the channels of the next branch, followed by the same convolution operation. This process continues, with each branch concatenating channels with the features extracted from all previous branches before convolution. Finally, the feature channels extracted from the five branches are concatenated and then residual-connected.

[0028] Features processed by DOM and PFEM are first concatenated by channels, and then further enhanced using spatial attention and channel attention mechanisms respectively. Afterwards, high-level features are upsampled to the same size as low-level features using bilinear interpolation.

[0029] like Figure 2 , 5 As shown, the above features are input into the Multi-Scale Feature Fusion (MFFM) module to fuse and optimize the aliasing effect caused by upsampling. First, the pixel values ​​are summed, then the model is divided into four branches. Three of these branches are downsampled to 1 / 2, 1 / 4, and 1 / 8 of the original size, respectively. The image sampled to 1 / 2 of its original size is convolved with a 3×3 matrix, then downsampled by 1 / 2 again, and concatenated with the feature map channel downsampled to 1 / 4. This process is repeated until the resulting 1 / 2, 1 / 4, and 1 / 8 feature maps are uniformly upsampled to their original size, and then summed with the original pixel values. Finally, a 3×3 convolution is used to extract the features once.

[0030] like Figure 2 , 6 As shown, the structure approximates symmetry and is called the Feature Optimization Module (FOM). The input is duplicated into two branches. One branch directly extracts features using a 3×3 convolution, while the other branch first undergoes average pooling followed by a 3×3 convolution to extract features. Next, one of these branches is further divided into two sub-branches. One sub-branch directly extracts features using a 3×3 convolution + batchnorm normalization + ReLU + 3×3 convolution + batchnorm normalization, while the other sub-branch uses average pooling + 3×3 convolution + batchnorm normalization + ReLU + 3×3 convolution + batchnorm. Then, the output pixel values ​​of the two sub-branches are summed, and a ReLU activation function is used to increase non-linearity. The resulting pixel values ​​from the two branches are summed again, and then subjected to a 3×3 convolution + batchnorm normalization + ReLU to extract features. Finally, a residual connection is performed with the input features, followed by a 3×3 convolution + batchnorm normalization + ReLU operation to obtain the output.

[0031] Finally, a 3×3 convolution with batch normalization and ReLU is used to extract features, and a sigmoid activation function is used to obtain the final prediction map.

[0032] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A fully supervised salient object detection method, characterized in that, The method comprises the following steps: adopting a public DUTS-TR dataset as a data set, firstly, pre-processing and data enhancement are performed on the obtained data; then, a complete multi-branch feature fusion refinement network MFFRNet is constructed as a salient object detection model; then, the training set in the data set is input into the proposed MFFRNet model for training, and the MFFRNet model parameters are optimized through reverse propagation after each round of completion; the performance of the model is evaluated by using the test set in the data set; finally, the model after evaluation is used for salient object detection; The method for establishing the multi-branch feature fusion refinement network MFFRNet comprises a detail optimization module DOM, a pyramid feature extraction module PFEM, a multi-scale feature fusion module MFFM and a feature optimization module FOM; the picture in the data set is input into a ResNet-50 backbone network to encode the picture and generate five groups of feature maps with different sizes; the five groups of feature maps are subjected to convolution operations with different numbers to extract features, the second and third groups of feature maps are regarded as low-level features, and the fourth and fifth groups are regarded as high-level features; The low-level features are sent to the detail optimization module DOM for processing to suppress noise; The high-level features are sent to the pyramid feature extraction module PFEM for processing to obtain multi-scale features and rich context information; the output of the detail optimization module DOM and the pyramid feature extraction module PFEM is input into the multi-scale feature fusion module MFFM to aggregate features; the output features of the multi-scale feature fusion module MFFM are input into the feature optimization module FOM to optimize the multi-scale feature fusion; finally, a 3x3 convolution + batchnorm normalization + ReLU is used to extract features, and a sigmoid activation function is used to obtain the final prediction map; In the detail optimization module DOM, firstly, each group of low-level features is copied into five features to form five branches; four branches are used to extract features by using a 1x1 convolution, an asymmetric convolution composed of 1xn and nx1 and a combination of a 3-dilated convolution, and n = 1, 3, 5, 7; then, the features extracted by the four branches are spliced together, compressed by a 3x3 convolution and further extracted; finally, residual connection is used to superimpose the input and the input processing features to supplement the missing information in the feature extraction process, and a ReLU activation function is used to increase the nonlinearity factor and improve the expression ability of the neural network to the model; In the feature optimization module FOM, the input is copied into two branches, one branch directly extracts features by using a 3x3 convolution, and the other branch first performs average pooling and then extracts features by using a 3x3 convolution; then, one of the branches is divided into two sub-branches, one of which directly extracts features by using a 3x3 convolution + batchnorm normalization + ReLU + 3x3 convolution + batchnorm, and the other of which is average pooling + 3x3 convolution + batchnorm normalization + ReLU + 3x3 convolution + batchnorm; Then, the output pixel values of the two sub-branches are added, and a ReLU activation function is added to increase the non-linear capability; the pixel values of the results of the two branches are added, and then 3x3 convolution + batchnorm normalization + ReLU is performed to extract features; finally, residual connection is performed with the input features, and 3x3 convolution + batchnorm normalization + ReLU is performed to obtain the output.

2. The fully supervised salient object detection method of claim 1, wherein, In the pyramid feature extraction module PFEM, first, each group of high-level features is copied into six identical features, which respectively constitute six branches; the first branch is subjected to 3x3 hollow convolution and 3x3 ordinary convolution, and the extracted features are taken as input and spliced with the next branch channel, and then the same convolution operation as before is performed, and so on; each branch is spliced with the features extracted by all previous branches in the channel, and then convolved; finally, the features extracted by the five branches are spliced in the channel, and then residual connection is performed.

3. The fully supervised salient object detection method of claim 2, wherein, The features processed by the detail optimization module DOM and the pyramid feature extraction module PFEM are first spliced in the channel, and then the spatial attention and channel attention mechanisms are used to further strengthen the features; the high-level features are upsampled to the same size as the low-level features by bilinear interpolation.

4. The fully supervised salient object detection method of claim 3, wherein, In the multi-scale feature fusion module MFFM, the aliasing effect caused by upsampling is fused and optimized; first, the pixel values are added, and then divided into four branches, three of which are downsampled to 1 / 2, 1 / 4 and 1 / 8 of the original size respectively; the picture sampled to 1 / 2 is subjected to 3x3 convolution, and then downsampled by 1 / 2, and spliced with the feature map downsampled by 1 / 4 in the channel; in this way, finally, the 1 / 2, 1 / 4 and 1 / 8 feature maps obtained are upsampled to the size before downsampling, and then added to the pixel values before downsampling; finally, 3x3 convolution is used to extract features once.

5. The method of claim 1, wherein the method further comprises: The preprocessing is to smooth and denoise the DUTS-TR dataset, and randomly crop and flip to increase the diversity of the pictures.

Images