Salient object detection algorithm based on efficient multi-scale context exploration network
By introducing an efficient multi-scale context exploration network in significance object detection, the progressive multi-scale context extraction and hierarchical feature hybrid interaction module is used to solve the problem of poor performance of significant object detection in complex contexts and achieve higher detection Accuracy and generalization ability.
Patent Information
- Application Number
- CN202210279696.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-21
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2042-03-21
AI Technical Summary
The existing significance target detection methods are difficult to effectively detect and segment significant targets in complex contexts, mainly due to the lack of full utilization of the correlation between multi-scale and multi-received field context information, resulting in poor detection performance in complex structures.
An efficient multi-scale context exploration network (EMCENet) is proposed, consisting of an progressive multi-scale context extraction (PMCE) module and a hierarchical feature hybrid interaction (HFHI) module, through group convolution, hollow convolution, and asymmetric convolution and depth separation convolution and other operations to capture multi-receptive fields and multi-scale features, and generate rich multi-level features through feature splicing and residual features. Combined with the adaptive interaction features of the hierarchical feature hybrid interaction module, high-quality feature representations are generated. .
On six public significance detection benchmark datasets, EMCENet significantly improves detection performance, exhibits good generalization ability and accuracy, surpassing 13 existing methods.
Smart Images
Figure CN114612684B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of image processing and computer vision, and in particular to a salient object detection algorithm based on an efficient multi-scale context exploration network. Background Art
[0002] The statements in this section merely relate to the background technology related to the present invention and do not necessarily constitute the prior art.
[0003] Salient object detection (SOD) aims to detect the most attractive objects or regions in an image, making it widely used as a preprocessing stage in many computer vision tasks such as image retrieval, image segmentation, video compression, etc.
[0004] Early salient object detection (SOD) methods mainly used hand-crafted features and heuristic priors to predict salient objects. Since they cannot utilize high-level semantic information, they cannot effectively locate salient objects in complex backgrounds. Recently, convolutional neural networks (CNNs) and fully convolutional neural networks (FCNs) have successfully broken the limitation that traditional salient object detection methods can only utilize hand-crafted features and underlying visual prior features. Their excellent ability to extract high-level semantic features has made deep learning-based methods widely used in the field of salient object detection.
[0005] Recently, many salient object detection methods based on fully convolutional neural networks (FCNs) have been designed, which mainly detect and segment salient objects by aggregating multi-level features or integrating various contexts. Specifically, Hu et al. introduced short connections in the HED structure to propagate information from multi-level features to predict salient maps, and Zhang et al. directly integrated multi-level features into multi-resolutions in the Amulet network, and detected salient objects in a recursive manner by learning these aggregated features of different resolutions. The initial multi-level features contain not only important information but also a lot of noise information, so the performance of the salient maps generated by directly aggregating cannot achieve satisfactory results. Later, Zhang et al. proposed a multi-scale context-aware feature extraction module in the BDMPM network to obtain rich context information for multi-level features to locate salient objects. In the CPD network, Wu et al. designed a context module inspired by the RFB module to capture global contrast information to predict salient maps. Similarly, Li et al. proposed a parallel dilated convolution (PDC) module in the SUCA network to extract context-aware multi-scale multi-receptive field features to improve the ability of the initial multi-level features to locate salient objects. In addition, Pang et al. designed a self-interaction module (SIM) in the MINet network to obtain more multi-scale features from the integrated features, thereby segmenting salient objects with variable scales. Although the above methods consider the importance of contextual information, they are committed to obtaining salient object information by directly aggregating multiple independent context-aware information, while rarely considering the correlation of multi-scale and multi-receptive field context information, which is not conducive to the effective detection and segmentation of salient objects with complex structures under complex backgrounds. Summary of the invention
[0006] To alleviate the above problems, in this paper, we propose a new efficient multi-scale context exploration network for salient object detection, named EMCENet. Specifically, the EMCENet method consists of two modules, namely the Progressive Multi-scale Context Extraction (PMCE) module and the Hierarchical Feature Hybrid Interaction (HFHI) module. The former aims to efficiently capture and utilize multi-scale context-aware information in a progressive manner to improve the performance of the initial multi-level features, and the latter aims to adaptively and efficiently fuse multi-level features to generate high-quality feature representations for accurate prediction and segmentation of salient objects. Extensive experimental results show that the proposed EMCENet method has great advantages over 13 existing SOD methods in different evaluation metrics on six public saliency detection benchmark datasets.
[0007] The technical solution of the present invention is to provide a salient target detection algorithm integrating a multi-source feature network, the method comprising the following steps:
[0008] 1. Input RGB image and use the pre-trained ResNet50 network as the backbone network to extract initial multi-level features;
[0009] 1.1) Collect and organize relevant datasets in the field of saliency detection, including SOD dataset, ECSSD dataset, PASCAL-S dataset, DUT-OMRON dataset, HKU-IS dataset and DUST dataset.
[0010] 1.2) Considering that model training requires large-scale data, this patent uses the DUTS dataset containing 10,553 images as the training dataset, and uses the ECSSD, PASCAL-S, HKU-IS, DUT-OMRON and DUTS-TE datasets as the test datasets to detect the generalization performance of the model.
[0011] 1.3) Use the pre-trained ResNet50 network as the backbone network to extract the initial multi-level features I = {I i |i=1,2,3,4,5}.
[0012] 2. The progressive multi-scale context exploration module that integrates group convolution operations, dilated convolution operations, and asymmetric convolution operations is used to efficiently capture multi-receptive field and multi-scale features, thereby increasing the salient target information in the initial multi-level features;
[0013] 2.1) In order to save computing resources, we first use a 1×1 group convolution that can process the initial multi-level features in a multi-branch manner to improve the accuracy of the model, and the group is set to 4. For the initial multi-level features, we reconstruct them into 128 channels, and then we use asymmetric convolution with a convolution kernel size of (1×(2n+1), (2n+1)×1) to extract contextual information, where n represents the i-th stage (i=1, 2, 3, 4).
[0014] 2.2) Subsequently, we use depthwise separable convolution to extract multi-scale contextual information again, which can powerfully extract multi-scale contextual information with fewer parameters, and the padding rate is set to 2n. In addition, we also introduce two types of residual connections, namely direct residual connections and indirect residual connections (using convolution kernels of size 1×1 convolution operations to process input features again) to increase the diversity of input features.
[0015] 2.3) Use the context extracted from the small receptive field branch as a guide to input the large receptive field branch, and gradually enhance the correlation between multi-scale and multi-receptive field contexts to capture context information. n The relevant formula is as follows:
[0016]
[0017] Here σ represents a dimensionality reduction operation, C1 represents a convolution operation with a kernel size of 1×1, A represents an asymmetric convolution operation with a kernel size of (1×(2n+1), (2n+1)×1), ∑ represents an element-wise addition operation, and D n Represents a depthwise separable convolution with padding rate 2n.
[0018] 2.4) Perform feature concatenation on these context information Mn and introduce residual features to generate multi-level features Q={Q i |i=1,2,3,4,5}. The feature Q is defined as follows:
[0019] Q=Σ(σ(I),σ(Cat(M1,...,M n ))), n = 4 (2)
[0020] where Cat represents the feature concatenation operation, and σ and ∑ are similar to Equation 1. Note that each initial multi-level feature I generates a corresponding feature Q through a progressive multi-scale contextual exploration module.
[0021] 3. Based on the hierarchical feature hybrid interaction module, multi-level features are adaptively interacted to generate high-quality feature representation;
[0022] 3.1) The input strategy we use is similar to that in the DSS network, but the difference is that we input the multi-level features Q generated in the progressive multi-scale context exploration module as guidance at the same level.
[0023] 3.2) Each multi-level feature Q can interact with information four times in the hierarchical feature mixing interaction module. We first perform element-wise multiplication and element-wise addition operations on all input features.
[0024] 3.3) Then, in order to improve the utilization of the multi-level features Q, we again perform element-level addition and element-level multiplication under the guidance of feature Q in the second stage. In the third stage, the two sets of features are sequentially concatenated. Finally, a dimensionality reduction operation is performed to generate a high-quality feature representation P = {P i |i=1,2,3,4,5}, the formula is defined as:
[0025]
[0026] Where σ represents the dimensionality reduction operation, Cat represents the feature concatenation operation, and ∑ and * represent element-wise addition and multiplication operations.
[0027] 4. Perform a series of dimension reduction activation operations on the high-quality feature representation to generate an initial saliency map and use the true value labels for supervised training.
[0028] 4.1) In the proposed method, we use the binary cross entropy (BCE) loss function and the IoU loss function as the loss function and use the deep supervision strategy to train the model. Mathematically, the loss function is calculated as follows:
[0029] δ=δ bce +δ iou (4)
[0030] where δ bce Represents the binary cross entropy (BCE) loss function, which can calculate the loss of each predicted pixel independently and is a widely used loss in segmentation and binary classification. iou stands for the IoU loss function, which can help improve the training process and increase the confidence of salient object predictions.
[0031] Advantages of the present invention: The present invention makes full use of the multi-level features extracted from the pre-trained ResNet50 network, increases the saliency information in the initial multi-level features through a progressive multi-scale context exploration module, and then uses a hierarchical feature hybrid interaction module to adaptively interact with different information in the multi-level features, thereby generating high-quality feature representations for accurate detection of salient targets. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 Network Flowchart
[0033] Figure 2 Network training mode diagram
[0034] Figure 3 Visual comparison chart DETAILED DESCRIPTION
[0035] The following will be combined with the drawings in the examples of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. In addition, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this research direction without creative work are within the scope of protection of the present invention.
[0036] The flowchart framework of the present invention is as follows Figure 1 As shown, the present invention is based on a salient target detection method of an efficient multi-scale context exploration network, and its specific operation is described as follows:
[0037] 1. Input RGB image and use the pre-trained ResNet50 network as the backbone network to extract initial multi-level features;
[0038] 1.1) Collect and organize relevant datasets in the field of saliency detection, including SOD dataset, ECSSD dataset, PASCAL-S dataset, DUT-OMRON dataset, HKU-IS dataset and DUST dataset.
[0039] 1.2) Considering that model training requires large-scale data, this patent uses the DUTS dataset containing 10,553 images as the training dataset, and uses the ECSSD, PASCAL-S, HKU-IS, DUT-OMRON and DUTS-TE datasets as the test datasets to detect the generalization performance of the model.
[0040] 1.3) Use the pre-trained ResNet50 network as the backbone network to extract the initial multi-level features I = {I i |i=1,2,3,4,5}.
[0041] 2. The progressive multi-scale context exploration module that integrates group convolution operations, dilated convolution operations, and asymmetric convolution operations is used to efficiently capture multi-receptive field and multi-scale features, thereby increasing the salient target information in the initial multi-level features, such as Figure 1 As shown;
[0042] 2.1) In order to save computing resources, we first use a 1×1 group convolution that can process the initial multi-level features in a multi-branch manner to improve the accuracy of the model, and the group is set to 4. For the initial multi-level features, we reconstruct them into 128 channels, and then we use asymmetric convolution with a convolution kernel size of (1×(2n+1), (2n+1)×1) to extract contextual information, where n represents the i-th stage (i=1, 2, 3, 4).
[0043] 2.2) Subsequently, we use depthwise separable convolution to extract multi-scale contextual information again, which can powerfully extract multi-scale contextual information with fewer parameters, and the padding rate is set to 2n. In addition, we also introduce two types of residual connections, namely direct residual connections and indirect residual connections (using convolution kernels of size 1×1 convolution operations to process input features again) to increase the diversity of input features.
[0044] 2.3) Use the context extracted from the small receptive field branch as a guide to input the large receptive field branch, and gradually enhance the correlation between multi-scale and multi-receptive field contexts to capture context information. n The relevant formula is as follows:
[0045]
[0046] Here σ represents a dimensionality reduction operation, C1 represents a convolution operation with a kernel size of 1×1, A represents an asymmetric convolution operation with a kernel size of (1×(2n+1), (2n+1)×1), ∑ represents an element-wise addition operation, and D n Represents a depthwise separable convolution with padding rate 2n.
[0047] 2.4) Perform feature concatenation on these context information Mn and introduce residual features to generate multi-level features Q={Q i |i=1,2,3,4,5}. The feature Q is defined as follows:
[0048] Q=Σ(σ(I),σ(Cat(M1,...,M n ))), n = 4 (2)
[0049] where Cat represents the feature concatenation operation, and σ and ∑ are similar to Equation 1. Note that each initial multi-level feature I generates a corresponding feature Q through a progressive multi-scale contextual exploration module.
[0050] 3. Based on the hierarchical feature hybrid interaction module, multi-level features are adaptively interacted to generate high-quality feature representations, such as Figure 1 As shown;
[0051] 3.1) The input strategy we use is similar to that in the DSS network, but the difference is that we input the multi-level features Q generated in the progressive multi-scale context exploration module as guidance at the same level.
[0052] 3.2) Each multi-level feature Q can interact with information four times in the hierarchical feature mixing interaction module. We first perform element-wise multiplication and element-wise addition operations on all input features.
[0053] 3.3) Then, in order to improve the utilization of the multi-level features Q, we again perform element-level addition and element-level multiplication under the guidance of feature Q in the second stage. In the third stage, the two sets of features are sequentially concatenated. Finally, a dimensionality reduction operation is performed to generate a high-quality feature representation P = {P i |i=1,2,3,4,5}, the formula is defined as:
[0054]
[0055] Where σ represents the dimensionality reduction operation, Cat represents the feature concatenation operation, and ∑ and * represent element-wise addition and multiplication operations.
[0056] 4. Perform a series of dimensionality reduction activation operations on the high-quality feature representation to generate the initial saliency map, and use the true value label for supervised training, such as Figure 2 shown.
[0057] 4.1) In the proposed method, we use the binary cross entropy (BCE) loss function and the IoU loss function as the loss function and use the deep supervision strategy to train the model. Mathematically, the loss function is calculated as follows:
[0058] δ=δ bce +δ iou (4)
[0059] where δ bce Represents the binary cross entropy (BCE) loss function, which can calculate the loss of each predicted pixel independently and is a widely used loss in segmentation and binary classification. iou stands for the IoU loss function, which can help improve the training process and increase the confidence of salient object predictions.
[0060] 5. In order to fully demonstrate the efficiency and generalization ability of our model, we qualitatively and quantitatively compare the proposed salient object detection algorithm based on efficient multi-scale context exploration network with some existing algorithms, such as Figure 3 shown.
[0061] The above is for the preferred implementation of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A salient object detection algorithm based on an efficient multi-scale context exploration network, characterized by: The method comprises the following steps: 1) Input RGB image and use the pre-trained ResNet50 network as the backbone network to extract initial multi-level features; 2) The progressive multi-scale context exploration module that integrates group convolution operations, dilated convolution operations, and asymmetric convolution operations is used to efficiently capture multi-receptive field and multi-scale features, thereby increasing the salient target information in the initial multi-level features; 3) Based on the hierarchical feature hybrid interaction module, multi-level features are adaptively interacted to generate high-quality feature representation; 4) Perform a series of dimensionality reduction activation operations on the high-quality feature representation to generate an initial saliency map, and use the true value labels for supervised training; The specific method of step 2 is: 3.1) In order to save computing resources, we first use a 1×1 group convolution that can process the initial multi-level features in a multi-branch manner to improve the accuracy of the model, and the group is set to 4. For the initial multi-level features, we reconstruct them into 128 channels, and then we use asymmetric convolution with a convolution kernel size of (1×(2n+1), (2n+1)×1) to extract context information, where n represents the i-th stage (i=1, 2, 3, 4); 3.2) Subsequently, we use depthwise separable convolution to extract multi-scale contextual information again, which can powerfully extract multi-scale contextual information with fewer parameters, and the filling rate is set to 2n; in addition, we also introduce two types of residual connections, namely direct residual connection and indirect residual connection, and use a convolution kernel of size 1×1 convolution operation to process the input features again to increase the diversity of input features; 3.3) Use the context extracted from the small receptive field branch as the guide input to the large receptive field branch to gradually enhance the correlation between multi-scale and multi-receptive field contexts, thereby capturing context information. n , the relevant formula is as follows: Here σ represents a dimensionality reduction operation, C1 represents a convolution operation with a kernel size of 1×1, A represents an asymmetric convolution operation with a kernel size of (1×(2n+1), (2n+1)×1), ∑ represents an element-wise addition operation, and D n represents a depth-wise separable convolution with a filling rate of 2n; 3.4) Perform feature concatenation on these context information Mn and introduce residual features to generate multi-level features Q={Q i |i=1,2,3,4,5}, the feature Q is defined as follows: Q=∑(σ(I),σ(Cat(M1,...,M n ))),n=4 (2) where Cat represents the feature concatenation operation, σ and ∑ are similar to Equation 1. Note that each initial multi-level feature I generates the corresponding feature Q through a progressive multi-scale context exploration module.
2. The salient object detection algorithm based on efficient multi-scale context exploration network according to claim 1 is characterized in that: The specific method of step 1) is: 2.1) Collect and organize relevant datasets in the field of saliency detection, including SOD dataset, ECSSD dataset, PASCAL-S dataset, DUT-OMRON dataset, HKU-IS dataset and DUST dataset; 2.2) Considering that model training requires large-scale data, the DUTS dataset containing 10,553 images is used as the training dataset, and the ECSSD, PASCAL-S, HKU-IS, DUT-OMRON and DUTS-TE datasets are used as test datasets to detect the generalization performance of the model; 2.3) The pre-trained ResNet50 network is used as the backbone network to extract the initial multi-level features I = {I i |i=1,2,3,4,5}.
3. The salient object detection algorithm based on efficient multi-scale context exploration network according to claim 1 is characterized in that: The specific method of step 3) is: 4.1) The input strategy used is similar to that in the DSS network, but the difference is that the multi-level features Q generated in the progressive multi-scale context exploration module are input together as guidance at the same level; 4.2) Each multi-level feature Q can interact with each other four times in the hierarchical feature mixing interaction module. We first perform element-wise multiplication and element-wise addition operations on all input features; 4.3) Then, in order to improve the utilization of the multi-level features Q, we again perform element-level addition and element-level multiplication under the guidance of feature Q in the second stage. In the third stage, the two sets of features are sequentially concatenated. Finally, a dimensionality reduction operation is performed to generate a high-quality feature representation P = {P i |i=1,2,3,4,5}, the formula is defined as: Where σ represents the dimensionality reduction operation, Cat represents the feature concatenation operation, and ∑ and * represent element-wise addition and multiplication operations.
4. The salient object detection algorithm based on efficient multi-scale context exploration network according to claim 1 is characterized in that: The specific method of step 4) is: 5.1) In the proposed method, we use the binary cross entropy (BCE) loss function and the IoU loss function as the loss function and use the deep supervision strategy to train the model; mathematically, the loss function is calculated as follows: d=d bce +d iou (4) where δ bce represents the binary cross entropy (BCE) loss function, which can independently calculate the loss of each predicted pixel and is a widely used loss in segmentation and binary classification. iou stands for the IoU loss function, which can help improve the training process and increase the confidence of salient object predictions.