Double-path multi-branch feature residual method for salient target detection

Abstract

The invention discloses a double-path multi-branch feature residual method for salient target detection, which is characterized in that an encoder comprises a CNN branch and a lightweight ViT branch, and a context enrichment module is used for performing jump connection of the CNN branch; the CEM firstly reduces channels through 3 * 3 convolution, then obtains context features through k * k convolution, obtains multi-scale global feature information through MFIM after feature extraction is carried out through an encoder, and gradually refines target features through a residual structure. The performance of the model is improved through the local features of the CNN branches and the long-term dependency relationship from the lightweight ViT branches; the MFIM interacts with features of different scales to promote information transmission and fusion among the features of different scales, the MFRM better integrates different features by using branches of different scales, and in addition, the module adopts a residual structure to allow a network to learn residual information of the features layer by layer, so that key details in an image are captured more effectively.

Description

Technical Field

[0001] The present invention relates to the technical field of natural image processing, and more specifically, to a dual-path multi-branch feature residual method for salient object detection. Background Art

[0002] Salient object detection (SOD) refers to identifying the most important objects in an image or video. The human visual system has excellent recognition capabilities in complex scenes and can easily find regions of interest, even if these regions are similar to the background. Therefore, obtaining salient objects from images plays an important role in many computer vision applications, including object tracking, image segmentation, video segmentation, and scene classification. As a key component of computer vision, SOD is of considerable significance for performing complex visual tasks. It can reduce the difficulty of subsequent more complex tasks by extracting important objects.

[0003] In recent years, the rapid development of deep learning technology in the field of computer vision has greatly promoted the development of SOD. Convolutional neural networks (CNNs) are the cornerstone of this development. They are good at extracting high-level and low-level semantic information, which is an ability that surpasses traditional methods. At present, most SOD methods are based on the framework of fully convolutional networks (FCNs). These methods can skillfully extract robust semantic and global contextual information and use feature fusion to produce conclusive predictions. However, CNN-based methods face the problem of capturing long-range dependencies between distant image pixels. Utilizing these long-range dependencies has been shown to help improve the overall performance of the model. Visual transformers (ViT) have recently shown excellent performance in various image processing tasks. Vit-based models have greatly improved the accuracy of image processing assignments, including significant object segmentation. Although Vit-based methods are able to skillfully capture long-term relationships, they encounter limitations in extracting local features, which hinders the model's ability to identify detailed object features. In the context of SOD tasks, both long-range dependencies and local feature relationships are of equal importance. Summary of the invention

[0004] The purpose of the present invention is to provide a dual-path multi-branch feature residual method for salient target detection, to solve the problem that traditional CNN cannot obtain long-range dependencies well, and to enhance boundary information so that it helps to restore the fine details of the image.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows: a dual-path multi-branch feature residual method for salient target detection, the method is based on a dual-path encoder framework, comprising:

[0006] The encoder consists of a CNN branch and a lightweight ViT branch, which are used to capture local features and long-term dependencies between image patches, respectively;

[0007] A context enrichment module is used for skip connections between CNN branches;

[0008] CEM initially reduces the channels through 3×3 convolution and then obtains context features through k×k convolution, where k is set to 3, 3, 5, 5, 7;

[0009] After feature extraction through the encoder, MFIM is used to obtain multi-scale global feature information;

[0010] The multi-scale global feature information is input into a series of MRFMs in the decoder, which adopts a multi-scale branching strategy to effectively aggregate features and gradually refine the target features using the residual structure.

[0011] Preferably, the CNN branch utilizes Res2Net, which includes four residual blocks and a backbone. Res2Net is used to extract feature information at different scales, and the result output of the CNN branch is recorded as FC.

[0012] Preferably, the ViT branch consists of four lightweight transformers connected in parallel, each transformer including a patch embedding layer and a transformer encoding layer, which contains four attention heads. The branch takes four feature patches of different sizes as input. The ViT branch divides the input features into HW / 162 (R=4, 8, 16, 32) blocks, where H and W represent height and width, respectively. These patches are then input into four parallel transformer blocks. Then, each patch vector is linearly projected to obtain a patch embedding. After that, the patch embedding and the position embedding are input together into a lightweight multi-head self-attention layer and a feed-forward network to generate the output. The multi-head self-attention operation processes the three weight matrices of input query Q, key K and value V to calculate the attention score, as shown in the following formula

[0013]

[0014] Where dk is the dimension of the key, KT is the transposed matrix of matrix K, and then the output Ft is obtained through a feed-forward network with two linear layers with ReLU activation function. Subsequently, the ViT branch connects the outputs of four parallel transformer blocks and uses a convolutional layer to generate the final output;

[0015] Finally, the encoder uses a dual attention module to merge the output Fc of the CNN branch and the output Ft of the ViT branch.

[0016] Preferably, the dual-path encoder comprises a plurality of MFIMs of parallel multi-scale global feature branches, and is interactively connected using skip connections, wherein the MFIM extracts multi-scale contextual information from the output features of the backbone network using branches of different scales, and the MFIM is composed of 5 branches of different scales for screening features at different scales, and the three middle branches adopt 3×3 dilated convolutions, and the dilation rates are 2, 3 and 5 from left to right, respectively, and the purpose of using dilated convolutions is to increase the receptive field without increasing the computational overhead, and the remaining two branches include a 1×1 convolution layer and a maximum pooling layer, and the original features are retained by setting the minimum and maximum receptive fields, and then, a skip connection method is used to promote the interaction between branches of different scales;

[0017] The skip connection method is to introduce the feature information extracted by the small-scale branches in the three extended convolution branches into the large-scale branches, realizing cross-scale information aggregation. Finally, the features (I1, I2, ..., I5) extracted by the five branches are aggregated and added to the original feature IO, and the global feature IF is generated through a 1×1 convolution layer, which is expressed as:

[0018] I F =conv(concat(I 1 ,I 2 ,I 3 ,I 4 ,I 5 ))+conv(I o ).

[0019] Preferably, the CEM consists of four convolutional layers, the initial layer is a 3×3 convolutional layer, followed by two convolutional layers of size k×k, with parameter k set to 3, 3, 5, 5, and finally, the output is enhanced by a 3×3 convolutional layer with 64 channels.

[0020] Preferably, the MFRM uses a trunk and three sub-branches to downsample the simply fused features, perform convolution operations, and then upsample. The three sub-branches use sampling rates of 2, 4, and 8 to capture local features of different scales, respectively. The trunk is regarded as residual information and no operation is performed on it. Channel attention units are applied to the three branches to direct attention to weights at different scales. In the attention unit, global average pooling is first used on the input feature Fo to encode channel-level information:

[0021]

[0022] Among them, H and W represent the image height and width respectively, V represents the statistical vector, i and j represent the horizontal and vertical position information of the feature respectively, then the attention unit uses two fully connected layers with sigmoid and ReLU activation functions respectively to obtain the dependency between channels and obtain the weight vector VW:

[0023] V w =FC sigmoid (FC ReLU (V)

[0024] Among them, FC represents a fully connected layer, sigmoid(.) represents the use of the sigmoid function for the formula in the brackets, and ReLU represents the ReLU function. Subsequently, the channel-level multiplication between the input feature Fo and the weight scalar VW is used as the output Fa of the sub-branch. Finally, the output features of the three sub-branches are fused with the trunk features, and a 3×3 convolution is used to reduce the aliasing effect caused by the upsampling operation.

[0025] Preferably, the decoder uses a residual structure, and the MFRM output It is added element by element to generate the current MFRM result

[0026]

[0027] In the formula, up represents the upsampling operation. In addition, in order to enhance the edge features of the object, the last MFRM is edge supervised. The edge map is obtained from the input and output of MF and the output of MFRM using two consecutive convolutional structures and residual structures.

[0028] Preferably, the image is defined using a binary cross entropy loss as follows:

[0029]

[0030] Where P and Y represent the prediction result and the corresponding label respectively, j represents the index of the pixel, and M represents the total number of pixels in P. The network also uses IoU loss to calculate the image-level loss, which is calculated as follows:

[0031]

[0032] Compared with the prior art, the advantages of the present invention are:

[0033] Improve the performance of the model by integrating local features from the CNN branch and long-term dependencies from the lightweight ViT branch;

[0034] In order to locate salient objects more effectively, the present invention introduces an MFIM, which promotes information transfer and fusion between features of different scales by interacting with them, which effectively captures various representations of salient objects in the image regardless of their size or position;

[0035] The present invention designs an MFRM which utilizes branches of different sizes to better integrate different features. In addition, the module adopts a residual structure, which allows the network to learn the residual information of features layer by layer, thereby more effectively capturing the key details in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a data graph of quantitative comparison between the method of the present invention and other methods on 6 public data sets;

[0038] Figure 2 The present invention is an overall structural diagram of the model of the present invention;

[0039] Figure 3 is a structural diagram of the MFIM of the present invention;

[0040] Figure 4 is a structural diagram of the MFRM of the present invention;

[0041] Figure 5 It is a flow chart of the present invention;

[0042] Figure 6 It is a visual comparison diagram of the scenes in the data set used in the present invention. DETAILED DESCRIPTION

[0043] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0044] See also Figure 2-Figure 5 As shown, the present invention provides a dual-path multi-branch feature residual method for salient target detection, which is based on a dual-path encoder framework and includes:

[0045] The S1 encoder consists of a CNN branch and a lightweight ViT branch, which are used to capture local features and long-term dependencies between image blocks, respectively. This combination can effectively reduce the computational overhead;

[0046] S2, a context enrichment module is used for skip connections of CNN branches to enhance side output characteristics;

[0047] S3, CEM initially reduces the channels through 3×3 convolution, and then obtains context features through k×k convolution, where k is set to 3, 3, 5, 5, 7;

[0048] S4, after feature extraction by the encoder, MFIM is used to obtain multi-scale global feature information;

[0049] S5. The multi-scale global feature information is input into a series of MRFMs in the decoder. MRFM adopts a multi-scale branching strategy to effectively aggregate features and uses the residual structure to gradually refine the target features.

[0050] The model is trained using DUTS-TE data, and tested using six datasets: PASCAL-S, SOD, ECSSD, HKU-IS, DUT-OMRON and DUTS-TE.

[0051] Four evaluation metrics are used in our experiments, namely, mean absolute error (MAE), F-measure (F β ), structural similarity (S α ) and the enhanced alignment metric (F ξ ), which is described in detail as follows:

[0052] Mean Absolute Error (MAE): MAE is the average difference between the measured ground truth G and the saliency map S. The more similar the saliency map S is to the ground truth G, the smaller the MAE value is, and it is calculated as follows:

[0053]

[0054] F-measure: F-measure is widely used as a metric, which represents the weighted harmonic mean of recall and precision under non-negative weights and is defined as:

[0055]

[0056] Structural Similarity(): Considering object-aware and region-aware structural similarity, the structural similarity between the saliency map and the binary ground truth is evaluated, and the structural similarity between object-aware and region-aware is expressed separately. The expression is as follows:

[0057] Sα =α×S o +(1-α)×S r

[0058] Among them, α is set to 0.5;

[0059] Enhanced alignment metric (E ξ ): E ξ Taking into account both the local pixel matching and the global mean matching of the input image, the expression is as follows:

[0060]

[0061] Where φ is the enhanced alignment matrix.

[0062] In this embodiment, the dual-path encoder includes a CNN branch and a ViT branch. Specifically, the CNN branch utilizes Res2Net, which includes four residual blocks and a backbone. Res2Net facilitates the extraction of feature information at different scales and enhances the ability to capture local feature correlation in the image. The result output of the CNN branch is recorded as FC.

[0063] In this embodiment, the ViT branch is composed of four lightweight transformers connected in parallel, each transformer includes a patch embedding layer and a transformer encoding layer, which contains four attention heads. In addition, the branch has a single-layer structure, so the computational cost is low. The branch uses four feature patches of different scales as input, which can effectively capture the long-term dependencies between features of different scales. Through multi-scale feature fusion, the feature blocks of four different scales can be combined to form a more comprehensive and representative feature representation. This feature representation can provide global context information at different scales, thereby improving the model's ability to understand image content. The ViT branch divides the input features into HW / 162 (R=4, 8, 16, 32) blocks, where H and W represent height and width, respectively, and then inputs these patches into four parallel transformer blocks. Then, each patch vector is linearly projected to obtain a patch embedding. After that, the patch embedding and the position embedding are input together into a lightweight multi-head self-attention layer and a feed-forward network to generate the output. The multi-head self-attention operation processes the three weight matrices of input query Q, key K and value V to calculate the attention score, as shown in the following formula

[0064]

[0065] Where dk is the dimension of the key, KT is the transposed matrix of matrix K, and then the output Ft is obtained through a feed-forward network with two linear layers with ReLU activation function. Subsequently, the ViT branch connects the outputs of four parallel transformer blocks and uses a convolutional layer to generate the final output;

[0066] Finally, the encoder uses a dual attention module to merge the output Fc of the CNN branch and the output Ft of the ViT branch. This configuration allows the model to focus on both spatial and channel information. Therefore, the model cleverly integrates random dependencies (from the ViT branch) and local features (from the CNN branch), thereby enhancing representation learning. This enables it to more comprehensively understand and utilize different levels of information when processing data such as images.

[0067] In this embodiment, the dual-path encoder includes a plurality of MFIMs of parallel multi-scale global feature branches, and uses skip connections for interactive connection. The MFIM uses branches of different scales to extract multi-scale context information from the output features of the backbone network. The MFIM consists of 5 branches of different scales for filtering features at different scales. The three middle branches use 3×3 dilated convolutions, and the dilation rates are 2, 3, and 5 from left to right, respectively. The purpose of using dilated convolutions is to increase the receptive field without increasing the computational overhead. The remaining two branches include a 1×1 convolution layer and a maximum pooling layer, and the original features are retained by setting the minimum and maximum receptive fields. Subsequently, a skip connection method is used to promote the interaction between branches of different scales.

[0068] The skip connection method is to introduce the feature information extracted by the small-scale branches in the three extended convolution branches into the large-scale branches, realizing cross-scale information aggregation. The above operation is unnecessary for the 1×1 convolution layer and the maximum pooling layer because they have the inherent characteristics of preserving global context information. Finally, the features (I1, I2, ..., I5) extracted by the five branches are aggregated and added to the original feature IO, and the global feature IF is generated through the 1×1 convolution layer, which is expressed as:

[0069] I F =conv(concat(I 1 ,I 2 ,I 3 ,I 4 ,I 5 ))+conv(I o )

[0070] The above process enables the model to obtain enhanced global context information.

[0071] In this embodiment, the CEM consists of four convolutional layers. The initial layer is a 3×3 convolutional layer, followed by two convolutional layers of size k×k, with the parameter k set to 3, 3, 5, and 5. Finally, the output is enhanced by a 3×3 convolutional layer with 64 channels.

[0072] In this embodiment, an MFRM using a multi-branch fusion (MF) scheme is used, and a residual structure is used to more comprehensively refine features. The MFRM uses a trunk and three sub-branches to downsample the simply fused features, perform a convolution operation, and then upsample. The three sub-branches use sampling rates of 2, 4, and 8 to capture local features of different scales, respectively. The trunk is regarded as residual information and is not operated on. In order to further identify more effective features, channel attention units are applied to the three branches to direct attention to weights at different scales. In the attention unit, global average pooling is first used on the input feature Fo to encode channel-level information:

[0073]

[0074] Among them, H and W represent the image height and width respectively, V represents the statistical vector, i and j represent the horizontal and vertical position information of the feature respectively, then the attention unit uses two fully connected layers with sigmoid and ReLU activation functions respectively to obtain the dependency between channels and obtain the weight vector VW:

[0075] V w =FC sigmoid (FC ReLU (V)

[0076] Among them, FC represents a fully connected layer, sigmoid(.) represents the use of the sigmoid function for the formula in the brackets, and ReLU represents the ReLU function. Subsequently, the channel-level multiplication between the input feature Fo and the weight scalar VW is used as the output Fa of the sub-branch. Finally, the output features of the three sub-branches are fused with the trunk features, and a 3×3 convolution is used to reduce the aliasing effect caused by the upsampling operation.

[0077] To further refine the features, the decoder uses a residual structure and the MFRM output It is added element by element to generate the current MFRM result

[0078]

[0079] In the formula, up represents the upsampling operation. In addition, in order to enhance the edge features of the object, the last MFRM is edge supervised. The edge map is obtained from the input and output of MF and the output of MFRM using two consecutive convolutional structures and residual structures.

[0080] Preferably, the image is defined using a binary cross entropy loss as follows:

[0081]

[0082] Where P and Y represent the prediction result and the corresponding label respectively, j represents the index of the pixel, and M represents the total number of pixels in P. The network also uses IoU loss to calculate the image-level loss, which is calculated as follows:

[0083]

[0084] Result analysis:

[0085] In this section, to verify the advantages of the model performance, we compare the proposed model with 16 methods, including CPD, BASNet, SIBA, GCPANet, F3Net (Wei et al., 2020), DFI, RAFL, RAFL, DNA, DCENet, ACT-Net, TCF-Net, and GCGRAN. In order to make a fair comparison, all experimental environments are consistent and the same evaluation code is used, and no post-processing tools are used during the experiments.

[0086] Figure 1 The quantitative comparison of the proposed method with other methods on 6 public datasets is shown. It can be seen from the table that the proposed method outperforms other methods on most datasets, and the remaining other results can also be ranked second, with only a slight gap from the best indicators. For example, the method obtained the best MAE index on all 6 datasets. For the remaining three indicators, the method proposed in this paper works best in most cases. In addition, the method obtains a higher total score on F-measure. Since F-measure is more sensitive to target details, it confirms that the proposed method produces more detailed results. As shown in Table 5, the method proposed in this paper has certain advantages on 5 public datasets. In most cases, the method is effective for SOD tasks. Compared with CNN-based methods or CNN-based methods, our proposed model has obvious advantages. In terms of CNN-ViT method, the model also outperforms other methods in most indicators and shows good competitiveness. In addition, the model contains about 59.73 million parameters and has a traffic of 37.15G. Figure 6 A comparison of parameters, FLOPs and FPS numbers of this method with the three best performing methods is given. Experimental results show that this method can better balance performance and model complexity while ensuring FPS. Compared with CNN or ViT-based methods, this model has achieved a competitive advantage in performance methods. Compared with CNN-ViT-based methods, this model has good efficiency competitiveness while maintaining good performance.

[0087] Although the embodiments of the present invention are described in conjunction with the accompanying drawings, the patent owner may make various variations or modifications within the scope of the appended claims. As long as they do not exceed the protection scope described in the claims of the present invention, they should be within the protection scope of the present invention.

Claims

1. A dual-path multi-branch feature residual method for salient object detection, based on a dual-path encoder framework, characterized in that: include: The encoder consists of a CNN branch and a lightweight ViT branch, which are used to capture local features and long-term dependencies between image patches, respectively; A context enrichment module is used for skip connections between CNN branches; CEM initially reduces the channels through 3×3 convolution and then obtains context features through k×k convolution, where k is set to 3, 3, 5, 5, 7; After feature extraction through the encoder, MFIM is used to obtain multi-scale global feature information; The multi-scale global feature information is input into a series of MRFMs in the decoder, which adopts a multi-scale branching strategy to effectively aggregate features and gradually refine the target features using the residual structure.

2. The dual-path multi-branch feature residual method for salient target detection according to claim 1, characterized in that: The CNN branch uses Res2Net, which includes four residual blocks and a backbone. Res2Net is used to extract feature information at different scales. The result output of the CNN branch is recorded as FC.

3. The dual-path multi-branch feature residual method for salient target detection according to claim 2, characterized in that: The ViT branch consists of four lightweight transformers connected in parallel, each transformer includes a patch embedding layer and a transformer encoding layer, which contains four attention heads. The branch takes four feature patches of different sizes as input. The ViT branch divides the input features into HW / 162 blocks, R=4, 8, 16, 32, H and W represent height and width respectively, and then inputs these patches into four parallel transformer blocks. Linear projection is performed on each patch vector to obtain patch embedding. After that, the patch embedding and position embedding are input together into a lightweight multi-head self-attention layer and a feed-forward network to generate the output. The multi-head self-attention operation processes the three weight matrices of input query Q, key K and value V to calculate the attention score, as shown below: Where dk is the dimension of the key, KT is the transposed matrix of matrix K, and the output Ft is obtained through a feed-forward network with two linear layers with ReLU activation function. The ViT branch connects the outputs of four parallel transformer blocks and uses a convolutional layer to generate the final output; The encoder uses a dual attention module to merge the output Fc of the CNN branch and the output Ft of the ViT branch.

4. The dual-path multi-branch feature residual method for salient target detection according to claim 3, characterized in that: The dual-path encoder contains multiple parallel multi-scale global feature branches of MFIM, and uses skip connections for interactive connection. The MFIM uses branches of different scales to extract multi-scale contextual information from the output features of the backbone network. The MFIM consists of 5 different scale branches for filtering features at different scales. The three middle branches use 3×3 dilated convolutions, and the dilation rates are 2, 3 and 5 from left to right. The purpose of using dilated convolutions is to increase the receptive field without increasing the computational overhead. The remaining two branches include a 1×1 convolution layer and a maximum pooling layer. The original features are retained by setting the minimum and maximum receptive fields. Subsequently, the skip connection method is used to promote the interaction between branches of different scales. The skip connection method is to introduce the feature information extracted by the small-scale branches in the three extended convolution branches into the large-scale branches, realizing cross-scale information aggregation. Finally, the features (I1, I2, ..., I5) extracted by the five branches are aggregated and added to the original feature IO, and the global feature IF is generated through a 1×1 convolution layer, which is expressed as: I F =conv(concat(I1,I2,I3,I4,I5))+conv(I o )。 5. The dual-path multi-branch feature residual method for salient target detection according to claim 1, characterized in that: The CEM consists of four convolutional layers. The initial layer is a 3×3 convolutional layer, followed by two convolutional layers of size k×k, with the parameter k set to 3, 3, 5, 5. Finally, the output is enhanced by a 3×3 convolutional layer with 64 channels.

6. The dual-path multi-branch feature residual method for salient target detection according to claim 4, characterized in that: The MFRM uses a trunk and three sub-branches to downsample the simple fused features, perform convolution operations, and then upsample. The three sub-branches use sampling rates of 2, 4, and 8 to capture local features of different scales. The trunk is regarded as residual information and no operations are performed on it. Channel attention units are applied to the three branches to direct attention to weights at different scales. In the attention unit, global average pooling is first used on the input feature Fo to encode channel-level information: Among them, H and W represent the image height and width respectively, V represents the statistical vector, i and j represent the horizontal and vertical position information of the feature respectively, then the attention unit uses two fully connected layers with sigmoid and ReLU activation functions respectively to obtain the dependency between channels and obtain the weight vector VW: In w =FC sigmoid (FC ReLU (V)) Among them, FC represents a fully connected layer, sigmoid(.) represents the use of the sigmoid function for the formula in the brackets, ReLU represents the ReLU function, and then the channel-level multiplication between the input feature Fo and the weight scalar VW is used as the output Fa of the sub-branch. Finally, the output features of the three sub-branches are fused with the trunk features, and 3×3 convolution is used to reduce the aliasing effect caused by the upsampling operation.

7. The dual-path multi-branch feature residual method for salient target detection according to claim 6, characterized in that: The decoder uses a residual structure, and the MFRM output It is added element by element to generate the current MFRM result In the formula, up represents the upsampling operation. In addition, in order to enhance the edge features of the object, the last MFRM is edge supervised. The edge map is obtained from the input and output of MF and the output of MFRM using two consecutive convolutional structures and residual structures.

8. The dual-path multi-branch feature residual method for salient target detection according to claim 1, characterized in that: The image is defined using a binary cross entropy loss as follows: Where P and Y represent the prediction result and the corresponding label respectively, j represents the index of the pixel, and M represents the total number of pixels in P. The network also uses IoU loss to calculate the image-level loss, which is calculated as follows:

Images