A salient object detection method based on a double-branch network

Abstract

The application discloses a kind of based on significant target detection method of double branch network.First, two branches of CNN and Tranformer are used as the encoder of model respectively to extract the features of different branches;Then feature interaction module is used to interact the features extracted by the last three layers of two branches respectively, to obtain more rich initial feature information;Then the features of the corresponding layers of two branches are fused by feature fusion module;The fused features are then sent into edge supplement module to enhance edge information;Finally, the decoder is used to obtain the saliency map.The application obtains more rich global multi-scale features through feature interaction module.At the same time, in order to solve the difference between the features of different branches, feature fusion module is used, so that the features of two branches can be effectively fused.Moreover, for the common edge blur problem at present, the edge information of the features is enhanced by the edge supplement module.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer image processing, and particularly relates to a salient object detection method based on a double-branch network. BACKGROUND

[0002] The human visual system attempts to locate the most visually distinctive areas through the visual attention mechanism, which is the basis of salient object detection. The purpose of salient object detection is to identify and segment the most attractive objects in a scene. In recent years, salient object detection has received widespread attention in the world due to its successful application in image / video segmentation, image / video compression, image editing, image quality assessment, redirection, visual classification and many other research fields. More and more salient object detection networks based on deep learning are proposed, such as (Xia et al. 2017). Compared with many traditional algorithms that rely on low-level features, deep learning-based methods can more effectively utilize detailed information and semantic information. In particular, after the appearance of the Fully Convolutional Neural Network (FCN), salient object detection based on deep learning can better reflect its advantages.

[0003] The key to the SOD algorithm based on deep learning is to obtain strong feature expression. Hou et al. and Chen et al. use a deep convolutional network as an encoder to extract multi-level features and design various modules to fuse them in the FPN style. Ma et al. and Xu et al. experiment with various feature connection paths, suppressing the loss of details while avoiding semantic dilution. Pang et al. consider extracting scale information from adjacent features and design an AIM module to extract more information between adjacent features. They also design a SIM to obtain more feature information from the features themselves. Wei et al. consider the differences between features and design an FCM module that uses a feature multiplication method to avoid introducing noise as much as possible. In addition, Wei et al. generate a clear boundary saliency map by explicitly supervising the edge pixels. The widespread use of vision has also led to new developments in salient object detection. Liu et al. use T2Tvit as the backbone and design a pure transformer architecture multi-task decoder to perform RGB and RGB-D saliency detection. Many salient object detection methods currently begin to try to use double-branch networks, where the feature fusion of different branches often uses relatively simple addition or multiplication operations. However, this ignores the differences between the features of different branches, resulting in the inability to obtain high-quality saliency maps. At the same time, due to consecutive pooling or downsampling layers, pixel-level positioning information is often lost, resulting in blurred target boundaries. In view of the above problems, the present application proposes a new salient object detection method based on a double-branch network. SUMMARY

[0004] The technical problem to be solved by the present application is that in the detection of a salient object, the features of different branches cannot be effectively and sufficiently fused, and the boundary of the salient map is blurred due to insufficient edge information. In view of the above actual situation, the present application provides a salient object detection method based on a double-branch network. By applying a new feature fusion module and an edge supplement module, the performance of salient object detection is enhanced.

[0005] The method of the present application is to first use CNN and Tranformer two branches as the encoder of the model to extract features of different branches; then use a feature interaction module (FIM) to interact the features extracted by the two branches in the last three layers respectively to obtain more rich initial feature information; then use a feature fusion module (FFM) to fuse the features of the corresponding layers of the two branches; then send the fused features into an edge supplement module (ESM) to enhance the edge information; and finally obtain a salient map through a decoder.

[0006] A salient object detection method based on a double-branch network, comprising the following steps:

[0007] Step 1, pre-process the training data set and the test data set.

[0008] Step 2, input the input image into the model for training, and the specific steps are as follows:

[0009] 1) Use CNN and Tranformer backbone networks to extract features from the image respectively to obtain initial features.

[0010] 2) In order to make the initial features of the last three layers of the backbone network more diversified, a feature interaction module (FIM) is used to supplement rich multi-scale information.

[0011] 3) Use a feature fusion module (FFM) to fuse the features of the corresponding layers of the two branches.

[0012] 4) Use an edge supplement module (ESM) to enhance the edge information of the fused features.

[0013] 5) Finally, send the features into a decoder to obtain the final salient map.

[0014] Step 3, design a loss function to calculate the loss of the salient map.

[0015] Further, the specific method of step 1 is as follows:

[0016] 1) Download the training set and the test set respectively, use DUTS-TR as the training data set and DUT-TE as the test data set.

[0017] 2) Preprocessing of the training set. In the training process, in order to ensure the size of each picture is consistent, first adjust the image in the training data set to 256*256 size. In order to increase the data amount of samples in the data set, the training data set is augmented by using random flipping, translation and cropping.

[0018] 3) Preprocessing of the test set. In the test process, the size of each image is adjusted to 256*256, and then no post-processing is performed.

[0019] Further, the specific steps of step 2 are as follows:

[0020] 1) The CNN branch and the Transformer branch respectively adopt Resnet101 and Swin-B as the backbone network to extract the initial features of the image.

[0021] For the Resnet101 network, it is composed of five layers of Conv, Res2, Res3, Res4 and Res5. Among them, the Conv layer is changed from a 7*7 convolution layer to a 3*3 convolution layer. The features of the Conv layer are not used, so the available features can be denoted as {f i C |i=2,3,4,5}。

[0022] For the Swin-B network, it is composed of five layers of SBlock1, SBlock2, SBlock3, SBlock4 and SBlock5, and the features obtained by each layer can be denoted as {f j T |j=1,2,3,4,5}. Subsequently, the features obtained by SBlock3, SBlock4 and SBlock5 layers are mainly used.

[0023] 2) In order to make the initial features of the last three layers of the backbone network more diverse, a feature interaction module (FIM) is used to supplement rich multi-scale information.

[0024] The specific definition of the FIM model can be represented as:

[0025] I k =δ(conv1(conv2(gap(t k ))·t k )) (1)

[0026]

[0027]

[0028] In the formula, f k represents the features f kC Or the feature f obtained from the k-th layer of the Swin-B network k T . and t k This represents an intermediate variable generated during the calculation. k This represents the enhanced features obtained after the k-th layer features in the ResNet101 network undergo FIM. Or the enhanced features obtained after FIM of the features of the k-th layer in the Swin-B network. Where dconv represents a combination of dilated convolution, batch normalization, and ReLU activation function; up represents upsampling operation; conv1 represents a combination of 3*3 convolutional layer, BN, and ReLU; conv2 represents a combination of 1*1 convolutional layer, BN, and ReLU; gqp represents global average pooling; and δ represents the softmax function.

[0029] 3) Through step 2), the enhancement features of the last three layers of the ResNet101 network were obtained respectively. and and the enhancement features of the last three layers of the Swin-B network and The enhanced features obtained from the corresponding layers of the two branches are fused using a Feature Fusion Module (FFM) to obtain three fused features. and

[0030] The feature fusion module integrates self-attention and multimodal fusion mechanisms. Specifically, it can be defined as follows:

[0031]

[0032]

[0033]

[0034]

[0035] in, This represents the augmentation features of the m-th layer in the ResNet101 network. This represents the enhancement feature of the m-th layer in the Swin-B network. and This represents an intermediate variable in the calculation process. The fusion features of two branches enhanced features are obtained by FFM module; m is 3, 4, 5 respectively. conv3 is a 3*3 convolution layer; ChannelAtten represents channel attention; SpatialAtten represents spatial attention; [.] represents Concat operation; Residual represents a residual block.

[0036] 4) Use edge supplement module (ESM) to enhance the edge information of the fused features.

[0037] The Res2 layer in Resnet101 network is used to enhance the edge information of the features. In order to reduce the noise of the extracted features f2 C , the features f2 C are sent into the convolution attention module (CAM) for feature enhancement. Finally, the enhanced features of the Res2 layer are concatenated with the three fusion features and obtained in step 3), and finally three features F3, F4 and F5 are obtained.

[0038] 5) Finally, the three features F3, F4 and F5 are sent into the decoder (Decoder) to obtain the final saliency map.

[0039]

[0040]

[0041]

[0042] UP n n represents up-sampling n times; conv4 represents a 3*3 convolution layer; [.] represents connecting features by channels; and represent intermediate variables in the operation process; S represents the final obtained saliency map.

[0043] Step 3, design loss function to calculate the loss of saliency map.

[0044] The IoU loss and binary cross-entropy loss are used to train the entire network end-to-end. It can be defined as:

[0045] L tol = L bce + L iou (11)

[0046] Wherein, the IoU loss can be expressed as:

[0047]

[0048] where (i, j) denotes the pixel position of the image. H and W denote the height and width of the image, respectively. sum(i, j) denotes the sum of the predicted saliency map and the ground-truth at pixel point (i, j). mul(i, j) denotes the product of the predicted saliency map and the ground-truth at pixel point (i, j). e is set to 1e-6 to prevent division by zero.

[0049] Binary cross-entropy loss L bce which can be expressed as:

[0050]

[0051] where g denotes the ground-truth, and p denotes the predicted saliency map.

[0052] The present application has the following advantages:

[0053] The present application introduces a feature interaction module (FIM) to obtain more rich global multi-scale features. At the same time, in order to solve the difference between the features of different branches, a feature fusion module (FFM) is adopted, so that the features of the two branches can be effectively fused. Moreover, in view of the common edge blur problem, the present application introduces an edge supplement module (ESM) to enhance the edge information of the features. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a model framework diagram of an embodiment of the present application;

[0055] Figure 2 is a structure diagram of a feature interaction module (FIM) of an embodiment of the present application;

[0056] Figure 3 is a structure diagram of a feature fusion module (FFM) of an embodiment of the present application;

[0057] Figure 4 is a structure diagram of a decoder (Decoder) of an embodiment of the present application.

[0058] DETAILED DESCRIPTION

[0059] The present application will be described in detail below in combination with specific implementation manners.

[0060] The salient object detection method based on the dual-branch network proposed by the present application is mainly implemented according to the following steps.

[0061] Step 1, pre-processing operation is performed on the training data set and the test data set, and the specific steps are as follows:

[0062] 1) Download the training set and the test set respectively, and in this invention, DUTS-TR is used as the training data set and DUT-TE is used as the test data set.

[0063] 2) Preprocess the training set. In the training process, in order to ensure the size of each picture is consistent, first adjust the images in the training data set to 256*256 size. In order to increase the data amount of the samples in the data set, the training data set is augmented by using random flipping, translation and cropping.

[0064] 3) Preprocess the test set. In the test process, the size of each image is adjusted to 256*256, and then no post-processing is performed.

[0065] Step 2, input the preprocessed image into the model for training, and the specific steps are as follows:

[0066] 1) In order to reduce the calculation consumption and memory usage, the CNN branch and the Transformer branch of the invention respectively use Resnet101 and Swin-B as the backbone network to extract the initial features of the image, and the backbone network is as shown in Figure 1 .

[0067] For the Resnet101 network, it mainly consists of five layers of Conv, Res2, Res3, Res4 and Res5. Among them, the Conv layer is changed from a 7*7 convolution layer to a 3*3 convolution layer. Although the Conv layer contains a lot of edge information, it has a lot of noise, so the features of this layer are not used. Therefore, the available features can be recorded as {f i C |i=2,3,4,5}.

[0068] For the Swin-B network, it mainly consists of five layers of SBlock1, SBlock2, SBlock3, SBlock4 and SBlock5, and the features obtained by each layer can be recorded as {f j T |j=1,2,3,4,5}. Since the high-level features of the Swin-B network have more accurate global semantic information, the invention mainly uses the features obtained by the SBlock3, SBlock4 and SBlock5 layers.

[0069] 2) In order to make the initial features of the last three layers of the backbone network more diverse, a feature interaction module (FIM) is used to supplement rich multi-scale information.

[0070] After obtaining the initial features of the last three layers of the Resnet101 network and the Swin-B network, the application designs a feature interaction module to obtain more initial feature information. The features extracted from the backbone have sufficient size information and hierarchical information between details and semantics. The FIM makes full use of them to provide more useful information for the initial features.

[0071] As shown in Figure 2 , the specific definition of the FIM model can be represented as:

[0072] I k = δ(conv1(conv2(gap(t k ))·t k )) (1)

[0073]

[0074]

[0075] In the formula, f k represents the feature f k C obtained by the kth layer of the Resnet101 network, or the feature f k T obtained by the kth layer of the Swin-B network. and t k represent intermediate variables generated during the operation. K I C represents the enhanced feature obtained after the kth layer feature of the Resnet101 network passes through the FIM, or the enhanced feature obtained after the kth layer feature of the Swin-B network passes through the FIM. wherein dconv represents the combination of dilated convolution, batch normalization and Relu activation function; up represents the up-sampling operation; conv1 represents the combination of 3*3 convolution layer, BN and Relu; conv2 represents the combination of 1*1 convolution layer, BN and Relu; gap represents global average pooling; and δ represents the softmax function.

[0076] 3) Through step 2), the enhanced features of the last three layers of the Resnet101 network are obtained respectively and and the enhanced features of the last three layers of the Swin-B network are obtained respectively and Through the feature fusion module (FFM), the enhanced features obtained by the corresponding layers of the two branches are fused respectively, and three fusion features are obtained and

[0077] In order to combine the enhanced features from the last three layers of Resnet101 network and Swin-B network more effectively, the application proposes a new feature fusion method (FFM), as shown in Figure 3 The feature fusion module combines self-attention and multi-modal fusion mechanism. The specific definition can be represented as:

[0078]

[0079]

[0080]

[0081]

[0082] wherein, represents the enhanced feature of the mth layer in the Resnet101 network, represents the enhanced feature of the mth layer in the Swin-B network. and represent intermediate variables in the operation process. represents the fusion feature obtained by the FFM module from the enhanced features of the two branches; m is 3, 4, and 5, respectively. conv3 is a 3*3 convolution layer; ChannelAtten represents channel attention; SpatialAtten represents spatial attention; [.] represents the Concat operation; Residual represents a residual block.

[0083] 4) An edge supplement module (ESM) is used to enhance the edge information of the fused features. The module structure is as shown in Figure 1 .

[0084] In the Resnet101 network, compared with the Conv layer, the Res2 layer not only contains more rich edge information, but also has less noise, so the application adopts the Res2 layer to enhance the edge information of the features. In order to reduce the noise of the extracted feature f2 C , the feature f2 C is sent into the convolution attention module (CAM) for feature enhancement. Finally, the enhanced features of the Res2 layer are respectively concatenated with the three fusion features and obtained in step 3), and finally three features F3, F4 and F5 are obtained.

[0085] 5) Finally, the three features F3, F4 and F5 are sent into the decoder (Decoder) to obtain the final saliency map. The module structure of the decoder is as shown in Figure 4 .

[0086]

[0087]

[0088]

[0089] where UP n denotes upsampling n times; conv4 denotes a 3*3 convolutional layer; [.] denotes concatenating features in channel; and represent intermediate variables in the operation process; S denotes the final obtained saliency map.

[0090] Step 3, design a loss function to calculate the loss of the saliency map.

[0091] The entire network is trained end-to-end using the IoU loss and the binary cross-entropy loss. It can be defined as:

[0092] L tol = L bce + L iou (11)

[0093] where the IoU loss can be expressed as:

[0094]

[0095] where (i,j) represents the pixel position of the image. H and W represent the height and width of the image, respectively. sum(i,j) represents the sum of the predicted saliency map and the ground-truth at pixel point (i,j). mul(i,j) represents the product of the predicted saliency map and the ground-truth at pixel point (i,j). e is set to 1e-6 to prevent division by zero.

[0096] The binary cross-entropy loss L bce can be expressed as:

[0097]

[0098] where g represents the ground-truth, and p represents the predicted saliency map.

[0099] The above is a further detailed description of the present application in combination with specific / preferred embodiments, and cannot be considered as limiting the specific implementation of the present application to these descriptions. For ordinary skilled persons in the technical field to which the present application belongs, without departing from the concept of the present application, they can make several alternatives or modifications to these described embodiments, and these alternatives or modifications should be considered as belonging to the protection scope of the present application.

[0100] Portions of the disclosure which are not specifically recited herein are deemed to be known by those skilled in the art.

Claims

1. A method for salient object detection based on a two-branch network, characterized in that, The method comprises the following steps: Step 1, preprocessing operation is performed on the training data set and the test data set; Step 2, inputting the input image into the model for training, and the specific steps are as follows: 1) initial features are obtained by using CNN and Transformer backbone network to extract features from the image respectively; 2) in order to make the initial features of the last three layers of the backbone network more diversified, a feature interaction module FIM is used to supplement rich multi-scale information; 3) the features of the corresponding layers of the two branches are fused through a feature fusion module FFM; 4) an edge supplement module ESM is used to enhance the edge information of the fused features; 5) finally, the features are sent into the decoder to obtain the final saliency map; Step 3, designing a loss function to calculate the loss of the saliency map; The specific steps of step 2 are as follows: 1) Resnet101 and Swin-B are used as backbone networks in the CNN branch and the Transformer branch respectively to extract initial features of the image; For the Resnet101 network, five layers of Conv, Res2, Res3, Res4 and Res5 are included; among them, the Conv layer is changed from a 7*7 convolutional layer to a 3*3 convolutional layer; no features of the Conv layer are used, so the available features are denoted as {f i C |i=2,3,4,5} For the Swin-B network, it is composed of five layers of SBlockl, SBlock2, SBlock3, SBlock4, and SBlock5, and the features obtained by each layer are denoted as {f j T |j = 1, 2, 3, 4, 5}; and the features obtained by SBlock3, SBlock4, and SBlock5 layers are mainly used subsequently; 2) in order to make the initial features of the last three layers of the backbone network more diversified, a feature interaction module FIM is used to supplement rich multi-scale information; The specific definition of the FIM model is as follows: （1） （2） （3） In the formula, represents the feature f obtained by the kth layer in the Resnet101 network k C or the feature f obtained by the kth layer in the Swin-B network k T ; , and then represents an intermediate variable generated in the operation process; represents the enhanced feature obtained after the kth layer feature in the Resnet101 network passes through the FIM or the enhanced feature obtained after the kth layer feature in the Swin-B network passes through the FIM ; wherein, represents the combination of dilated convolution, batch normalization and Relu activation function; represents the up-sampling operation; represents the combination of 3*3 convolution layer, BN and Relu; represents the combination of 1*1 convolution layer, BN and Relu; represents the global average pooling; represents the softmax function; 3) Through step 2), the enhanced features of the last three layers of the Resnet101 network are obtained respectively , and ; and the enhanced features of the last three layers of the Swin-B network are obtained respectively , and ; and the enhanced features obtained by the two branches corresponding to the layers are fused respectively through the feature fusion module FFM, and three fusion features , and are obtained; The feature fusion module fuses self-attention and multi-modal fusion mechanism; the specific definition is as follows: （4） （5） （6） （7） wherein, represents the enhanced feature of the mth layer in the Resnet101 network, represents the enhanced feature of the mth layer in the Swin-B network; , and then represents an intermediate variable in the operation process; represents the fusion feature obtained by the FFM module from the enhanced features of the two branches; m is 3, 4, 5, respectively; is a 3*3 convolution layer; represents channel attention; represents spatial attention; [.] represents the Concat operation; represents a residual block; 4) an edge supplement module ESM is used to enhance the edge information of the fused features; The Res2 layer in the ResNet101 network is used to enhance the edge information of the features; in order to reduce the f2 of the features extracted by the Res2 layer. C The noise will feature f2 C The features are fed into the convolutional attention module (CAM) for feature enhancement; finally, the features enhanced by the Res2 layer are fused with the three features obtained in step 3). , and Performing the cat operation ultimately yields three features. , and ; 5) Finally, the three features , and are fed into the decoder Decoder to obtain the final saliency map; （8） （9） （10） where, denotes up-sampling sub-sampling denotes a 3*3 convolutional layer; [.] denotes concatenating features in channels; and denotes intermediate variables in the operation process; S denotes the final obtained saliency map. 2.The method of claim 1, wherein, The specific method of step 1 is as follows: 1) download the training set and the test set respectively, use DUTS-TR as the training data set and DUT-TE as the test data set; 2) preprocessing is performed on the training set; in the training process, in order to ensure the consistency of the size of each picture, first, the images in the training data set are adjusted to 256*256 size; in order to increase the data amount of the samples in the data set, random flipping, translation and clipping are used to expand the training data set; 3) preprocessing is performed on the test set; in the test process, the size of each image is adjusted to 256*256, and then no post-processing is performed.

3. The method of claim 2, wherein the method comprises: The specific method of step 3 is as follows: The whole network is trained end to end by using IoU loss and binary cross entropy loss; the definition is as follows: （11） Wherein, the IoU loss is represented as: （12） where, represents the pixel position of the image; H and W represent the height and width of the image, respectively; represents the sum of the predicted saliency map and the ground-truth at the pixel point represents the product of the predicted saliency map and the ground-truth at the pixel point represents the product of the predicted saliency map and the ground-truth at the pixel point e is set to 1e-6 to prevent division by zero; Binary cross-entropy loss is represented as: （13） wherein, denotes ground-truth, denotes predicted saliency map.

Images