Significant target detection method based on pvt interaction network

Abstract

The application discloses a salient object detection method based on a PVT interaction network, global features are extracted through layer-by-layer fusion of the PVT network; a high-level semantic information fusion module is designed for the top layer of an encoder block to fully mine semantic information of high-level features; a self-refinement module is designed for other encoder blocks except the top layer to extract rich detail information; the high-level semantic information is propagated from top to bottom and fused with features at all levels in a feature reconciliation manner; a dense lightweight refinement decoder constantly fuses the high-level semantic information and effective information perceived by the encoder to perceive multi-scale information, further expands the receptive field of the whole network, and constantly refines salient objects. The application can extract the most interesting target content of human eyes in any given image, suppress background interference, and pay more attention to salient regions.

Description

TECHNICAL FIELD

[0001] The application relates to a salient object detection method, in particular to a salient object detection method based on a PVT interaction network, and belongs to the field of computer vision. BACKGROUND

[0002] In recent years, the development of the new media industry is becoming more and more rapid, and digital information such as short videos and image tweets is becoming more and more closely related to human life. The image and video information appearing in the Internet greatly enriches the original life and entertainment of people. Facebook company released a report that the company's servers receive more than 450 million pictures uploaded by users every day on average, and these picture data are still growing in an explosive manner. It can be seen that in such a large number of digital images, it is difficult to extract valuable information only by human senses, and it needs to be realized by a computer. Therefore, it brings greater challenges to the research of computer vision.

[0003] Digital image information is widely used in network resource transmission because it can very intuitively reflect the content that human beings want to express. However, the effective identification of these digital image information is a difficult problem, and even through computer resources, it will also produce a greater burden. Therefore, it is necessary to make full use of limited computer resources and pay more attention to important content in digital image information, so as to reduce the burden of collecting information. In addition, based on the particularity of the human visual system, people usually pay special attention to the key areas in the image, such as photos of people, food and scenery in WeChat Moments, and ignore the useless information in the background area. It plays an important role in quickly processing images. Therefore, how to apply this ability to pay attention to the key areas of the image to the computer so as to complete the processing task of complex images has become the focus of the current computer vision researchers.

[0004] In recent years, the U-shaped structure proposed by Ronneberger et al. can extract rich feature information by constructing a multi-level top-down path and has achieved good performance, so it has attracted great attention. For example, many networks proposed by Cheng et al. use a U-shaped multi-scale hierarchical encoder-decoder structure as the basic structure of the network. However, the above method extracts features through convolution operations with local receptive fields, and for some cases of saliency target detection relying on global semantic information in some scenes, it cannot capture the relevance of the global context well. And the above method uses a cross-layer connection structure to directly connect the features of the encoder to the decoder, lacks interference control, and can introduce misleading context information into the decoder, causing the truly useful features of each layer to be not fully utilized. In addition, these methods do not fully consider the relationship between the features of each layer and the information provided, and the information is not fully mined, and rich context information cannot be obtained. Most of the current convolutional neural network (CNN) based works use pre-trained image classification models for saliency detection, such as VGG and ResNet as encoders, focusing on designing effective decoders by aggregating multi-level features. However, due to the model structure characteristics of the convolutional neural network (CNN), which is modeled for local information aggregation, it is difficult to model long periods, and for some cases of saliency target detection relying on global semantic information in some scenes, it cannot capture the global context information well. Therefore, it is still a great challenge to accurately and completely extract saliency targets from complex scenes. Recently, we found that the Pyramid Vision Transformer (PVT) has a complete sequence of long-range dependencies in image processing and is good at modeling global dependencies, opening up new avenues for complete extraction of saliency targets. SUMMARY

[0005] The purpose of the present application is to propose a saliency target detection method based on PVT interaction network.

[0006] To solve the above technical problems, the technical solution to be solved by the present application is: a saliency target detection method based on PVT interaction network, comprising the following steps:

[0007] Step 1: image preprocessing: adjust the size of the input image to a preset size tensor X;

[0008] Step 2: Establishing a PVT interaction network: the PVT interaction network comprises first to fourth feature processing units and an advanced semantic fusion module ASFM; the first to third feature processing units are of the same structure; the first feature processing unit comprises a feature encoder PVTE1, a self-refinement module SRM1, a feature harmonization and dense light refinement decoder DLRB1; the fourth feature processing unit comprises a feature encoder PVTE4 and an advanced semantic fusion module ASFM; after the tensor X is processed by the feature encoders PVTE1-PVTE4 in turn, first to fourth feature tensors are obtained;

[0009] The fourth feature tensor and its adjacent feature tensors are processed by the advanced semantic fusion module ASFM to obtain fourth advanced semantic information; the first feature tensor and its adjacent feature tensors are processed by the first processing unit to obtain first encoder effective information; the third encoder effective information and the fourth advanced semantic information are input into the dense light refinement decoder DLRB3 after being harmonized by features; and a first decoded feature vector is output as a saliency map;

[0010] Step 3: Detecting a saliency map: inputting a tensor, and obtaining a saliency map after processing by a PVT interaction network.

[0011] Further, the tensor size is 256x256x3, and the sizes of the first to fourth feature tensors are 64x64x64, 32x32x128, 16x16x320 and 8x8x512 respectively.

[0012] Further, the feature encoders PVTE3 and PVTE4 input the advanced semantic fusion module, generate a rough saliency map for position guidance Pred through channel / space attention, convolution, activation and reverse attention operations; the current branch PVTE4 in the ASFM is expanded by three dilated convolutions to expand the receptive field, uses a stitching-convolution operation to summarize and fuse the output features, and obtains a foreground map through a channel attention mechanism to reduce redundant information and then through spatial attention a background map is obtained through a reverse attention operation, and the background map is used to highlight the non-salient area at the feature level, the obtained foreground map and background map are spliced and fused, and the result is denoted as The adjacent branch provides global auxiliary information for the current branch, and the branch from before to the current is denoted as After effective coordination, the output features of the two branches are integrated with the original branch features, and the result is denoted as The integrated features are activated through feature selection to generate a rough saliency map Pred for position guidance. It can be expressed as:

[0013]

[0014] In the formula, is element addition, and sigmod(·) is an activation function, and conv1×1 (·) is a convolutional layer with a size of 1x1.

[0015] By applying the advanced semantic fusion module, each salient region can be accurately activated, multi-level advanced semantic information can be effectively fused, and the contrast between the salient region and the non-salient region can be enhanced to obtain an accurate position guidance map.

[0016] Further, the self-refinement module SRM i The adjacent features are taken as inputs, and sampling operations are performed to make the feature scales the same, and then the channel splicing and convolution operations are performed to effectively fuse the detailed information of the adjacent features. The self-refinement module SRM i The first, second, and third feature tensors are subjected to channel / space attention, reverse attention, convolution, up / down sampling, and CBR operations to obtain saliency target detailed information from different dimensions. The obtained attention map is reversed to obtain a reverse attention map In addition, the fused features are subjected to channel and space attention to mine detailed information from the global, and a spatial channel attention map is obtained Finally, the three results are added and fused by a convolutional layer to obtain the SRM i output of the self-refinement module SRM which can be expressed as:

[0017]

[0018]

[0019] In the formula, conv 3×3 (·) is a convolutional layer with a size of 3x3, and is a channel-by-channel multiplication, is an element-by-element multiplication, Down(·) is a down-sampling operation, and Up(·) is an up-sampling operation. conv 1×1 (·) is a convolutional layer with a size of 1x1, SA(·) / CA(·) is a spatial / channel attention, and CBR is a convolution, normalization, and activation operation.

[0020] Further, the position guidance map containing advanced semantic information and the detailed feature map are fused by feature reconciliation to generate a final output which is delivered to the decoder to generate an output of the decoder feature extractor. This can be expressed as:

[0021]

[0022] wherein the dense lightweight refinement decoder DLRB3 is a saliency map with the same size as the input image;

[0023] The feature encoders PVTE1-PVTE3 take the self-refinement module SRM i as input; and the self-refinement module SRMi The feature reconciliation reduces the dimension of the output of the advanced semantic fusion module ASFM and uses a bilinear interpolation method for upsampling, so that the output dimension and scale are the same as the output of the SRM i The effective information of the encoder, the output of the advanced semantic fusion module ASFM and the output of the self-refinement module SRM i are fused and transmitted into the dense light refinement decoder DLRB.

[0024] Further, the dense light refinement decoder DLRB first performs convolution operation on the input feature map by using deep separable convolution with a dilation rate of {2, 4, 6}; after each convolution operation, 1x1 convolution is used to combine the captured features; and then a bilinear interpolation method is used for upsampling to obtain the output feature map of the original size.

[0025] By using the above technical solutions, the present application achieves the following technical effects:

[0026] The pyramid vision transformer is introduced to model the global dependency relationship and obtain more powerful and robust features; the advanced semantic fusion module ASFM is designed for the top layer of the encoder to collect high-level semantic information and generate a rough saliency map for position guidance, and the saliency map is propagated to each level of feature in a top-down progressive path; in order to obtain more details of the saliency map, the self-refinement module SRM i is designed for the decoder blocks other than the top layer to mine the detail information in the low-level features, and the high-level semantic information provided by the ASFM is fused through feature reconciliation, so that more effective context information in the encoder is transmitted into the decoder to improve the distinguishability of the whole network; in addition, the feature interaction strategy is used to coordinate adjacent features to fully mine the context information contained in the adjacent features and expand the coverage of feature interaction; and the dense light refinement decoder DLRB i is introduced to better fuse the high-level semantic information and the encoder detail information.

[0027] Further, the dense light refinement decoder DLRB first performs convolution operation on the input feature map by using deep separable convolution with a dilation rate of {2, 4, 6}; after each dilation convolution operation, 2x2 convolution is used to combine the captured features; and then a bilinear interpolation method is used for upsampling to obtain the output feature map of the original size.

[0028] Further, the dense light refinement decoder DLRB first performs convolution operation on the input feature map by using deep separable convolution with a dilation rate of {2, 4, 6}; after each dilation convolution operation, 3x3 convolution is used to combine the captured features; and then a bilinear interpolation method is used for upsampling to obtain the output feature map of the original size. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a framework diagram of the present application.

[0030] Figure 2 is a structure diagram of the advanced semantic fusion module ASFM of the present application.

[0031] Figure 3 is a structure diagram of the self-refinement module SRM of embodiment 1 of the present application.

[0032] Figure 4 is a structure diagram of the self-refinement module SRM of embodiment 2 of the present application.

[0033] Figure 5 is an input image of embodiment 1 of the present application.

[0034] Figure 6 is a saliency map detected by embodiment 1 of the present application. DETAILED DESCRIPTION

[0035] Embodiment 1:

[0036] Referring to Figure 1 , a saliency object detection method based on a PVT interaction network, comprising the following steps:

[0037] Step 1: image preprocessing: adjusting the size of the input image to a preset size of a tensor X, in this embodiment, the tensor size is 256x256x3;

[0038] Step 2: establishing a PVT interaction network: the PVT interaction network includes first to fourth feature processing units and a dense light refinement decoder DLRB; the first to third feature processing units have the same structure; the first feature processing unit includes a feature encoder PVTE1, a self-refinement module SRM1, a feature harmonization, and a dense light refinement decoder DLRB1; the fourth feature processing unit includes a feature encoder PVTE4 and an advanced semantic fusion module ASFM; after the tensor X is processed by the feature encoders PVTE1-PVTE4 in turn, first to fourth feature tensors are obtained;

[0039] the fourth feature tensor and its adjacent feature tensors are processed by the advanced semantic fusion module ASFM to obtain fourth advanced semantic information; the first feature tensor and its adjacent feature tensors are processed by the first processing unit to obtain first encoder effective information; after the first encoder effective information and the fourth advanced semantic information are harmonized by the feature, they are input into the dense light refinement decoder DLRB3; and a first decoding feature vector is output as a saliency map;

[0040] Step 3: detecting a saliency map: inputting a tensor, processing it by a PVT interaction network to obtain a saliency map.

[0041] The sizes of the first to fourth feature tensors in the embodiment are 64x64x64, 32x32x128, 16x16x320, and 8x8x512, respectively;

[0042] The high-level semantic fusion module ASFM at the top layer of the encoder obtains multi-scale features through dilated convolution operation, further expands the receptive field, collects global context information, and more accurately captures the exact position of the salient object;

[0043] The feature encoders PVTE3 and PVTE4 input the high-level semantic fusion module, generate a rough saliency map for position guidance Pred through channel / space attention, convolution, activation, and reverse attention operation. The ASFM includes two branches, the current branch PVTE4 is denoted as The features with different dilation rates are obtained through three dilated convolutions with dilation rates of 1, 3, and 5 The receptive field is expanded, and the output features are summarized and fused using the concatenation-convolution operation to obtain features with rich context clues The features after summarizing and fusing are obtained through channel attention mechanism to obtain features to reduce redundant information, and then the foreground map is obtained through spatial attention denoted as which can be expressed as:

[0044]

[0045]

[0046]

[0047] In the formula, DConv σ is a dilated convolution, is a convolution kernel with 3x3, r i is a dilation rate, Conv σ is a convolution layer, Concat is a concatenation operation between channel axes, CA(·) is channel attention, and is multiplied by channel.

[0048] The background map is obtained through reverse attention operation The background map is used to highlight the non-salient area at the feature level, and the obtained foreground map and background map are concatenated and fused denoted as which can be expressed as:

[0049]

[0050]

[0051] In the formula, is element subtraction, SA(·) is spatial attention, is element-wise multiplication, conv 3×3 (·) is a 3x3 convolutional layer, Concat is a concatenation operation between channel axes, SA(·) is spatial attention, is element-wise multiplication, Down(·) is down-sampling operation.

[0052] The adjacent branch provides global auxiliary information for the current branch, denoted as The branch from previous to current is denoted as After effective coordination, we integrate the output features of the two branches with the original branch features, and then activate the integrated features through feature selection to generate a rough saliency map Pred for location guidance. It can be expressed as follows:

[0053]

[0054]

[0055] In the formula, SA(·) is spatial attention, is element-wise multiplication, Down(·) is down-sampling operation, is element-wise addition, sigmod(·) is an activation function, conv 1×1 (·) is a 1x1 convolutional layer.

[0056] By applying the advanced semantic fusion module, each salient region can be accurately activated and comprehensively coordinated, and multi-level advanced semantic information can be effectively fused to enhance the contrast between salient and non-salient regions and obtain an accurate location guidance map.

[0057] The feature encoder PVTE i and its adjacent encoded features input the self-refinement module SRM i ; the channel / space attention, reverse attention, convolution, up / down sampling and CBR operations are performed on the feature tensor to obtain saliency target detail information from different dimensions. It can be expressed as follows:

[0058]

[0059] In the formula, conv 3×3 (·) is a 3x3 convolutional layer, is a channel multiplication, Down(·) is a down-sampling operation, and Up(·) is an up-sampling operation.

[0060] The obtained attention map is reversed to obtain a reverse attention map In addition, the fused features are subjected to channel and spatial attention to mine detail information from the global, and obtain a spatial channel attention map Finally, the three results are added and fused through a convolutional layer to obtain SRM ithe output of the ASFM

[0061]

[0062]

[0063]

[0064] conv 1×1 (·) is a 1x1 convolutional layer, SA(·) / CA(·) is spatial / channel attention, and is element-wise multiplication, CBR is a convolution, normalization, and activation operation.

[0065] To prevent high-level semantic information from being diluted in the top-down path, we provide high-level features to the feature map of each feature layer using the feature harmonics provided by the advanced semantic fusion module ASFM, and provide multi-scale information to each decoder:

[0066] Feature harmonics performs dimensionality reduction on the output of the advanced semantic fusion module ASFM and uses bilinear interpolation for upsampling, so that it is the same as the output dimension and scale of the SRM i ; element-wise multiplication fuses semantic and detail information and continuously fuses various features at the element level; by feature harmonics, local context information of different scales is obtained, and then these information is combined to ensure that feature maps of different scales can be effectively fused. It can be expressed as:

[0067]

[0068] Up(·) is an upsampling operation, is element-wise multiplication, is element-wise addition.

[0069] The self-refinement module (SRM i ) fuses adjacent branches to capture details in low-level features from different dimensions, and fuses the position guidance information generated by the ASFM through feature harmonics to generate more rich and diverse context information.

[0070] Processing process of the dense lightweight refinement decoder DLRB:

[0071] The feature map after feature harmonics is used as input, and three depth separable convolutions DSConvs with expansion rates of 2, 4, and 6 are used in turn. The expanded DSConvs expand the receptive field and fully capture multi-scale features; a 1x1 convolutional layer is used after each DSConvs, which is responsible for merging the captured features; finally, a 3x3 convolutional layer is used to remove the aliasing effect after feature merging. Taking DLRB3 as an example, it can be expressed as:

[0072]

[0073]

[0074]

[0075] wherein is element-wise addition, conv 1×1 (·) is a 1x1 convolutional layer, DSconv r (·) is a dilated separable convolution, r is a dilation rate, conv 3×3 (·) is a 3x3 convolutional layer.

[0076] wherein the dense lightweight refinement decoder DLRB3 is a saliency map with the same size as the input image;

[0077] In this way, the DLRB can perceive multi-scale information, further expand the receptive field of the entire network, and bring strong feature representation in the refinement stage. Reduce the aliasing effect caused by model upsampling and capture more detailed information.

[0078] Embodiment 2:

[0079] The difference from embodiment 1 is that the dense lightweight refinement decoder DLRB first convolves the input feature map using deep separable convolution with a dilation rate of {2, 4, 6}; after each dilated convolution operation, 2x2 convolution is used to merge the captured features; and then bilinear interpolation is used for upsampling to obtain the original size of the output feature map.

[0080] Embodiment 3:

[0081] The difference from embodiments 1 and 2 is that the dense lightweight refinement decoder DLRB first convolves the input feature map using deep separable convolution with a dilation rate of {2, 4, 6}; after each dilated convolution operation, 3x3 convolution is used to merge the captured features; and then bilinear interpolation is used for upsampling to obtain the original size of the output feature map.

[0082] It is specially pointed out that at present, the technical scheme of the application has been carried out in a pilot test, user use research has been carried out in a small range, the research results show that the user satisfaction is high, that is, the preparation of the achievement transformation will start soon, and the intellectual property risk early warning research and other work are also carried out.

Claims

1. A salient target detection method based on PVT interaction networks, characterized in that, Includes the following steps: Step 1: Image preprocessing: Resize the input image to a tensor X of a preset size; Step 2: Establish the PVT interaction network: The PVT interaction network includes the first to fourth feature processing units and the advanced semantic fusion module ASFM; the first to third feature processing units have the same structure; The first feature processing unit includes a feature encoder PVTE1, a self-refinement module SRM1, and a feature harmonic and dense lightweight refinement decoder DLRB1; the fourth feature processing unit includes a feature encoder PVTE4 and an advanced semantic fusion module ASFM; after the tensor X is processed by the feature encoders PVTE1-PVTE4 in sequence, the first to fourth feature tensors are obtained. The fourth feature tensor and its neighboring feature tensors are processed by the Advanced Semantic Fusion Module (ASFM) to obtain the fourth advanced semantic information; the first feature tensor and its neighboring feature tensors are processed by the first processing unit to obtain the first encoder effective information; the first encoder effective information and the fourth advanced semantic information are input into the Dense Lightweight Refinement Decoder (DLRB1) after feature harmonization; the first decoded feature vector is output as a saliency map. Self-refinement module SRM i Taking adjacent features as input, the feature scale is made the same through sampling, and then the detailed information of adjacent features is effectively fused through channel concatenation and convolution. This is the self-refinement module SRM. i Channel / spatial attention, reverse attention, convolution, upsampling / downsampling, and CBR operations are performed on the first, second, and third feature tensors to obtain salient target detail information from different dimensions. The resulting attention map is then inverted to obtain the reverse attention map. Furthermore, the fused features are used to extract detailed information from the global perspective through channel and spatial attention to obtain a spatial channel attention map. Finally, the three results are added and fused through a convolutional layer to obtain the SRM. i Output The expression is as follows: In the formula conv 3×3 (·) represents a 3x3 convolutional layer, and ⊙ represents channel-wise multiplication. For element-wise multiplication, Down(·) is the downsampling operation, Up(·) is the upsampling operation, and conv 1×1 (·) represents a 1x1 convolutional layer, SA(·) / CA(·) represents spatial / channel attention, and CBR represents convolution, normalization, and activation operations.

2. The salient target detection method based on PVT interaction network according to claim 1, characterized in that, The input tensor size is 256×256×3, and the sizes of the first to fourth feature tensors are 64×64×64, 32×32×128, 16×16×320, and 8×8×512, respectively.

3. The salient target detection method based on PVT interactive networks according to claim 1, characterized in that, Feature encoder PVTE3 and PVTE4 are input into the advanced semantic fusion module, which generates a coarse saliency map for location-guided Pred through channel / spatial attention, convolution, activation, and inverse attention operations. The current branch PVTE4 in ASFM expands the receptive field through three dilated convolutions, and uses concatenation-convolution operations to summarize and fuse the output features. The summarized and fused features are then processed through a channel attention mechanism to reduce redundant information, and finally through spatial attention to obtain the foreground map. A background image is obtained through a reverse attention operation. This background image is then used to highlight non-salient regions at the feature level. The resulting foreground and background images are then stitched together and merged, denoted as […]. Adjacent branches provide global auxiliary information to the current branch; the previous branch leading to the current branch is denoted as... After effective coordination, the output features of the two branches are integrated with the original branch features and denoted as follows: The integrated features are activated through feature selection to generate a coarse saliency map Pred for location guidance, as described below; In the formula For element-wise addition, sigmoid(·) is the activation function, and conv is the function for adding elements together. 1×1 (·) represents a 1x1 convolutional layer. By applying an advanced semantic fusion module, each salient region is activated accurately and comprehensively, effectively fusing multi-level advanced semantic information, enhancing the contrast between salient and non-salient regions, and obtaining an accurate location guidance map.

4. The salient target detection method based on PVT interaction network according to claim 1, characterized in that, Location guidance maps and detail feature maps containing high-level semantic information The final output is generated through feature harmonics. This is then fed into the decoder to generate the output of the decoder's feature generator, which can be expressed as: The dense lightweight refinement decoder DLRB3 is a saliency map of the same size as the input image; Feature encoders PVTE1-PVTE3 and feature input self-refinement modules (SRM) i SRM (Self-Refining Module) i Explore detailed information from different dimensions; Feature harmonicization performs dimensionality reduction on the output of the Advanced Semantic Fusion Module (ASFM) and upsamples it using bilinear interpolation to make it compatible with the SRM. i The output dimension and scale are the same; the encoder's effective information, the output of the Advanced Semantic Fusion Module (ASFM), and the self-refinement module (SRM) are all present. i The outputs are then fused and fed into the dense lightweight refinement decoder DLRB.

5. The salient target detection method based on PVT interactive networks according to claim 4, characterized in that, The Dense Lightweight Refinement Decoder (DLRB) first performs convolution operations on the input feature map using depthwise separable convolutions with dilation rates of 2, 4, and 6; after each dilation convolution operation, a 1x1 convolution is used to merge the captured features. Then, the original size of the output feature map is obtained by upsampling using bilinear interpolation.

6. The salient target detection method based on PVT interactive networks according to claim 4, characterized in that, The Dense Lightweight Refinement Decoder (DLRB) first performs convolution operations on the input feature map using depthwise separable convolutions with dilation rates of 2, 4, and 6; after each dilation convolution operation, a 2x2 convolution is used to merge the captured features. Then, the original size of the output feature map is obtained by upsampling using bilinear interpolation.

7. The salient target detection method based on PVT interactive networks according to claim 4, characterized in that, The Dense Lightweight Refinement Decoder (DLRB) first performs convolution operations on the input feature map using depthwise separable convolutions with dilation rates of 2, 4, and 6; after each dilation convolution operation, a 3x3 convolution is used to merge the captured features. Then, the original size of the output feature map is obtained by upsampling using bilinear interpolation.

Images