A salient target detection method based on PVT-gated networks

Abstract

This invention discloses a salient object detection method based on a PVT-gated network. It extracts global features through layer-by-layer fusion of the PVT network; a transition layer is added after the four encoders at different scales of the PVT; multi-level gate units are inserted between the transition layer and the decoder layer; and a pyramid pooling module is introduced at the top of the encoder to extract high-level semantic information. This high-level semantic information propagates from top to bottom; the feature aggregation decoder continuously fuses high-level semantic information, effective encoder information, and decoder features at different scales through element-level addition. This invention can extract the most interesting target content from any given image, focusing more on salient regions and suppressing background interference.

Description

Technical Field

[0001] This invention relates to a salient target detection method, and more particularly to a salient target detection method based on a PVT-gated network, belonging to the field of computer vision. Background Technology

[0002] In recent years, with the rapid development of the new media industry, digital information such as short videos and image-based tweets has become integrated into daily life. Tens of thousands of images and videos are uploaded to the internet every day, greatly enriching people's lives, work, and entertainment. The daily upload volume of digital images to WeChat alone reaches a staggering 1 billion. This demonstrates that quickly selecting valuable content from this massive amount of digital images is difficult to achieve solely through human senses, posing a significant challenge to research in computer vision.

[0003] Digital image information can intuitively reflect the content people want to express and is widely used in the transmission of network resources. However, the large amount of data is difficult for people to effectively identify, and even with the assistance of computer resources, it still creates a heavy burden. Therefore, we need to extract important content from digital image information to integrate it with limited computing resources and reduce the burden on people to collect useful information. On the other hand, in the human visual system, people can focus their gaze on important areas of an image, such as faces in selfies on WeChat Moments or ingredients in food photos, without paying attention to useless information in the background. This is crucial for rapid image processing. Therefore, how to apply this ability to focus on key areas of an image to computers to help people complete more complex image processing tasks is a key focus of current computer vision research.

[0004] In recent years, U-shaped structures have attracted considerable attention due to their ability to construct rich feature maps and achieve good performance by building multi-level top-down paths. Currently, many salient object detection networks employ a U-shaped multi-scale hierarchical encoder-decoder structure as their basic network architecture. However, these methods directly use cross-layer connections, directly connecting encoder features to the decoder, lacking interference control. This approach introduces misleading contextual information from the encoder into the decoder, preventing truly useful features from being fully utilized. Furthermore, existing methods use deep learning-based fully convolutional neural networks (FCNs), mostly employing pre-trained image classification models such as VGG and ResNet as encoders, focusing on designing effective decoders by aggregating multi-level features. However, the model structure of CNNs is characterized by modeling local information convergence, making it difficult to model long-term patterns. Accurately and completely extracting salient objects from complex scenes remains a significant challenge. The Pyramid Vision Transformer (PVT) opens a new avenue for the complete extraction of salient objects. Summary of the Invention

[0005] The purpose of this invention is to provide a salient target detection method based on PVT-gated networks.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is: a salient target detection method based on a PVT-gated network, comprising the following steps:

[0007] Step 1: Image preprocessing: Resize the input image to a tensor X of a preset size;

[0008] Step 2: Establish the PVT gated network: The PVT gated network includes the first to fourth feature processing units and the pyramid pooling module PPM; the first to third feature processing units have the same structure; the first feature processing unit includes the feature encoder PVTE1, the transition layer T1, the gate unit G1, and the decoder feature generator FAD1; the fourth feature processing unit includes the feature encoder PVTE4, the transition layer T4, the gate unit G4, and the decoder feature generator FAD4; the tensor X is processed sequentially by the feature encoders PVTE1-PVTE4 to obtain the first to fourth feature tensors;

[0009] The fourth feature tensor is processed by the pyramid pooling module PPM to obtain the fourth high-level semantic information; the fourth feature tensor is processed by the transition layer T4 and concatenated with the fourth feature tensor before being input into the gate unit G4, and then multiplied with the output of the transition layer T4 to obtain the fourth encoder effective information; the fourth encoder effective information is added to the fourth high-level semantic information and then input into the decoder feature unit FAD4; the decoder feature unit FAD4 outputs the fourth decoded feature vector.

[0010] The first feature tensor is concatenated with the second decoded feature vector output by the second feature processing unit FAD2 and then input into gate unit G1. The first feature tensor is processed by the transition layer T1 and multiplied by the output of gate unit G1 to obtain the first encoder effective information. The first encoder effective information, the fourth high-level semantic information, and the second decoded feature vector are added and then input into the decoder feature generator FAD1. The first decoded feature vector is output as a saliency map.

[0011] Step 3: Detect the saliency map: Input tensor, and obtain the saliency map after processing by PVT gated network.

[0012] Furthermore, the tensor size is 384×384×3, and the sizes of the first to fourth feature tensors are 96×96×64, 48×48×128, 24×24×320, and 12×12×512, respectively.

[0013] Furthermore, the feature encoder PVTE i and decoder features FAD i+1 The process involves ensemble processing, followed by convolution, activation, and pooling operations to calculate the gate value.

[0014]

[0015] In the formula, Cat(·) is the concatenation operation between channel axes, Conv(·) is the convolution operation, S(·) is the element-wise sigmoid function, and P(·) is the global average pooling.

[0016] A threshold is applied to weight the transition layer features T1-T4, which are generated by reducing the dimensionality of the feature encoders PVTE1-PVTE4 using the generated 3×3 convolutions.

[0017] Furthermore, the pyramid pooling module (PPM) performs adaptive average pooling operations on the fourth feature tensor at four different scales, resulting in four feature maps of different sizes: 1×1, 2×2, 3×3, and 6×6. A 1×1 convolution is used to reduce the corresponding level channels to 1 / 4 of their original size. The size before pooling is obtained through bilinear interpolation, and then concatenated with the fourth feature tensor along the channel dimension.

[0018] Furthermore, the output of the decoder featurer is:

[0019]

[0020] The feature aggregation decoder FAD1 is a saliency map of the same size as the input image;

[0021] Feature Encoder PVTE i Feature Input Transition Layer T i In the module; transition layer Ti Reduce the number of channels and feed them to the gate unit G i The output of the pyramid pooling module PPM is reduced in dimensionality and then upsampled using bilinear interpolation to make it consistent with T. i The output dimensions and scales are the same; the encoder's effective information, the pyramid pooling module PPM output, and the feature aggregation decoder FAD are all the same. i+1 The outputs of the fused function are then fed into the feature aggregation decoder FAD. i middle.

[0022] Furthermore, the Feature Aggregation Decoder (FAD) first performs average pooling on the input feature map using three downsampling methods {2, 4, 8}; then it uses bilinear interpolation to upsample the input feature map to obtain the original size output feature map.

[0023] By adopting the above technical solution, the present invention achieves the following technical effects:

[0024] This invention introduces a pyramid visual transformer to powerfully model global dependencies and obtain more robust and powerful features. By introducing a gating module between the encoder and decoder, information filtering is performed, allowing more effective contextual information from the encoder to be passed to the decoder, making it focus more on salient regions and suppressing background interference. A pyramid pooling module (PPM) is introduced at the top of the encoder to expand the receptive field and collect high-level semantic information. High-level semantic information is propagated to each level of pyramid features via a top-down progressive path to compensate for the signal dilution that occurs in U-shaped networks from top to bottom. Furthermore, by introducing a feature aggregation decoder (FAD) after continuously fusing various features through element-level addition, a better balance is struck between fusing high-level semantic information, effective encoder information, and decoder features at different scales in the top-down path. Attached Figure Description

[0025] Figure 1 This is a framework diagram of the present invention.

[0026] Figure 2 This is a structural diagram of the gate unit module of the present invention.

[0027] Figure 3 This is a structural diagram of the Feature Aggregation Decoder (FAD) of the present invention.

[0028] Figure 4 This is the input image of Embodiment 1 of the present invention.

[0029] Figure 5 This is a saliency graph detected in Embodiment 1 of the present invention. Detailed Implementation

[0030] The following examples are used to illustrate the present invention.

[0031] Example 1

[0032] Reference Figure 1 A salient target detection method based on PVT-gated networks includes the following steps:

[0033] Step 1: Image preprocessing: Adjust the size of the input image to a tensor X of a preset size. In this embodiment, the tensor size is 384×384×3.

[0034] Step 2: Establish the PVT gated network: The PVT gated network includes the first to fourth feature processing units and the pyramid pooling module PPM; the first to third feature processing units have the same structure; the first feature processing unit includes the feature encoder PVTE1, the transition layer T1, the gate unit G1, and the decoder feature generator FAD1; the fourth feature processing unit includes the feature encoder PVTE4, the transition layer T4, the gate unit G4, and the decoder feature generator FAD4; the tensor X is processed sequentially by the feature encoders PVTE1-PVTE4 to obtain the first to fourth feature tensors;

[0035] The fourth feature tensor is processed by the pyramid pooling module PPM to obtain the fourth high-level semantic information; the fourth feature tensor is processed by the transition layer T4 and concatenated with the fourth feature tensor before being input into the gate unit G4, and then multiplied with the output of the transition layer T4 to obtain the fourth encoder effective information; the fourth encoder effective information is added to the fourth high-level semantic information and then input into the decoder feature unit FAD4; the decoder feature unit FAD4 outputs the fourth decoded feature vector.

[0036] The first feature tensor is concatenated with the second decoded feature vector output by the second feature processing unit FAD2 and then input into gate unit G1. The first feature tensor is processed by the transition layer T1 and multiplied by the output of gate unit G1 to obtain the first encoder effective information. The first encoder effective information, the fourth high-level semantic information, and the second decoded feature vector are added and then input into the decoder feature generator FAD1. The first decoded feature vector is output as a saliency map.

[0037] Step 3: Detect the saliency map: Input tensor, and obtain the saliency map after processing by PVT gated network.

[0038] In this embodiment, the dimensions of the first to fourth feature tensors are 96×96×64, 48×48×128, 24×24×320, and 12×12×512, respectively.

[0039] Feature Encoder PVTE i and decoder features FAD i+1 The process involves ensemble processing, followed by convolution, activation, and pooling operations to calculate the gate value.

[0040]

[0041] In the formula, Cat(·) is the concatenation operation between channel axes, Conv(·) is the convolution operation, S(·) is the element-wise sigmoid function, and P(·) is the global average pooling.

[0042] A gating system is applied to weight the transition layer features T1-T4, which are generated by dimensionality reduction of the feature encoders PVTE1-PVTE4 using generated 3×3 convolutions. Through multi-level gate units, we can suppress and balance information flowing from different encoder blocks to the decoder. Multi-level gate units can significantly suppress interference in each encoder block and enhance the contrast between salient and insalient regions.

[0043] A pyramid pooling module (PPM) is introduced at the top layer of the encoder. The high-level semantic features learned in the feature encoder PVTE4 are input into the pyramid pooling module PPM. Multi-scale pooling features are obtained through different pooling operations, which further expands the receptive field, collects global contextual information, and more accurately captures the exact location of salient objects.

[0044] The pyramid pooling module (PPM) performs adaptive average pooling operations on the fourth feature tensor at four different scales, resulting in four feature maps of different sizes: 1×1, 2×2, 3×3, and 6×6. A 1×1 convolution is used to reduce the corresponding level channels to one-quarter of their original size. Bilinear interpolation is used to obtain the unpooled size, and then these are concatenated along the channel dimension to include the fourth feature tensor. The final output is a composite feature map that integrates multiple scales, thus achieving a balance between global semantic information and local detail information. Element-wise addition is used to continuously fuse features from different levels of the transition layers T1-T4.

[0045] PVTE feature encoder i The obtained encoded features are input into the transition layer T i In the module; to reduce the number of parameters, the transition layer T i Reduce the number of channels and feed them to the gate unit G. i The PPM features are dimensionality reduced and upsampled using bilinear interpolation to match the T features. i The dimensions and scales are the same; the effective information of the encoder, the output of the pyramid pooling module PPM, and the output of the pyramid pooling module FAD are combined through element-wise additive convolutional layers. i+1 The outputs of the fused function are then fed into the feature aggregation decoder FAD. i The output process of each decoder layer can be described as follows:

[0046]

[0047] The feature aggregation decoder FAD1 is a single-channel feature map of the same size as the input image;

[0048] To prevent high-level semantic information from being diluted in a top-down path, we directly aggregate the high-level features provided by the pyramid pooling module (PPM) into the feature maps of each feature layer, providing multi-scale information for each level of decoder.

[0049] After continuously fusing various features through element-level addition, a Feature Aggregation Decoder (FAD) is introduced to encode effective features, high-level features, and features at all levels in the top-down path. By mapping the fused features into multiple feature spaces, local context information at different scales is obtained. Then, this information is combined to ensure that feature maps at different scales can be seamlessly fused.

[0050] The processing procedure of the Feature Aggregation Decoder (FAD):

[0051] The feature map obtained by element-wise addition is used as input, and pooling techniques are used to transform the fused feature map into multiple feature spaces. The operation is to first use three downsampling average pooling methods {2, 4, 8} to obtain three feature maps of different sizes;

[0052] The feature maps of three different sizes are upsampled back to their original size to obtain the same size;

[0053] The integrated feature map is obtained by adding the four pixel branches, including the input of the Feature Aggregator Decoder (FAD).

[0054] Feed it into the first 3×3 convolutional layer; the scale dimension remains unchanged.

[0055] Then, a second 3×3 convolutional layer is used to reduce the dimensionality, enabling the feature aggregation decoder (FAD) to work. i The outputs of i=2,3,4 and the low-order features T of the transition layer i-1 The output dimension is the same, and the feature aggregation decoder FAD is... i When i=1, the dimension is directly reduced to 1.

[0056] Finally, the feature aggregation decoder FAD i For i = 2, 3, 4, upsampling matching of transition layer T is performed using bilinear interpolation. i-1 The scale. FAD i When i=1, a 4x upsampling is performed, and the scale becomes 384×384.

[0057] Example 2

[0058] The difference from Example 1 lies in the processing procedure of the Feature Aggregation Decoder (FAD):

[0059] The feature map obtained by element-wise addition is used as input, and pooling techniques are used to transform the fused feature map into multiple feature spaces. The operation is to first use three downsampling average pooling methods {4, 16, 64} to obtain three feature maps of different sizes; then upsample the three feature maps of different sizes back to their original sizes to obtain the same size.

[0060] Example 3

[0061] The difference from Examples 1 and 2 is that the Feature Aggregation Decoder (FAD) first performs average pooling on the input feature map using three downsampling methods {2, 4, 8}; then it uses a super-resolution method to upsample the input feature map to obtain the original size output feature map. The feature map obtained by the super-resolution method has clearer edges.

Claims

1. A salient target detection method based on a PVT-gated network, characterized in that: Includes the following steps: Step 1: Image preprocessing: Resize the input image to a tensor X of a preset size; Step 2: Establish the PVT gated network: The PVT gated network includes the first to fourth feature processing units and the pyramid pooling module PPM; the first to third feature processing units have the same structure; the first feature processing unit includes the feature encoder PVTE1, the transition layer T1, the gate unit G1, and the decoder feature generator FAD1; the fourth feature processing unit includes the feature encoder PVTE4, the transition layer T4, the gate unit G4, and the decoder feature generator FAD4; the tensor X is processed sequentially by the feature encoders PVTE1-PVTE4 to obtain the first to fourth feature tensors; The fourth feature tensor is processed by the pyramid pooling module PPM to obtain the fourth high-level semantic information; the fourth feature tensor is processed by the transition layer T4 and concatenated with the fourth feature tensor before being input into the gate unit G4, and then multiplied with the output of the transition layer T4 to obtain the fourth encoder effective information; the fourth encoder effective information is added to the fourth high-level semantic information and then input into the decoder feature generator FAD4. The decoder feature generator FAD4 outputs the fourth decoded feature vector; The first feature tensor is concatenated with the second decoded feature vector output by the second feature processing unit FAD2 and then input into gate unit G1. The first feature tensor is processed by the transition layer T1 and multiplied by the output of gate unit G1 to obtain the first encoder effective information. The first encoder effective information, the fourth high-level semantic information, and the second decoded feature vector are added and then input into the decoder feature generator FAD1. The first decoded feature vector is output as a saliency map. Step 3: Detect the saliency map: Input tensor, and obtain the saliency map after processing by PVT gated network.

2. The salient target detection method based on PVT gating networks according to claim 1, characterized in that: The input tensor size is 384×384×3, and the sizes of the first to fourth feature tensors are 96×96×64, 48×48×128, 24×24×320, and 12×12×512, respectively.

3. The salient target detection method based on PVT gating networks according to claim 1, characterized in that: Feature encoder PVTE i and decoder features FAD i+1 The process involves ensemble processing, followed by convolution, activation, and pooling operations to calculate the gate value. In the formula, Cat(·) is the concatenation operation between channel axes, Conv(·) is the convolution operation, S(·) is the element-wise sigmoid function, and P(·) is the global average pooling. A threshold is applied to weight the transition layer features T1-T4, which are generated by reducing the dimensionality of the feature encoders PVTE1-PVTE4 using the generated 3×3 convolutions.

4. The salient target detection method based on PVT gating networks according to claim 1, characterized in that: The pyramid pooling module (PPM) performs adaptive average pooling operations on the fourth feature tensor at four different scales, resulting in four feature maps of different sizes: 1×1, 2×2, 3×3, and 6×6. A 1×1 convolution is used to reduce the corresponding level channels to 1 / 4 of their original size. The size before pooling is obtained through bilinear interpolation, and then concatenated with the fourth feature tensor along the channel dimension.

5. The salient target detection method based on PVT gating networks according to claim 1, characterized in that: The output of the decoder featurer is: The feature aggregation decoder FAD1 is a saliency map of the same size as the input image; Feature Encoder PVTE i Feature Input Transition Layer T i In the module; transition layer T i Reduce the number of channels and feed them to the gate unit G i ; The output of the pyramid pooling module PPM is reduced in dimensionality and then upsampled using bilinear interpolation to make it consistent with T. i The output dimensions and scales are the same; the encoder's effective information, the pyramid pooling module PPM output, and the feature aggregation decoder FAD are all the same. i+1 The outputs of the fused function are then fed into the feature aggregation decoder FAD. i middle.

6. The salient target detection method based on PVT gating networks according to claim 5, characterized in that: The Feature Aggregation Decoder (FAD) first performs average pooling on the input feature map using three downsampling methods: {2, 4, 8}; then it upsamples the input feature map using bilinear interpolation to obtain the output feature map of the original size.

7. The salient target detection method based on PVT gating networks according to claim 5, characterized in that: The Feature Aggregation Decoder (FAD) first performs average pooling on the input feature map using three downsampling methods: {2, 4, 8}; then it uses a super-resolution method to upsample the input feature map to obtain the original output feature map.

8. The salient target detection method based on PVT gating networks according to claim 5, characterized in that: The Feature Aggregation Decoder (FAD) first uses three downsampling average pooling methods ({4, 16, 64}) on the input feature map to obtain three feature maps of different sizes; then it upsamples the three feature maps of different sizes back to their original sizes to obtain the same size.

Images