Weakly supervised salient object detection method based on complementary fusion pseudo label

Abstract

The weakly supervised salient object detection method based on complementary fusion pseudo label belongs to the field of computer vision, in order to solve the problem of high difficulty of obtaining pixel-by-pixel label and saliency map prediction, the gist is to input the image into a fusion learning network for object detection, the learning network outputs the salient features of the image, the fusion learning network includes a Transformer encoder, a convolutional encoder, a channel attention CA and a decoder, and the effect is to enhance the saliency prediction map.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and relates to a weakly supervised saliency target detection method based on complementary fusion pseudo-labels. Background Technology

[0002] Saliency object detection is a crucial research area in computer vision, capable of identifying the most salient regions or objects in an image. Early salient object detection relied on low-level features such as color and background. Since the development of deep learning, convolutional neural networks (CNNs) have dominated the field of salient object detection. Traditional fully supervised CNN-based networks typically consist of an encoder that extracts features at different levels and a decoder that aggregates features from different stages. In recent years, some network models using Transformers have achieved relatively good performance under fully supervised conditions.

[0003] Convolutional operations excel at extracting local image features, but they fall short in understanding higher-level semantic information. This is because understanding higher-level semantic information requires establishing global dependencies between local features. A common approach is to stack convolutional layers to expand the receptive field, but this can lead to the loss of some feature information, introduce more parameters to the network, and consume more computational resources. While Transformer-based network models focus more on global information, they cannot focus on salient objects. Furthermore, fully supervised methods require precise pixel-by-pixel labeling of salient objects, consuming significant human and material resources. Summary of the Invention

[0004] To address the challenges of obtaining pixel-by-pixel labels and predicting saliency maps, this application provides a weakly supervised saliency target detection method based on complementary fusion pseudo-labels. This method inputs an image into a fusion learning network for saliency target detection, and the learning network outputs the saliency features of the image.

[0005] The fusion learning network according to some embodiments of this application includes a Transformer encoder, a convolutional encoder, a channel attention CA, and a decoder.

[0006] The fusion learning network according to some embodiments of this application has the following steps during the training phase:

[0007] S101. Obtain the saliency image X s pseudo-label Y′ S ;

[0008] S102. Transform the saliency image X s The feature pyramid F is obtained by inputting the Transformer encoder. CASpecifically, this includes the saliency image X output by the Transformer encoder. s Given i feature pyramids, each of the i feature pyramids is input into an i-layer channel attention CA layer to obtain the feature pyramid F. CA Feature Pyramid F CA Equation (1) represents:

[0009]

[0010] Among them, the feature pyramid i represents the number of layers in the channel attention CA. This represents the pyramid feature of the CA output of the i-th channel attention layer. It is channel attention, E T (·) represents Transformer Encoder, Θ CA The parameter Θ represents channel attention learning. T These represent the parameters of the Transformer encoder;

[0011] S103. Transform the saliency image X s The feature pyramid F is obtained by inputting a convolutional encoder. C Specifically, this includes the output feature pyramid F of the convolutional encoder. C Feature Pyramid F C Equation (2) represents:

[0012] F C =E C (X|Θ C (2)

[0013] Among them, the feature pyramid i represents the number of layers in the convolutional encoder. The input to the first layer of the convolutional encoder is the saliency image X. s The intermediate convolutional encoders have the same first and second output terminals, while the last convolutional encoder only has a second output terminal. The first output terminal of the previous convolutional encoder is connected to the input terminal of the next convolutional encoder. E represents the pyramid feature output by the second output of the i-th layer convolutional encoder. c (·) represents the CNN Encoder, Θ C This represents the parameters learned by the convolutional encoder;

[0014] S104. Move the feature pyramid F CA and Feature Pyramid F C The feature pyramid F is obtained by fusion, and specifically, the feature pyramid F is represented by equation (3):

[0015]

[0016] Where, F = {F i |F i ∈F}, where i represents the number of fusion layers, F i This represents the pyramid feature output of the fused layer i, where cat(·) represents concatenation, and α i The representative trade-off parameters are represented by the feature pyramid. The i-th layer is learned from the global average pooling layer, convolutional layer, and fully connected layer.

[0017] S105. Input the feature pyramid F into the decoder to obtain the saliency image X. s Resulting graph Specifically, the decoder has a first input, a second input, a third input, and a first output. The first output of the previous layer decoder is connected to the second input of the next layer decoder. The feature pyramid F of the i-th layer fusion... i The feature pyramid F is input into the first input terminal of each of the i corresponding decoders. The acquired pseudo-labels are input as supervision signals into the third input terminal of the decoder to supervise the decoder. The first output terminal of the fourth layer decoder is the saliency image X. s Resulting graph

[0018] S201. Transform the non-significant image X n The feature pyramid F is obtained by inputting the Transformer encoder. CA Specifically, this includes the non-saliency image X output by the Transformer encoder. n Given i feature pyramids, each of which is input into an i-level channel attention CA, we obtain feature pyramid F. CA Feature Pyramid F CA Equation (1) represents:

[0019]

[0020] Among them, the feature pyramid i represents the number of layers in the channel attention CA. This represents the pyramid feature of the CA output of the i-th channel attention layer. It is channel attention, E T (·) represents Transformer Encoder, Θ CA The parameter Θ represents channel attention learning. T These represent the parameters of the Transformer encoder;

[0021] S202. Transform the non-significant image X n The feature pyramid F is obtained by inputting a convolutional encoder.C Specifically, this includes the output feature pyramid F of the convolutional encoder. C Feature Pyramid F C Equation (2) represents:

[0022] F C =E C (X|Θ C (2)

[0023] Among them, the feature pyramid i represents the number of layers in the convolutional encoder. The input to the first layer of the convolutional encoder is a non-saliency image X. n The intermediate convolutional encoders have the same first and second output terminals, while the last convolutional encoder only has a second output terminal. The first output terminal of the previous convolutional encoder is connected to the input terminal of the next convolutional encoder. E represents the pyramid feature output by the second output of the i-th layer convolutional encoder. c (·) represents the CNN Encoder, Θ C This represents the parameters learned by the convolutional encoder;

[0024] S203. The feature pyramid F CA and Feature Pyramid F C The feature pyramid F is obtained by fusion, and specifically, the feature pyramid F is represented by equation (3):

[0025]

[0026] Where, F = {F i |F i ∈F}, where i represents the number of fusion layers, F i This represents the pyramid feature output of the fused layer i, where cat(·) represents concatenation, and α i The representative trade-off parameters are represented by the feature pyramid. The i-th layer is learned from the global average pooling layer, convolutional layer, and fully connected layer.

[0027] S204. Input the feature pyramid F into the decoder to obtain the non-significant image X. n Resulting graph Specifically, the decoder has a first input, a second input, a third input, and a first output. The first output of the previous layer decoder is connected to the second input of the next layer decoder. The feature pyramid F of the i-th layer fusion... i The feature pyramid F is input into the first input terminal of each of the i corresponding decoders. The label of the pure black image is input into the third input terminal of the decoder as a supervision signal to supervise the decoder. The first output terminal of the fourth layer decoder is the non-saliency image X.n Resulting graph

[0028] S301. Distinguish whether images in the input training dataset contain salient objects using a saliency discriminator, specifically including:

[0029] The fourth-layer fusion output is connected to the input of the saliency discriminator, and the pyramid features F of the fourth-layer fusion output are combined. 4 Input saliency discriminator, saliency image X s The resulting feature pyramid F is S, and the non-significant image X n The resulting feature pyramid F is N, and the saliency discriminator is represented by equation (5):

[0030]

[0031] in, M represents the discriminator, and Θ represents the output of the discriminator. S The parameters represent the discriminator's learning parameters, with j and k set to 4, and S... j Let N represent the j-th feature of S. k Let N represent the k-th feature, and let the discriminator output the judgment of whether the image contains a salient object.

[0032] The fusion learning network according to some embodiments of this application has the following steps for image denoising during the training phase using a contrastive learning method: including...

[0033] Paired RGB saliency images X s and the corresponding prediction significance plot For positive samples, the non-significant image X n1 and X n2 All samples are negative. Both positive and negative samples are fed into a Vgg19 encoder. The image obtains corresponding features through the Vgg19 encoder. In this process, the principle is followed that salient objects are closer to objects in the original image, while being farther away from objects in the non-salient image. The features from each layer of the Vgg19 encoder are used for computation. As shown in the following formula:

[0034]

[0035] Where φ represents smoothness L1 function, This represents a significance prediction plot, X. s Representing saliency images, and All images are non-significant, and μ is set to 10. -7 .

[0036] The pseudo-label Y′ according to some embodiments of this application S The generation method involves generating initial pseudo-labels using class-independent activation graphs, then using these pseudo-labels as supervision to train a residual network. The residual network is then used to obtain the saliency of the input image X. s pseudo-label Y′ S For the input image, multiple intermediate feature maps a = {a_{n}} are obtained from the residual network. m |a m ∈ a}, m∈{1, ..., 8}, a m The m-th feature map is obtained from the residual network, and the intermediate feature map a is fused to obtain the fused feature map G. a The salient target map G is obtained through the Grabcut method. b , will fuse feature map G a With salient target map G b Addition and fusion to generate pseudo-label Y s .

[0037] The loss function of the fusion learning network according to some embodiments of this application includes binary cross-entropy loss L. BCE Structural similarity loss L SSIM Crossover ratio loss L IoU Significance discriminant loss L S and contrast loss L C The details are as follows:

[0038] L = L BCE +L SSIM +L IoU +L S +L C

[0039] L BCE The binary cross-entropy loss is defined as:

[0040]

[0041] in Y′ s The saliency image X s The pseudo-tag, Y′ n Representing the non-saliency image X n The tag, The saliency image X s Resulting image Representing the non-saliency image X n Resulting graph;

[0042] SSIM is a measure of structural similarity, defined as:

[0043]

[0044]

[0045]

[0046] in and μ(Y′) s ,Y′ n () represent the significance prediction plot and the pseudo-label, respectively. and σ(Y′) s ,Y′ n C1 and C2 represent the standard deviations of the significance prediction plot and the pseudo-label, respectively, with C1 and C2 being 0.01 and 0.01, respectively. 2 and 0.03 2 ;

[0047] IOU loss is defined as:

[0048]

[0049] The saliency discrimination loss, mainly used to distinguish between salient and non-salient images, is defined as:

[0050] L S = -[ylog(M) + (1-y)log(1-M)]

[0051] Where y represents the significance label, y∈{0,1}, and M represents the prediction result of the significance discriminator.

[0052] The detection method according to some embodiments of this application includes:

[0053] The input image is fed into the Transformer encoder to obtain the feature pyramid F. CA Specifically, this includes the i feature pyramids of the input image output by the Transformer encoder, with each i feature pyramid input to the i-level channel attention CA, resulting in the feature pyramid F. CA Feature Pyramid F CA Equation (1) represents:

[0054]

[0055] Among them, the feature pyramid i represents the channel attention CA layer number. This indicates the characteristic pyramid at the CA output of the i-th channel. It is channel attention, E T (·) represents TransformerEncoder, Θ CA The parameter Θ represents channel attention learning. TThese represent the parameters of the Transformer encoder;

[0056] The input image is fed into a convolutional encoder to obtain the feature pyramid F. C Specifically, this includes the output feature pyramid F of the convolutional encoder. C Feature Pyramid F C Equation (2) represents:

[0057] F C =E C (X|Θ C (2)

[0058] Among them, the feature pyramid 'i' represents the number of layers in the convolutional encoder. The first layer of the convolutional encoder takes the input image as input. The middle layers of the convolutional encoder have the same first and second outputs. The last layer of the convolutional encoder only has the second output. The first output of the previous layer of the convolutional encoder is connected to the input of the next layer of the convolutional encoder. E represents the feature pyramid output by the second output of the i-th layer convolutional encoder. c (·) represents the CNN Encoder, Θ C This represents the parameters learned by the convolutional encoder;

[0059] Feature pyramid F CA and Feature Pyramid F C The feature pyramid F is obtained by fusion, and specifically, the feature pyramid F is represented by equation (3):

[0060]

[0061] Where, F = {F i |F i ∈F}, i∈{1,2,3,4}, where i represents the number of fusion layers, F i This represents the feature pyramid of the fused output of the i-th layer, cat(·) represents concatenation, and α i The representative trade-off parameters are represented by the feature pyramid. The i-th layer is learned from the global average pooling layer, convolutional layer, and fully connected layer.

[0062] The feature pyramid F is input into the decoder to obtain the result image of the input image. Specifically, the decoder has a first input, a second input, a third input, and a first output. The first output of the previous layer decoder is connected to the second input of the next layer decoder. The feature pyramid F of the i-th layer fusion... iThe feature pyramid F is input into the first input terminal of each of the i decoders, and the first output terminal of the fourth decoder is the result image of the input image. Beneficial effects: Most existing weakly supervised salient object detection methods rely on position-aware weak annotation labels or image-level category labels and derived class activation maps. Compared with class activation maps, position-aware weak annotation labels still require manual annotation, while class activation map methods, although more accurate in locating salient objects, produce more ambiguous pseudo-labels and saliency prediction maps. This invention proposes a weakly supervised salient object detection method based on complementary fusion pseudo-labels, including a weakly supervised learning network and a pseudo-label generation method. Firstly, an unsupervised class-independent activation map method is introduced in pseudo-label generation, and then combined with bounding boxes to improve the pseudo-labels. Secondly, a fusion network structure of Transformer and CNN is proposed to reduce feature redundancy during the fusion process. Thirdly, a saliency discriminator is added, and a non-salient image (background image) is added to the network, with the labels of the salient image set to 0 and 1 respectively, thereby enabling the network to better understand the difference between salient objects and the background. Fourthly, a contrastive learning branch is introduced, which enhances the saliency prediction map by combining salient features in the original image, while comparing non-saliency negative samples to maximize their distance from the background.

[0063] Experimental results on four common datasets demonstrate that the method of this invention outperforms all existing weakly supervised methods, and compared with current state-of-the-art methods, F... β This represents a 0.6% improvement, while remaining comparable to fully supervised methods. Further research on S... m E m and Quantitative comparisons were conducted, and the results showed that the present invention outperformed other weakly supervised methods on all four datasets. For DUTS-TE, the present invention's method, compared to PSOD, showed S... m Increased by 1.3%, E m An increase of 2.3%, An improvement of 1.3%. For DUT-OMRON, the method of this invention, compared to PSOD, S m Increased by 1.9%, E m An increase of 3.4%. The improvement is 2.4%. Apart from the PSOD method, the PR curve of the method of this invention is superior to that of the weakly supervised method in the comparative example, comparable to the PSOD method, and closer to the expected axis than the weakly supervised method in the comparative example. The results of this invention show a significant improvement in quality compared to other methods. Attached Figure Description

[0064] Figure 1 This is a diagram illustrating the generation of pseudo-tags.

[0065] Figure 2 It is a holistic network model.

[0066] Figure 3 This is the network diagram of the saliency discriminator.

[0067] Figure 4 This is a comparison graph of the prediction graph of this invention with prediction graphs of other weakly supervised methods.

[0068] Figure 5 It's a PR curve.

[0069] Figure 6 This is a comparison diagram of pseudo-labels generated by the method of this invention and two other methods. Detailed Implementation

[0070] Example: The weakly supervised saliency target detection method based on complementary fusion pseudo-labels of the present invention mainly includes generating pseudo-labels Y′. S Construct a fusion learning network using pseudo-labels Y′ S Train a fusion learning network and use it for target detection.

[0071] During the training phase, this invention further employs a saliency discriminator to distinguish whether images in the input training dataset contain salient objects, and a contrastive learning method for image denoising. The loss function in the training phase is also determined based on the specific fusion learning network described in this invention, combined with the saliency discriminator and contrastive learning methods.

[0072] Specifically, the pseudo-labels mentioned in this invention introduce a class-agnostic activation map (C... 2 Weak supervision from salient bounding boxes (CMM) and salient image segmentation (WAS) is used to fuse the labels of both methods, considering their complementarity, to obtain high-quality pseudo-labels. Currently, most weakly supervised salient image segmentation methods require the generation of pseudo-labels, which are then used as supervision for the next training step; therefore, the quality of pseudo-label generation is crucial. Regarding pseudo-label generation, Grabcut, as a traditional image segmentation algorithm, uses bounding boxes to segment the location of objects, while WSB, as the first method in deep learning to adopt bounding boxes, introduces them into the training process of neural networks. Although both methods can obtain high-quality pseudo-labels, the quality of the labels obtained by Grabcut and WSB drops significantly when the salient object and background are relatively similar. To address this issue and generate high-quality pseudo-labels, this invention introduces an unsupervised method to generate class-independent activation maps, in addition to weak supervision from salient bounding boxes. Specifically, this invention uses CMM to generate class-independent activation maps, and further incorporates weak supervision from salient bounding boxes. 2AM is used to obtain non-class activation maps based on unsupervised contrastive learning. Then, these activation maps are combined with the maps obtained by the bounding box method to improve the quality of pseudo-labels. Figure 1 The pseudo-tag generation method of this work is shown. Specifically, this invention is by C 2 AM generates initial pseudo-labels. Then, this invention uses these initial pseudo-labels as supervision signals to train a residual network, obtaining multiple intermediate feature maps a = {a...} from the residual network. m |a m In the subsets {a} and {m}, where a ∈ {a}, m ∈ {1, ..., 8}, m This is the m-th feature map obtained from the residual network. These intermediate feature maps are then fused to obtain the fused feature map G. a Simultaneously, the Grabcut method is applied to obtain the salient target map G. b , will fuse feature map G a With G a Addition and fusion to generate pseudo-label Y′ s Pseudo-label Y′ s The output image used to supervise the decoder in the fusion network will be described in detail in the fusion network section.

[0073] Specifically, the backbone network of this invention integrates Transformer and Convolutional Neural Network (CNN), which can better represent the global and local features of an image compared to using CNN alone to extract features. Figure 2 The overall network model proposed in this invention is illustrated. The fusion learning network takes an image as input to a Transformer Encoder and a Convolutional Neural Network (CNN) Encoder. Then, the Transformer feature pyramid is processed by channel attention CA to obtain the F-value. CA As shown in the following formula:

[0074]

[0075] in, i represents the number of layers in the channel attention CA. This represents the feature pyramid at the output of the attention CA of the i-th channel. This refers to channel attention; ET(·) represents TransformerEncoder. Θ CA The parameter Θ represents channel attention learning. T This represents the parameters of the Transformer encoder.

[0076] Meanwhile, the feature pyramid obtained through convolution is F C As shown below:

[0077] F C =E C (X|Θ C (2)

[0078] in, 'i' represents the number of layers in the convolutional encoder. The first layer of the convolutional encoder takes the input image as input. The middle layers of the convolutional encoder have the same first and second outputs. The last layer of the convolutional encoder only has the second output. The first output of the previous layer of the convolutional encoder is connected to the input of the next layer of the convolutional encoder. E represents the pyramid feature output by the second output of the i-th layer convolutional encoder. c (·) represents the CNN Encoder, Θ C This represents the parameters learned by the convolutional encoder.

[0079] F CA and F C The features are merged to obtain the final feature pyramid F = {F} i |F i ∈F}, i∈{1, 2, 3, 4}. This invention introduces a learnable tradeoff parameter, which can be expressed as:

[0080]

[0081] Among them, F i This represents the pyramid feature output of the fused layer i, where cat(·) represents concatenation, and α i This represents the trade-off parameter, which is determined by... The i-th layer is learned from the global average pooling layer, convolutional layer, and fully connected layer.

[0082] Finally, the feature pyramid F is input into the decoder to obtain the result image of the input image. Specifically, the decoder has a first input terminal, a second input terminal, a third input terminal, and a first output terminal. The first output terminal of the previous layer decoder is connected to the second input terminal of the next layer decoder. The feature pyramid F of the i-th layer is fused. i The feature pyramid F is input into the first input terminal of the corresponding decoder of each of the i decoders, and the acquired label is used as a supervision signal and input into the third input terminal of the i-layer decoder. The first output terminal of the fourth layer decoder is the result image of the input image.

[0083] It should be noted that, during the training phase, this invention sequentially uses pairs of saliency images X from the training set. s Non-saliency image X n Input the data into the fusion learning network separately, and for the saliency image X... sTraining is performed using the pseudo-labels obtained in the previous stage as supervision signals for non-saliency images X. n Training is performed using the labels of pure black images as supervisory signals.

[0084] Specifically, this invention aims to implicitly improve the Transformer encoder and convolutional encoder by introducing a saliency discriminator to learn the differences between the salient image and the background image. The input image is encoded to obtain the feature pyramid F from Equation 3. The feature pyramid F obtained from the input salient and non-salient images is divided into a salient feature pyramid S and a non-salient feature pyramid N. Figure 3 As shown, the fourth-layer fusion output is connected to the input of the saliency discriminator, and the pyramid features F of the fourth-layer fusion output are combined. 4 The input to the saliency discriminator is the deepest feature in the pyramid, with a size of [value missing]. S j and N k Let S represent the j-th feature of S and the k-th feature of N, and W and H represent the width and height of the input image, respectively. A classifier is used to distinguish whether the input image contains salient objects. The salient discriminator is shown in the following equation:

[0085]

[0086] in M represents the discriminator, and Θ represents the output of the discriminator. S The parameters representing the discriminator's learning are set to 4 for both j and k. The loss function is applied using the saliency discriminator to aid network training and analyze the presence of salient objects. Based on the scheme described in this invention, the loss function is adaptively determined according to the saliency discriminator, and the loss function is specifically described in the loss function section below.

[0087] As can be seen, the S of the present invention j This represents the saliency image X. s The pyramid feature F output from the fourth layer of fusion during processing 4 (The pyramid feature of the final layer fusion output), the N of this invention k This represents the non-saliency image X. n The pyramid feature F output from the fourth layer of fusion during processing 4 (Pyramid features of the final layer fusion output).

[0088] Specifically, in the contrastive learning branch, during salient object detection, although the resulting salient prediction map is typically a binary image, it contains representational features such as the location and edges of prominent objects. This invention can leverage this by introducing contrastive learning into the technical approach of this study to improve the prediction map. For example... Figure 1 As shown, the paired RGB images X s and the corresponding prediction significance plot For positive samples, the non-significant image X n1 and X n2 All samples are negative. By introducing a contrastive learning method for image denoising, positive and negative samples are fed into a Vgg19 encoder, and the image obtains corresponding features through the Vgg19 encoder. In this process, this invention follows the principle that salient objects will be closer to objects in the original image, while being farther away from objects in the non-salient image. This invention uses the features of each layer of the Vgg19 encoder for computation. As shown in the following formula:

[0089]

[0090] Where φ represents smoothness L1 function, This represents a significance prediction plot, X. s Representing saliency images, and All images are non-significant, and μ is set to 10. -7 To avoid the denominator being 0.

[0091] Specifically, the loss function used in this invention consists of five main parts, namely, binary cross-entropy loss L. BCE Structural similarity loss L SSIM Intersection and Union Ratio (IoU) Loss L IOU Significance discriminant loss L S and contrast loss L C The details are as follows:

[0092] L = L BCE +L SSIM +L IoU +L S +L C

[0093] L BCE The binary cross-entropy loss is defined as:

[0094]

[0095] Where Y′∈{Y′ s ,Y′ n},

[0096] SSIM is a measure of structural similarity, defined as:

[0097]

[0098]

[0099]

[0100] in and μ(Y′) s ,Y′ n () represent the significance prediction plot and the pseudo-label, respectively. and σ(Y′) s ,Y′ n C1 and C2 represent the standard deviations of the significance prediction plot and the pseudo-label, respectively, with C1 and C2 being 0.01 and 0.01, respectively. 2 and 0.03 2 To avoid the denominator being 0.

[0101] IOU loss is defined as:

[0102]

[0103] The saliency discrimination loss, mainly used to distinguish between salient and non-salient images, is defined as:

[0104] L S = -[ylog(M) + (1-y)log(1-M)]

[0105] Where y represents the significance label, y∈{0,1}, and M represents the prediction result of the significance discriminator.

[0106] First, this invention introduces an unsupervised class-independent activation map method in pseudo-label generation, and then combines it with bounding boxes to improve the pseudo-labels. Second, this invention proposes a fusion network structure of Tranformer and Convolutional Neural Network (CNN) to reduce feature redundancy during the fusion process. To enable the network to better learn the difference between salient objects and the background, this invention proposes a saliency discriminator to determine whether the pair of non-salient and salient images matches the labels 1 and 0. Furthermore, inspired by contrastive learning, this invention introduces a contrastive branch, assigning the input image as a positive sample and the non-salient image as a negative sample. Non-salient images generally include background images, textures, and other unclear objects, while salient images contain clearly identifiable salient objects.

[0107] Experimental Example: The datasets used in the training of this invention are DUTS-TR, SOC, and XPIE. DUTS, containing 10553 salient images, is used as the salient dataset. SOC and XPIE are used as non-salient datasets, containing 2K and 8K non-salient images, respectively. The salient and non-salient datasets are labeled with 1 and 0, respectively. All experiments were performed on an RTX 2080 Ti. For the pseudo-label generation network, this invention uses MoCov2 pre-trained parameters to train the DUTS dataset for 10 epochs with a learning rate of 0.001 and a batch size of 128. The backbone optimizer is Adam, with a learning rate of 1e-5 for the convolutional part and 1e-6 for the Transformer part. The learning rate is dynamically adjusted every 20 epochs, and the batch size is 2. The encoder part (ResNet+PVT) uses pre-trained weights from ImageNet for 100 epochs. This invention achieves satisfactory results on five commonly used evaluation metrics. β Mean Absolute Error (MAE) m s m and Four widely used prominent object detection datasets were evaluated: ECSSD, DUTS-TE, HKU-IS, and DUT-OMRON.

[0108] This invention compares its method with nine state-of-the-art fully supervised methods: PiCAnet, UCF, Amulet, PAGR, DGRL, PAGE, BASNet, TBIN, and CSF. Furthermore, it compares the method with eight weakly supervised methods: ASMO, WSS, MWS, WSB, SCA, MFNet, SCWS, and PSOD.

[0109] Experimental results: F β The quantitative comparison results of the MAE values ​​are shown in Table 1. It can be seen that the method of this invention outperforms all existing weakly supervised methods on the complex dataset (DUT-OMRON). Compared with state-of-the-art methods, F... β This improved the efficiency by 0.6%, while remaining comparable to the fully supervised method. The present invention further improves upon S... m E m and Quantitative comparisons were performed, as shown in Table 2. The results outperformed other weakly supervised methods on all four datasets. For DUTS-TE, the method of this invention, compared with PSOD, S m Increased by 1.3%, E m An increase of 2.3%, An improvement of 1.3%. For DUT-OMRON, the method of this invention, compared to PSOD, S mIncreased by 1.9%, E m An increase of 3.4%. An increase of 2.4%. Figure 5 The PR curves of the method of this invention and five other methods are shown. It can be seen that, except for the PSOD method, the PR curve of the method of this invention is superior to that of the weakly supervised method in the comparative example, comparable to the PSOD method, and closer to the expected axis than the weakly supervised method in the comparative example. The results of this invention show a significant improvement in quality compared to other methods.

[0110]

[0111] Table 1

[0112]

[0113] Table 2

[0114] like Figure 4 As shown, this invention compares the prediction map with those of other weakly supervised methods. The first column is the input image, and the last column is the Ground Truth. In terms of the quality of the saliency prediction map, the method of this invention significantly outperforms weakly supervised methods other than the PSOD method and is comparable to the PSOD method. For example, in the last row, the method of this invention performs better in scenes where prominent objects are difficult to distinguish. Figure 6 The pseudo-tags generated by the method of this invention were compared with those generated by two other methods. It can be seen that when the color boundaries are not obvious, the pseudo-tags generated by the method of this invention are of better quality and have a better effect.

[0115] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A weakly supervised salient object detection method based on complementary fusion pseudo-labels, characterized in that, inputting an image into a fusion learning network for salient object detection, the learning network outputting salient features of the image; the fusion learning network comprising a Transformer encoder, a convolutional encoder, a channel attention (CA), and a decoder; the fusion learning network having the following steps in a training phase: S101. Obtain a saliency image of pseudo-labels ; S102. obtaining the saliency image inputting the Transformer encoder to obtain a feature pyramid , specifically comprising a Transformer encoder outputting a saliency image of a feature pyramid, a feature pyramid is respectively inputted into a layer channel attention CA to obtain a feature pyramid , the feature pyramid is represented by formula (1): (1) wherein the feature pyramid , denotes the number of layers of the channel attention CA, denotes the pyramid feature output by the layer channel attention CA, is the channel attention, (·) denotes a Transformer Encoder, denotes the parameters of the Transformer encoder; S103. obtaining the saliency image inputting the convolutional encoder to obtain a feature pyramid , and the convolutional encoder outputs a feature pyramid , the feature pyramid is represented by formula (2): (2) wherein, , denotes the number of layers of the convolutional encoder, the input of the first layer of the convolutional encoder is the saliency image , the convolutional encoder of the intermediate layers has the same first output and a second output, the last layer of the convolutional encoder has only the second output, the first output of the last layer of the convolutional encoder is connected to the input of the next layer of the convolutional encoder, denotes the second output of the layer of the convolutional encoder, (·) denotes the CNN Encoder, represents the parameters learned by the convolutional encoder; S104. obtaining a feature pyramid and the feature pyramid fused to obtain a feature pyramid , specifically including the feature pyramid is represented by formula (3): (3) in, , Indicates the number of fusion layers. Indicates the first The pyramid feature output by layer fusion, where cat(·) represents concatenation. The representative trade-off parameters are represented by the feature pyramid. The The global average pooling layer, convolutional layer, and fully connected layer of the layer are learned; S105. inputting the feature pyramid into the decoder to obtain the saliency map of the result image , specifically comprising that the decoder has a first input end, a second input end and a third input end and a first output end, the first output end of the previous layer decoder is connected to the second input end of the next layer decoder, the feature pyramid of layer fusion is respectively inputted into the first input end of the corresponding decoder of the decoder, the feature pyramid is inputted into the decoder, the acquired pseudo label is inputted into the third input end of the decoder as a supervision signal to supervise the decoder, the first output end of the fourth layer decoder is the saliency map of the result image ; S201. Non-significant images The feature pyramid is obtained by inputting the Transformer encoder. Specifically, this includes the non-saliency image output by the Transformer encoder. of A pyramid of features Each feature pyramid is input separately. Layer channel attention CA, obtaining feature pyramid Feature Pyramid Equation (1) represents: (1) wherein the feature pyramid , denotes the number of layers of the channel attention CA, denotes the layer channel attention CA output, is the channel attention, (·) denotes a Transformer Encoder, denotes the parameters of the Transformer encoder; S202. The non-significant image is removed Input convolutional encoder to get feature pyramid , specifically including a convolutional encoder output feature pyramid , the feature pyramid Is represented by formula (2): (2) wherein, , denotes the number of layers of the convolutional encoder, the input of the first layer of the convolutional encoder is the non-saliency image , the convolutional encoder of the middle layer has the same first output and second output, the last layer of the convolutional encoder only has the second output, the first output of the last layer of the convolutional encoder is connected to the input of the next layer of the convolutional encoder, denotes the second output of the layer of the convolutional encoder, (·) denotes the CNN Encoder, represents the parameters learned by the convolutional encoder; S203. obtaining the feature pyramid and the feature pyramid fused to obtain the feature pyramid , specifically including the feature pyramid is represented by formula (3): (3) wherein, , represents the number of fused layers, represents the layer fused output pyramid feature, cat(·) represents splicing, represents a trade-off parameter, the trade-off parameter is learned by a global average pooling layer, a convolutional layer and a fully connected layer of the layer of the feature pyramid;​ S204. inputting the feature pyramid into the decoder , specifically including that the decoder has a first input end, a second input end and a third input end, and a first output end, the first output end of the previous layer decoder is connected to the second input end of the next layer decoder, , the feature pyramid of layer fusion is inputted into the corresponding decoder of the decoder respectively , the first input end of the corresponding decoder , the label of the pure black image is inputted into the third input end of the decoder as a supervision signal to supervise the decoder, the first output end of the fourth layer decoder is the result image of the non-significant image ;​​ S301. distinguishing whether an image in an input training dataset contains a salient object through a saliency discriminator, specifically comprising: The fourth layer fusion output is connected with the input of the saliency discriminator, and the pyramid feature of the fourth layer fusion output is taken as the input of the saliency discriminator The input saliency discriminator, the saliency image The obtained feature pyramid is S, the non-saliency image The obtained feature pyramid is N, and the saliency discriminator is represented by formula (5): (5) wherein, represents a discriminator, M represents an output of the discriminator, represents a parameter of the discriminator learning, j and k are set to 4, represents the jth feature of S, represents the kth feature of N, and whether the image contains a salient object is determined by the output of the discriminator.

2. The weakly supervised salient object detection method based on complementary fusion pseudo-labels according to claim 1, characterized in that, the fusion learning network having the following steps in the training phase for image denoising through a contrastive learning method, comprising Significant image of pair of RGB and corresponding result image is positive sample, non-significant image and are both negative samples, positive and negative samples are sent into a Vgg19 encoder, the corresponding features of the picture are obtained through the Vgg19 encoder, in this process, the principle of following the significant object will be closer to the object in the original image, while away from the object in the non-significant image is followed, and each layer feature of the Vgg19 encoder is used for operation, , as shown in the following formula: wherein represents function, set to 10 -7 .

3. The weakly supervised salient object detection method based on complementary fusion pseudo-labels according to claim 1 or 2, characterized in that, The pseudo label The method comprises the following steps: generating an initial pseudo label through a class-independent activation map; taking the initial pseudo label as supervision to train a residual network; and obtaining an input saliency image through the residual network The pseudo label Wherein, for an input image, a plurality of intermediate feature maps are obtained from the residual network , The mth feature map obtained by the residual network, the intermediate feature maps are fused to obtain a fused feature map The salient object map is obtained through a Grabcut method The fused feature map is added to the salient object map to generate a pseudo label .​ 4. The weakly supervised salient object detection method based on complementary fusion pseudo-labels according to claim 2, characterized in that, The loss function of the fusion learning network includes a binary cross-entropy loss , a structural similarity loss , an intersection over union loss , a saliency discriminator loss , and a contrastive loss , as follows: The binary cross-entropy loss, denoted as L, is defined as: wherein , denotes the pseudo-labels of salient images , denotes the labels of non-salient images , denotes the result maps of salient images , denotes the result maps of non-salient images ; SSIM is a measure of structural similarity, defined as: wherein and represent the saliency prediction map and the pseudo label, respectively, and represent the standard deviation of the saliency prediction map and the pseudo label, respectively, and are 0.01 2 and 0.03 2 , respectively. IOU loss, defined as: saliency discrimination loss, mainly used for distinguishing salient images and non-salient images, defined as: where y denotes the saliency label, M denotes the prediction result of the saliency discriminator.

Images