An adversarial purification method for salient object detection based on self-supervised learning

Abstract

The present invention proposes an adversarial purification method for salient target detection based on self-supervised learning, which belongs to the field of computer vision. The specific scheme is: obtain the input image of the training set, rotate it counterclockwise at four different angles; for the rotated image, obtain the rotation prediction loss and the salient target detection loss, minimize the joint training loss of the two to obtain the training model; obtain the input image of the test set, randomly rotate it by an angle; for the rotated input image, solve the method to minimize the rotation prediction loss to obtain the purified image; rotate it back to restore it, input it into the main task of salient target detection of the training model, and obtain the prediction map of salient target detection. The present invention purifies adversarial samples by adding a self-supervised task in the SOD network and then minimizing the self-supervised loss. Compared with adversarial training, this strategy greatly reduces the computational cost, and the purification method is adaptable to various test data sets.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a salient object detection adversarial purification method based on self-supervised learning. Background Art

[0002] Convolutional neural networks (CNNs) are vulnerable to a type of imperceptible perturbation, adversarial examples, which can trick the network into inferring incorrect predictions while nearly preserving visual invariance. Goodfellow IJ et al. generate adversarial examples by backpropagating gradients from the supervised loss to the input. Since salient object detection (SOD) models consist of CNN modules, they are also vulnerable to adversarial examples.

[0003] Many methods have been proposed to improve the robustness of neural networks. The most widely used defense strategies can be divided into two categories: (1) adversarial training and (2) adversarial purification. Adversarial training is a brute-force defense technique that adds adversarial samples to the training set and trains the network with perturbed and clean images. The adversarial samples are recalculated, which makes this method computationally expensive. The purpose of adversarial purification is to purify the input image before sending it to the network. Samangouei P et al. described an effective purification method based on generative adversarial networks (GANs), which exploits the powerful expressive power of GANs. However, these methods are highly dependent on the datasets they are trained on and are difficult to generalize to tasks with different types of input images. In addition, most defense techniques focus on simple classification tasks or tasks using small datasets such as MNIST or CIFAR10 & CIFAR100. Defense against big data SOD is still in its infancy, and a suitable defense strategy is urgently needed.

[0004] In summary, defense strategies against SOD networks face two major problems: the adversarial training method is computationally expensive; the GAN-based purification method is difficult to use for big data, and the GAN training efficiency is low. Summary of the Invention

[0005] In response to the above problems, the present invention uses self-supervised learning in a simple and effective way to establish a purifier for defending SOD networks, which can effectively defend against adversarial attacks.

[0006] To achieve this goal, the present invention proposes an adversarial purification method for salient object detection based on self-supervised learning. The method comprises a training part consisting of a main task of salient object detection based on a U-structure and a rotation prediction branch of a self-supervised task, and a purification part based on gradient purification input. The specific method includes:

[0007] (1) Obtain the training set input image and rotate it counterclockwise at four different angles: 0°, 90°, 180°, 270°, and assign a label;

[0008] (2) For the rotated training set input image, the self-supervised task rotation prediction branch outputs an angle prediction to obtain a rotation prediction loss, and the U-shaped structure-based salient object detection main task obtains a salient object detection loss;

[0009] (3) combining the rotation prediction loss with the salient object detection loss to obtain a joint training loss, and minimizing the joint training loss to obtain a training model;

[0010] (4) Obtain a test set input image and randomly rotate the test set input image to an angle among 0°, 90°, 180°, and 270°;

[0011] (5) inputting the rotated test set input image into the self-supervised task rotation prediction branch of the training model, solving the problem of minimizing the rotation prediction loss, and obtaining a purified image;

[0012] (6) The purified image is rotated and restored, and input into the main task of salient object detection based on the U structure of the training model to obtain a prediction map for salient object detection.

[0013] Furthermore, the U-structure-based salient object detection main task includes an encoder and a decoder; the self-supervised task rotation prediction branch includes an encoder and a classifier, and the U-structure-based salient object detection main task and the self-supervised task rotation prediction branch share an encoder.

[0014] Furthermore, the classifier of the rotation prediction branch of the self-supervised task consists of three convolutional layers, a global average pooling layer and a fully connected layer.

[0015] Furthermore, the joint training loss is:

[0016]

[0017] Where: L SOD and L RP They are the loss functions of the main task of salient target detection based on the U structure and the rotation prediction branch of the self-supervised task, respectively, and can be expressed as and α is the trade-off parameter, y a is the true value of the self-supervised task rotation prediction branch, y is the true value of the main task of salient object detection based on the U structure, θ enc represents the encoder parameters, θ decrepresents the decoder parameters, θ cls Represents the classifier parameters.

[0018] Furthermore, in step 5, the multi-step gradient descent algorithm inspired by the PGD attack is used to minimize the rotation prediction loss, specifically including:

[0019] (5.1) Add a vector with ε to the test set input image pfy The perturbation is bounded, that is, π(x′)=x′+δs.t.‖δ‖ ∞ ≤ε pfy ;

[0020] (5.2) The minimized rotation prediction loss is:

[0021]

[0022] (5.3) Calculate δ using the multi-step gradient descent method with a step size of γ:

[0023]

[0024] (5.4) Based on the obtained δ, π(x′)=x′+δ is calculated to obtain the purified image.

[0025] Furthermore, in step 6, F-measure and mean absolute error (MAE) values ​​are used as evaluation indicators of the main task of salient object detection based on the U structure of the training model to obtain a prediction map for salient object detection.

[0026] Furthermore, the F-measure is used to comprehensively evaluate the accuracy and recall:

[0027]

[0028]

[0029]

[0030] Where tp, fp, and fn represent true positive, false positive, and false negative, respectively.

[0031] Furthermore, the β 2 Set to 0.3.

[0032] Furthermore, the mean absolute error (MAE) is the average pixel-level absolute difference between the saliency map S and the ground truth G, and is defined as:

[0033]

[0034] Where W and H are the width and height of the saliency map.

[0035] Beneficial effects of the present invention:

[0036] (1) We study the perturbation effect between high-order and low-order features of the U-structured SOD network and show that high-order features are more important than low-order features for achieving robustness. Adding a self-supervision branch after the high-order features will benefit the SOD performance of our proposed purification process.

[0037] (2) A new SOD purification method is proposed, which takes advantage of the label independence of self-supervised learning. It purifies adversarial samples by adding a self-supervised task to the high-order features of the SOD network and then minimizing the self-supervised loss. A rotation prediction task is selected as the self-supervised task. During the purification process, the true label (angle) is known because the defender rotates the input. Compared with adversarial training, this training strategy greatly reduces the computational cost, and the purification method is adaptable to various test datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the process of the adversarial purification method for salient object detection based on self-supervised learning in this embodiment.

[0039] Figure 2 Schematic diagram of the overall framework of the adversarial purification method for salient object detection based on self-supervised learning in this embodiment.

[0040] Figure 3 2 is a comparison diagram of this embodiment and other methods (U-net).

[0041] Figure 4 A comparison diagram of this embodiment and other methods (U 2 -net). DETAILED DESCRIPTION

[0042] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] This embodiment provides a salient object detection adversarial purification method based on self-supervised learning, including a training part consisting of a salient object detection main task based on U structure and a self-supervised task rotation prediction branch, and a testing part based on gradient purification input. Figure 1 、 Figure 2 The specific implementation details are as follows:

[0044] S101. Obtain the training set input image and rotate it counterclockwise at four different angles: 0°, 90°, 180°, and 270°, and assign a label;

[0045] S102: for the rotated training set input image, output an angle prediction from the self-supervised task rotation prediction branch to obtain a rotation prediction loss, and obtain a salient object detection loss from the U-shaped structure-based salient object detection main task;

[0046] The main task of salient object detection based on the U structure includes an encoder and a decoder; the self-supervised task rotation prediction branch includes an encoder and a classifier. The main task of salient object detection based on the U structure and the self-supervised task rotation prediction branch share an encoder.

[0047] The classifier of the rotation prediction branch consists of three convolutional layers, followed by a h global average pooling, and finally a fully connected layer to output the angle prediction.

[0048] S103, combining the rotation prediction loss and the salient object detection loss to obtain a joint training loss, and minimizing the joint training loss to obtain a training model;

[0049] During training, the combined SOD loss and rotation prediction loss is:

[0050]

[0051] Among them L SOD and L RP They are the loss functions of the main task of salient target detection based on U structure and the rotation prediction branch of the self-supervised task, which can be expressed as and α is the trade-off parameter, y a is the true value of the self-supervised task rotation prediction branch, y is the true value of the main task of salient target detection based on the U structure, θ enc represents the encoder parameters, θ dec represents the decoder parameters, θ cls denotes the classifier parameters. This training scheme is based on the following assumptions: (1) The performance of rotation prediction is close to that of SOD because they use the same encoder. (2) Any method that can improve the performance of the rotation prediction task will also be effective for SOD. Compared with adversarial training, joint training greatly reduces the computational overhead because it does not require the generation of adversarial examples.

[0052] S104: Obtain a test set input image, and randomly rotate the test set input image by an angle selected from 0°, 90°, 180°, and 270°;

[0053] S105, inputting the rotated test set input image into the self-supervised task rotation prediction branch of the training model, solving the problem of minimizing the rotation prediction loss, and obtaining a purified image;

[0054] After joint training, the performance of the rotation prediction task is very close to that of the SOD task since they use the same encoder. RP It's L SOD The approximation of , so use L RP Replace L SOD The adversarial perturbation budget ε is used to RP As a proxy to set the appropriate ε pfy As a replacement for ε, the purification process is thus updated to a solvable minimal problem:

[0055]

[0056] The multi-step gradient descent method inspired by the PGD attack is used to solve the above equation. The goal of the PGD attack is to maximize the SOD loss L by using projected gradient ascent at the input. SOD , the purifier aims to minimize the rotation prediction loss L by using projected gradient descent on the input RP The goal is to add a pfy This is achieved by bounded perturbation, i.e., π(x′)=x′+δs.t.‖δ‖ ∞ ≤ε pfy .

[0057] We compute δ by multi-step gradient descent with a step size γ similar to PGD, so at each step t:

[0058]

[0059] According to the obtained δ, π(x′)=x′+δ is calculated to obtain the purified image.

[0060] S106 , rotating and restoring the cleaned image, and inputting the image into the main task of salient object detection based on the U-shaped structure of the training model to obtain a prediction map for salient object detection.

[0061] The present invention uses F-measure and mean absolute error (MAE) values ​​as evaluation indicators for the main task of target detection to obtain a prediction map for salient target detection.

[0062] F-measure is used to comprehensively evaluate the accuracy and recall rate:

[0063]

[0064]

[0065]

[0066] Among them, tp, fp, and fn represent true positive, false positive, and false negative, respectively. According to experience, β2 Set to 0.3.

[0067] MAE is the average pixel-level absolute difference between the saliency map S and the ground truth G, and is defined as:

[0068]

[0069] Where W and H are the width and height of the saliency map.

[0070] The model of the present invention is implemented based on PyTorch, and the graphics card configuration of the machine is GTX3090 (24GB). AdverTorch is used for attack and purification algorithms.

[0071] We conducted extensive experiments on three public salient object detection datasets: ECSSD, DUTOMRON, and HKU-IS to validate our proposed algorithm. ECSSD consists of 1,000 semantically meaningful but structurally complex images collected from the internet. DUT-OMRON contains 5,168 images containing one or two complex foreground objects. HKU-IS contains 4,447 images with multiple salient objects.

[0072] like Figure 3 As shown in the figure, for the U-net structure, it defeats all other defense methods under the FGSM attack while maintaining acceptable performance on the unperturbed data on all datasets. Under the PGD attack, the results of the proposed method on DUT-OMRON and HKU-IS are slightly worse than those of the PGD AT method, but are close enough.

[0073] like Figure 4 As shown, for U 2 -net, our method has similar performance to other methods. We defeat PGD AT under PGD attack on all datasets, which is the most challenging case for defense. This advantage may be due to the fact that U 2 The complex and deep structure of the -net network makes state-of-the-art PGD ATs more difficult to train. In contrast, our proposed method is not affected by this because it does not aim to directly enhance the model itself. Our method outperforms other methods in accuracy across all test sets. Compared to adversarial training, our method significantly reduces training time.

[0074] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A method for adversarial purification of salient object detection based on self-supervised learning, characterized by: The system consists of a training part consisting of a U-structured salient object detection main task and a self-supervised task rotation prediction branch, and a purification part based on gradient purification input; the U-structured salient object detection main task includes an encoder and a decoder; the self-supervised task rotation prediction branch includes an encoder and a classifier, and the U-structured salient object detection main task and the self-supervised task rotation prediction branch share a common encoder; Specific methods include: (1) Get the training set input image and rotate it counterclockwise at four different angles: , and assign labels; (2) For the rotated training set input image, the self-supervised task rotation prediction branch outputs an angle prediction to obtain a rotation prediction loss, and the U-structure-based salient object detection main task obtains a salient object detection loss; The joint training loss is: in: and They are the loss functions of the main task of salient target detection based on the U structure and the rotation prediction branch of the self-supervised task, respectively, and can be expressed as and ; To balance the parameters, The true value of the rotation prediction branch for the self-supervised task, is the true value of the main task of salient object detection based on the U structure, represents the parameters of the encoder, represents the parameters of the decoder, represents the classifier parameters; (3) Combining the rotation prediction loss with the salient object detection loss to obtain a joint training loss, and minimizing the joint training loss to obtain a training model; (4) Obtain the test set input image and put the test set input image into Randomly select an angle to rotate; (5) For the rotated test set input image, input it into the self-supervised task rotation prediction branch of the training model, and solve the problem of minimizing the rotation prediction loss to obtain a purified image; To minimize the rotation prediction loss, we use a multi-step gradient descent algorithm inspired by the PGD attack, which includes: (5.1) Add an input image to the test set with The disturbance bounded by ； (5.2) The minimization of the rotation prediction loss is: (5.3) Using step size The multi-step gradient descent method calculates : (5.4) According to the obtained ,calculate , obtaining the purified image; (6) The cleaned image is rotated and restored, and input into the main task of salient object detection based on the U structure of the training model to obtain a prediction map for salient object detection.

2. The method for salient object detection based on self-supervised learning according to claim 1, characterized in that: The classifier of the rotation prediction branch of the self-supervised task consists of three convolutional layers, a global average pooling layer and a fully connected layer.

3. The method for salient object detection based on self-supervised learning according to claim 1, characterized in that: In the step (6), F-measure and mean absolute error (MAE) values ​​are used as evaluation indicators of the main task of salient object detection based on the U structure of the training model to obtain a prediction map of salient object detection.

4. The method for adversarial purification of salient object detection based on self-supervised learning according to claim 3, characterized in that: The F-measure is used to comprehensively evaluate the accuracy and recall rate: in tp, fp, fn represent true positive, false positive, and false negative, respectively.

5. The method for adversarial purification of salient object detection based on self-supervised learning according to claim 4, characterized in that: described Set to 0.

3.

6. The method for adversarial purification of salient object detection based on self-supervised learning according to claim 3, characterized in that: The mean absolute error (MAE) is the average pixel-level absolute difference between the saliency map S and the ground truth G, and is defined as: in W, H are the width and height of the saliency map.

Images