Visual complexity guided diffusion model confrontation and defense method

Abstract

The invention provides a visual complexity guided diffusion model confrontation defense method, which comprises the following steps of: 1) taking an attacked video confrontation sample as input, and calculating the visual complexity of the confrontation video sample frame by frame; 2) selecting a proper diffusion purification scale according to a mapping relation between the visual complexity of the sample and the diffusion purification scale; (3) forward diffusion is carried out on the confrontation video sample, Gaussian noise with proper intensity is used for submerging confrontation disturbance according to the purification scale selected in the step (2), meanwhile, principal components of video frames are reserved as much as possible, and a diffusion video sample is obtained; 4) carrying out reverse diffusion on the diffusion video sample, and efficiently recovering the diffusion sample into a purified sample by adopting a one-time denoising strategy; and step 5) inputting the purified benign video into an intelligent video system to obtain a correct decision, and completing defense of the countermeasure attack.

Description

Technical Field

[0001] The present invention belongs to the field of visual model adversarial defense, and in particular relates to a diffusion model adversarial defense method guided by visual complexity. Background Art

[0002] Intelligent video systems are designed to automatically process and analyze video data. They integrate multiple functions, including tracking and identifying targets in videos. Therefore, they are widely used in many fields, such as video surveillance and human-computer interaction. However, studies have shown that existing intelligent video systems based on deep neural networks are very vulnerable to adversarial attacks. Once some adversarial perturbations that are difficult for the human eye to detect are added to naturally collected benign samples, those originally high-performance deep neural networks can be easily misled and make wrong predictions. Accordingly, adversarial defense aims to eliminate the negative effects of adversarial attacks, thereby maximizing the reliability and security of intelligent video systems.

[0003] Previous adversarial defense methods have the following design deficiencies: 1) Insufficient robustness. Existing methods are unable to balance the performance of the target model on benign samples and adversarial samples. In other words, adversarial defense methods cannot enhance the robustness of the model against adversarial samples without affecting the original performance of the model; 2) Insufficient versatility. Existing defense methods are usually designed for a certain adversarial attack or are highly coupled with the target model, while general defense methods can be applied to different types of adversarial attacks and target models; 3) Low computational efficiency. Since most existing defense methods are designed for image-level visual models, they do not take computational efficiency into consideration, and large-scale video data has high requirements for computational consumption and inference time of adversarial defense. Therefore, in view of the above-mentioned deficiencies in the existing technologies, a robust, highly versatile and efficient adversarial defense method is needed to address the security threats faced by intelligent video systems. Summary of the invention

[0004] Purpose of the invention: The present invention designs an adversarial defense method based on a diffusion model. Before the adversarial samples are input into the system, the diffusion model is used to purify them into benign samples. In order to solve the problem of high time consumption in the reverse diffusion process, the present invention adopts a one-time denoising strategy to ensure the purification effect while accelerating the calculation. In addition, based on the fact that the purification of complex scenes is often more difficult than that of simple scenes, the present invention designs a diffusion purification scale selection mechanism guided by visual complexity, which realizes adaptive adjustment of the purification process and achieves a balance between adversarial disturbance elimination and high-quality image reconstruction. Therefore, the present invention meets the requirements of robust, universal and efficient design of adversarial defense methods, and provides practical guarantees for the safe use of intelligent video systems.

[0005] Technical solution: In order to solve the above technical problems and achieve the above invention objectives, the present invention proposes a diffusion model countermeasure defense method guided by visual complexity, which includes the following steps:

[0006] Step 1) Take the attacked video adversarial sample as input and calculate the visual complexity of the adversarial video sample frame by frame;

[0007] Step 2) Selecting a diffusion purification scale according to the mapping relationship between the sample visual complexity and the diffusion purification scale;

[0008] Step 3) forward diffusion is performed on the adversarial video sample. According to the purification scale selected in step 2), Gaussian noise is used to submerge the adversarial disturbance while retaining the main components of the video frame to obtain a diffused video sample.

[0009] Step 4) reversely diffuse the diffused video samples and use a one-time denoising strategy to restore the diffused samples to high-quality purified samples;

[0010] Step 5) Input the purified benign video into the intelligent video system to obtain the correct decision and complete the defense against adversarial attacks.

[0011] Further, the method of step 1) is as follows:

[0012] Step 1.1) Given a naturally acquired benign video sequence X = [x 1 ,x 2 ,x 3 ,…,x N ], where N is the number of frames in the video, and the adversarial video sequence is defined as:

[0013]

[0014] Among them, IVS(·) represents the Intelligent Video System (IVS) based on deep neural network. are the parameters of the system; r and They represent the output decisions of the intelligent video system for benign video sequences and adversarial video sequences respectively. Since the difference between benign video sequences and adversarial video sequences must be imperceptible to the human eye, l is used here. p The norm constraint counteracts the dimension of the disturbance. Generally speaking, p=0,1,2,∞, and τ is the upper limit of the disturbance.

[0015] Step 1.2) Calculate the adversarial video sequence frame by frame The edge energy of is used as a measure of the visual complexity of the video frame. Given the adversarial video frame of the nth frame The frame is convolved using a Sobel kernel filter in the horizontal and vertical directions. Specifically, the Sobel operation in the horizontal direction can be expressed as:

[0016]

[0017] Correspondingly, the Sobel operation in the vertical direction is written as:

[0018]

[0019] Based on G h With G v It can be calculated The marginal density map G is:

[0020]

[0021] Adversarial video frames Visual complexity can be calculated by averaging G:

[0022]

[0023] Among them, vc represents The visual complexity of is, H and W represent the height and width of the edge density map G respectively, and i and j are the indices of the pixels in G.

[0024] Furthermore, in step 2), the optimal diffusion purification scale is selected according to the mapping relationship between the visual complexity and the diffusion model purification scale. The operation process is as follows:

[0025]

[0026] Among them, Proj(vc) is the mapping function between visual complexity and diffusion purification scale, with the visual complexity vc of the adversarial video frame as the independent variable, and the parameter controlling the purification scale of the diffusion model And it decreases monotonically with the increase of diffusion time step T) as the dependent variable to obtain the optimal purification scale Specifically, Proj(vc) is designed as a piecewise function: vc min With vc max is a manually set visual complexity threshold. If the adversarial video frame The visual complexity vc is less than the minimum visual complexity vc min Or greater than the maximum visual complexity vc max When the maximum purification scale is used directly or minimum purification scale As the optimal diffusion purification scale; and when vc min ≤vc≤vcmax When the present invention uses a linear function as vc and The mapping relationship is:

[0027] J(vc;w,b)=w·vc+b

[0028] in:

[0029]

[0030] Further, according to the optimal purification scale obtained in step 2)

[0031] Step 3) Adversarial video frames Forward diffusion is performed, and the process can be formally expressed as:

[0032]

[0033] Among them, ∈ is Gaussian noise, is the diffusion sample. From the above formula, we can see that the granularity of the forward diffusion is given by Considering the adversarial video frame is composed of benign video frames x n It is synthesized with the adversarial perturbation δ, that is, Therefore, the above formula can be rewritten as:

[0034]

[0035] Since the adversarial perturbation is imperceptible to the human eye, and As the diffusion time step T increases, it decreases monotonically, and the dimension of δ is much smaller than the added Gaussian noise. Therefore, if the time step T * The choice of is appropriate to diffuse adversarial information in the sample Gaussian noise While being submerged, The principal component information in benign image samples is preserved.

[0036] Further, step 4) is performed on the diffusion sample obtained in step 3) Backward diffusion is performed to generate high-quality purified samples. Considering the high computational requirements of video tasks, this paper adopts a one-time denoising strategy, directly from the final diffusion time step T * Directly predict the fully denoised image

[0037]

[0038] Among them, ∈ θ(·) is a deep generative network model parameterized by θ. The present invention uses the pre-trained DDIM diffusion model to predict the Gaussian noise ∈ added in step 3) through only one forward propagation. Compared with the T set by the standard diffusion model * =1000, the present invention will T * The value range of is limited to [25,60], that is, T min =25, T max =60, in reducing ∈ θ (·) While improving the difficulty of ∈ prediction, the computational efficiency of the diffusion model is greatly improved, and the one-time denoising strategy can achieve high-quality and efficient disturbance elimination.

[0039] In step 5), the purified video sequence Input to the intelligent video system:

[0040]

[0041] Make That is, the intelligent video system makes correct decisions, thereby completing the defense against adversarial attacks.

[0042] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0043] (1) The visual complexity-guided diffusion purification scale selection mechanism can enhance the quality of purified video sequences and thus enhance the robustness of intelligent video systems.

[0044] (2) The proposed defense method is independent of the intelligent video system of the protected object and therefore has good versatility.

[0045] (3) Using a one-time denoising strategy can significantly reduce the computational and time consumption required for defense. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0047] like Figure 1 As shown, the present invention proposes a diffusion model countermeasure defense method guided by visual complexity, which includes the following steps:

[0048] Step 1) Take the attacked video adversarial sample as input and calculate the visual complexity of the adversarial video sample frame by frame;

[0049] Step 2) Selecting a diffusion purification scale according to the mapping relationship between the sample visual complexity and the diffusion purification scale;

[0050] Step 3) forward diffusion is performed on the adversarial video sample. According to the purification scale selected in step 2), Gaussian noise is used to submerge the adversarial disturbance while retaining the main components of the video frame to obtain a diffused video sample.

[0051] Step 4) reversely diffuse the diffused video samples and use a one-time denoising strategy to restore the diffused samples to high-quality purified samples;

[0052] Step 5) Input the purified benign video into the intelligent video system to obtain the correct decision and complete the defense against adversarial attacks.

[0053] Further, the method of step 1) is as follows:

[0054] Step 1.1) Given a naturally acquired benign video sequence X = [x 1 ,x 2 ,x 3 ,…,x N ], where N is the number of frames in the video, and the adversarial video sequence is defined as:

[0055]

[0056] Among them, IVS(·) represents the Intelligent Video System (IVS) based on deep neural network. are the parameters of the system; r and They represent the output decisions of the intelligent video system for benign video sequences and adversarial video sequences respectively. Since the difference between benign video sequences and adversarial video sequences must be imperceptible to the human eye, l is used here. p The norm constraint counteracts the dimension of the disturbance. Generally speaking, p=0,1,2,∞, and τ is the upper limit of the disturbance.

[0057] Step 1.2) Calculate the adversarial video sequence frame by frame The edge energy of is used as a measure of the visual complexity of the video frame. Given the adversarial video frame of the nth frame The frame is convolved using a Sobel kernel filter in the horizontal and vertical directions. Specifically, the Sobel operation in the horizontal direction can be expressed as:

[0058]

[0059] Correspondingly, the Sobel operation in the vertical direction is written as:

[0060]

[0061] Based on G h With Gv It can be calculated The marginal density map G is:

[0062]

[0063] Adversarial video frames Visual complexity can be calculated by averaging G:

[0064]

[0065] Among them, vc represents , H and W represent the height and width of the edge density map G, respectively, and i and j are the indices of the pixels in G.

[0066] Furthermore, in step 2), the optimal diffusion purification scale is selected according to the mapping relationship between the visual complexity and the diffusion model purification scale. The operation process is as follows:

[0067]

[0068] Among them, Proj(vc) is the mapping function between visual complexity and diffusion purification scale, with the visual complexity vc of the adversarial video frame as the independent variable, and the parameter controlling the purification scale of the diffusion model And it decreases monotonically with the increase of diffusion time step T) as the dependent variable to obtain the optimal purification scale Specifically, Proj(vc) is designed as a piecewise function: vc min With vc max is a manually set visual complexity threshold. If the adversarial video frame The visual complexity vc is less than the minimum visual complexity vc min Or greater than the maximum visual complexity vc max When the maximum purification scale is used directly or minimum purification scale As the optimal diffusion purification scale; and when vc min ≤vc≤vc max When the present invention uses a linear function as vc and The mapping relationship is:

[0069] J(vc;w,b)=w·vc+b

[0070] in:

[0071]

[0072] Further, according to the optimal purification scale obtained in step 2)

[0073] Step 3) Adversarial video frames Forward diffusion is performed, and the process can be formally expressed as:

[0074]

[0075] Where, ∈ is Gaussian noise is the diffusion sample. From the above formula, we can see that the granularity of the forward diffusion is given by Considering the adversarial video frame is composed of benign video frames x n It is synthesized with the adversarial perturbation δ, that is, Therefore, the above formula can be rewritten as:

[0076]

[0077] Since the adversarial perturbation is imperceptible to the human eye, and As the diffusion time step T increases, it decreases monotonically, and the dimension of δ is much smaller than the added Gaussian noise. Therefore, if the time step T * The choice of is appropriate to diffuse adversarial information in the sample Gaussian noise While being submerged, The principal component information in benign image samples is preserved.

[0078] Further, step 4) is performed on the diffusion sample obtained in step 3) Backward diffusion is performed to generate high-quality purified samples. Considering the high computational requirements of video tasks, this paper adopts a one-time denoising strategy, directly from the final diffusion time step T * Directly predict the fully denoised image

[0079]

[0080] Among them, ∈ θ (·) is a deep generative network model parameterized by θ. The present invention uses the pre-trained DDIM diffusion model to predict the Gaussian noise ∈ added in step 3) through only one forward propagation. Compared with the T set by the standard diffusion model * =1000, the present invention will T * The value range of is limited to [25,60] (i.e., T min =25, T max =60), in reducing ∈ θ (·) While reducing the difficulty of ∈ prediction, the computational efficiency of the diffusion model is greatly improved. Therefore, the one-time denoising strategy can achieve high-quality and efficient disturbance elimination.

[0081] In step 5), the purified video sequence Input to the intelligent video system:

[0082]

[0083] Make That is, the intelligent video system makes correct decisions, thereby completing the defense against adversarial attacks.

[0084] The present invention defends the target tracking model SiamRPN++ and the behavior recognition model Non-Local on the public visual target tracking dataset OTB100 and the behavior recognition dataset UCF101. Table 1 and Table 2 respectively show the defense effect of the present invention on the above intelligent video system. In addition, the present invention also has advantages in computing efficiency and time consumption, with the number of floating point operations per second (FLOPs) of 2.25×10 3 G, and the number of frames per second (FPS) is 14.03. Experimental results in real scenarios show that the present invention can provide effective defense for intelligent video systems.

[0085] Table 1 The defensive effect of the present invention on the target tracking model SiamRPN++ on the OTB100 dataset

[0086] Raw performance Post-attack performance Post-defense performance Tracking success rate 69.6％ 32.4％ 61.1％ Tracking accuracy 91.4％ 47.1％ 83.3％

[0087] Table 2 The defense effect of the present invention on the behavior recognition model Non-local on the UCF101 dataset

[0088] Raw performance Post-attack performance Post-defense performance Recognition accuracy 94.6％ 27.3％ 80.5％

Claims

1. A diffusion model adversarial defense method guided by visual complexity, characterized in that: The method comprises the following steps: Step 1) Take the attacked video adversarial sample as input and calculate the visual complexity of the adversarial video sample frame by frame; Step 2) Selecting a diffusion purification scale according to the mapping relationship between the sample visual complexity and the diffusion purification scale; Step 3) forward diffusion is performed on the adversarial video sample. According to the purification scale selected in step 2), a preset Gaussian noise is used to submerge the adversarial disturbance while retaining the main components of the video frame to obtain a diffused video sample; Step 4) reversely diffuse the diffused video samples and use a one-time denoising strategy to restore the diffused samples to purified samples; Step 5) Input the purified benign video into the intelligent video system to obtain the correct decision and complete the defense against adversarial attacks.

2. The method for countermeasures against diffusion models guided by visual complexity according to claim 1, characterized in that: The method of step 1) is as follows: Step 1.1) Set the naturally acquired benign video sequence X = [x 1 , x 2 , x 3 , ..., x N ], where N is the number of frames in the video, and the adversarial video sequence is defined as: Among them, IVS(·) represents the intelligent video system based on deep neural network, is the parameter of the system, r and They represent the output decisions of the intelligent video system for benign video sequences and adversarial video sequences, respectively, and τ is the upper limit of disturbance; Step 1.2) Calculate the adversarial video sequence frame by frame The edge energy of is used as a measure of the visual complexity of the video frame. Given the adversarial video frame of the nth frame The frame is convolved with a filter using the Sobel kernel in the horizontal and vertical directions respectively. The Sobel operation in the horizontal direction is expressed as: The vertical Sobel operation is written as: Based on G h With G v Calculate The marginal density map G is: Adversarial video frames Visual complexity is calculated by averaging G: Among them, vc represents H and W represent the height and width of the edge density map G, respectively, and i and j are the indices of the pixels in G.

3. The visual complexity guided diffusion model countermeasure defense method according to claim 2, characterized in that: In step 2), the optimal diffusion purification scale is selected based on the mapping relationship between the visual complexity and the diffusion model purification scale. The operation process is as follows: Among them, Proj(vc) is the mapping function between visual complexity and diffusion purification scale, with the visual complexity vc of the adversarial video frame as the independent variable, and the parameter controlling the purification scale of the diffusion model is the dependent variable, which decreases monotonically with the increase of diffusion time step T. Among them, vc min With vc max is the visual complexity threshold set. If the adversarial video frame The visual complexity vc is less than the minimum visual complexity vc min Or greater than the maximum visual complexity vc max When the maximum purification scale is used directly or minimum purification scale As the optimal diffusion purification scale; when vc min ≤vc≤vc max When a linear function is used as vc and The mapping relationship is: J(vc;w,b)=w·vc+b in:

4. The visual complexity guided diffusion model countermeasure defense method according to claim 3, characterized in that: Step 3) The adversarial video frame Forward diffusion is performed, and the process is formally expressed as: Among them, ∈ is Gaussian noise, is the diffusion sample, I is the variance of the standard Gaussian noise, considering the adversarial video frame is composed of benign video frames x n It is synthesized with the adversarial disturbance δ, that is, Rewrite the above formula as: According to the optimal purification scale Diffusion of adversarial information in samples Gaussian noise While being submerged, The principal components of the video frames are retained to obtain diffuse video samples.

5. The visual complexity guided diffusion model countermeasure defense method according to claim 4, characterized in that: In step 4), the diffused video samples are reversely diffused, and a one-time denoising strategy is used to restore the diffused samples to purified samples, as follows: Among them, ∈ θ (·) is a deep generative network model parameterized by θ, which uses the pre-trained DDIM diffusion model and simultaneously removes the adversarial perturbation δ and Gaussian noise ∈ through only one forward propagation.

6. The visual complexity guided diffusion model countermeasure defense method according to claim 5, characterized in that: In step 5), the purified video sequence Input to the intelligent video system: Make That is, the intelligent video system makes correct decisions and completes the defense against adversarial attacks.

Images