Attack and defense methods of image dehazing model based on gradient information

Abstract

This invention discloses a gradient-information-based image defogging model attack and defense method. The method first trains a deep learning-based defogging model, then calculates perturbation information through a backpropagation algorithm. The perturbation information is added to a foggy image to obtain adversarial samples, which degrades the image quality of the defogging model output. Three attack algorithms for the defogging model are designed: an attack based on the original defogging image, a mask-based attack, and an attack based on a true clear image. Corresponding defense algorithms are designed for the attack algorithms, and adversarial training is used to conduct defense training on the defogging model, thereby increasing its security. During the defense training process, adversarial samples are dynamically generated, and the robustness of the defogging model is improved by minimizing the distance between the perturbed defogging output and the clear, fog-free image. This invention can be used for public safety, information security, image processing, and intelligent robotics tasks.

Description

Technical Field

[0001] The present invention relates to an attack and defense technology for an image defogging model based on gradient information, and belongs to the technical field of computer vision and image processing. Background Art

[0002] Image data is a crucial information carrier in modern society, and high-quality image data is crucial for information sharing and utilization. Foggy weather can negatively impact human life and production. Images captured in foggy conditions experience a certain degree of quality degradation, negatively impacting tasks such as photography, autonomous driving, and road monitoring. Image defogging technology is a key research area in computer vision and image processing. Neural network-based image defogging models have received extensive research and attention in recent years, and researchers have designed a variety of effective defogging algorithms to remove fog from images captured in foggy conditions.

[0003] Dehazing models designed using neural networks generally use a backpropagation algorithm for gradient calculation to update the parameters of the dehazing model. However, research in related fields has shown that models based on neural networks can be affected by attack algorithms, thereby reducing the performance of the model. The security issues of dehazing models based on convolutional neural networks are in urgent need of research. The security research of dehazing models is mainly divided into the attack process and the defense process. During the attack process, the performance of the dehazing model is reduced by the attack algorithm. During the defense process, the dehazing model adopts a defense strategy to reduce the negative impact of the attack algorithm. Yu et al. conducted a security study on the image dehazing task [Yu Yi, et al. Towards Robust Rain Removal Against Adversarial Attacks: A Comprehensive Benchmark Analysis and Beyond [C]. Conference on Computer Vision and Pattern Recognition. 2022: 6013-6022.], but the study did not provide attack and defense methods for the image dehazing model.

[0004] The continuous advancement of science and technology has brought even greater challenges to the security of image dehazing models. For one thing, current research lacks understanding of how image dehazing models' dehazing performance changes when under attack, making it impossible to quantitatively analyze the security of these models. Furthermore, when dehazing models are attacked, effectively defending them while maintaining their network structure, inference speed, and inference methods remains a significant research challenge. Therefore, it is necessary to develop attack and defense methods against dehazing models to ensure their security. Summary of the Invention

[0005] In order to solve the above problems, the present invention discloses an attack and defense method for an image defogging model based on gradient information, and designs attack and defense algorithms for a defogging model based on a convolutional neural network, thereby improving the security and robustness of the defogging model. The designed attack algorithm can dynamically generate disturbance information. By adding disturbance information to the foggy image, the peak signal-to-noise ratio and structural similarity of the output image of the defogging model can be reduced; the attack amplitude of the attack algorithm is controllable, and different attack effects can be obtained under different parameter configurations. The designed defense algorithm can reduce the negative impact of the attack algorithm on the defogging model and improve the robustness of the defogging model; the defense of the defogging model is only performed during the training process, which will not increase the number of parameters of the defogging model, nor will it increase the computing resources and computing time required for the actual inference process after the defogging model is deployed.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] First, the solution for training the defogging network and performing attacks and defenses is as follows:

[0008] Step S1: Train the dehazing model F based on convolutional neural network θ , where θ represents the parameters of the defogging model; the foggy image I is used as the defogging model F θ Input, get the original dehazed image J p =F θ (I).

[0009] Step S2: The goal of the attack process is to reduce the quality of the defogging image. During the attack process, the disturbance information δ is first added to the foggy image I to obtain the adversarial sample I. δ =I+δ, and then the adversarial sample is used as the input of the dehazing model, and the obtained perturbation dehazing output is

[0010] Step S3: The goal of the defense process is to fine-tune the parameters of the dehazing network through adversarial training without changing the model structure, so as to obtain a more secure and robust dehazing model, thereby improving the perturbation dehazing output. quality.

[0011] Secondly, the attack process is implemented using a gradient optimization algorithm. The implementation scheme is as follows:

[0012] Step S4: Attack target function L att is calculated as follows:

[0013]

[0014] In the above formula, X represents the reference image used by the attacker; R represents the quality metric used to measure The distance between X and the target function L att , three attack methods can be designed. The first method is to transform the original defogging image J obtained by the defogging network p As the reference image X, the second way is to use the original dehazed image J p As the reference image X and using the mask for partial occlusion attack, the third way is to use the real haze-free image J as the reference image X.

[0015] Third, during the attack process, the perturbation information δ is dynamically updated and constrained through iteration. The scheme is as follows:

[0016] Step S5: The disturbance information δ used to attack the defogging model is initialized by uniform distribution. In order to ensure that the disturbance information is not easily discovered after being added to the foggy image, the initialization value range is [-∈,∈], and the disturbance information δ needs to satisfy ‖δ‖ ∞ <ε, where ∈ represents the numerical boundary. The update process of the perturbation information δ requires calculating the gradient information of the back propagation process and updating the perturbation information δ as follows:

[0017]

[0018] In the above formula, t represents the number of iterations of the update process, α is the step size of the update process, sgn is the sign function, Represents the gradient calculation process. In addition, in order to further constrain δ, it is necessary to ensure that the sum of the perturbation information δ and the foggy image I is constrained to the maximum value range of the eight-bit image, that is, within the interval [0, 255].

[0019] Fourthly, three different attack target functions can be designed. The steps are as follows:

[0020] Step S6: Use the model's original dehazed image J p When X is used as the reference image, the attacker's goal is to increase the perturbation dehazing output F θ (I δ ) and the original dehazed image J p The attack target function is:

[0021] L P =R(F θ (I δ ),J p )

[0022] Step S7: The attacker can attack a part of the image. In this case, the attack mask can be used to control the area of action of the perturbation information δ and select some pixels from the image to attack. For the image data with three color channels of size H×W×C, where H, W and C represent the length, width and number of channels of the image respectively, first, the original foggy image I and the original defogging image J are used to compare the original foggy image I with the original defogging image J. p The mean u is calculated as follows:

[0023]

[0024] Then, for each pixel position (m,n) in the original foggy image I, if there is a pixel value on any channel k at this pixel position that satisfies I(m,n,k)-J p If (m,n,k)>u, the value of the mask at that position is 1, otherwise the value of the mask at that position is 0. The element-by-element multiplication operation "*" is used in the perturbation calculation process, and the mask-based attack target function can be obtained as:

[0025] L M =R(F θ (I+δ*mask),J p )

[0026] Step S8: When the real fog-free image J is used as the reference image X, the attack goal is to increase the perturbation dehazing output F θ (I δ ) and the distance between the real fog-free image J, the attack objective function is:

[0027] L G =R(F θ (I δ ),J)

[0028] During the attack, the attacker does not update the dehazing model parameters, but only needs to calculate the gradient information to maximize the objective function.

[0029] Fifth, the defense process of the dehazing model is implemented using adversarial training. Defense training does not require restarting the training from a randomly initialized dehazing model. The solution is as follows:

[0030] Step S9: When defense training starts, the defogging model is initialized by the model that has completed regular defogging training. During defense training, it is necessary to improve the disturbance defogging output. The quality of the image is divided into two defense cases. The first case is that the reference image used by the attacker is J p , using the guided network Guide the defense process to generate guided dehazed images The second case is that the reference image used by the attacker is the real fog-free image J, and there is no need to generate a guided defogging image. The three attack functions are L P 、L M With L G , where L P With L M The original dehazed image J p As the reference image, the defense method of the two is the same, both using the guided defogging image Conduct defensive training; L G Using the real fog-free image J as the reference image, there is no need to guide the defogging image As a guide for the defense training process. For two defense cases, the designed adversarial loss and as follows:

[0031]

[0032]

[0033] Step S10: Adversarial Loss and The calculation uses the adversarial sample I δ In order to ensure the defogging effect of the defogging model on the original foggy image I, the original foggy image I is used as the defogging network F θ Input and calculate the original image restoration loss R(F θ (I), J), add this loss term as an auxiliary term to the adversarial loss to obtain the defensive adversarial loss and as follows:

[0034]

[0035]

[0036] Step S11: During defensive training, defensive loss and The parameter λ in is used to balance the weight between the adversarial loss and the original image restoration loss.

[0037] The gradient-based attack and defense algorithm provided by this invention can improve the security of image defogging models and increase their reliability in practical applications. This invention can be used in image processing, information security, autonomous driving, and intelligent robotics. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is the defense training flowchart of the defogging model;

[0039] Figure 2 This is the attack defogging model flow chart. DETAILED DESCRIPTION

[0040] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are intended only to illustrate the present invention and are not intended to limit the scope of the present invention. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, and the terms "inward" and "outward" refer to directions toward or away from the geometric center of a particular component, respectively.

[0041] The dataset used in the experiment is the FoggyCity dataset [Marius Cordts, et al. The cityscapes dataset for semantic urban scene understanding. In IEEE Conference on Computer Vision and Pattern Recognition. 2016, pp: 3213–3223.], and the dehazing model used in the attack and defense process is 4KDehazing [Zhuoran Zheng, et al. Ultra-high-definition image dehazing via multi-guided bilateral learning. IEEE Conference on Computer Vision and Pattern Recognition. 2021: 16180-16189.]. The evaluation indicators are Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). The larger the PSNR and SSIM values, the better the dehazing effect of the dehazing model; the smaller the value, the better the attack effect of the attack algorithm.

[0042] First, when training, attacking, and defending against defogging networks, it is necessary to construct an image dataset containing both foggy and non-fog images. The overall attack and defense steps are as follows:

[0043] Step S1: Sample foggy images I and fog-free images J from the dataset and train a defogging model F based on a convolutional neural network. θ , the training process adopts the gradient descent algorithm, where θ represents the parameters of the defogging model; the foggy image I is used as the defogging model F θ Input, get the original dehazed image J p=F θ (I).

[0044] Step S2: Based on the trained model, create disturbance information δ and attack process. The goal of the attack process is to reduce the quality of the defogging image. During the attack process, the disturbance information δ is first added to the foggy image I to obtain the adversarial sample I. δ =I+δ, and then the adversarial sample is used as the input of the dehazing model, and the obtained perturbation dehazing output is The overall attack process is as follows Figure 2 shown.

[0045] Step S3: Create the structure of the defogging network to be defended. The network structure is consistent with the attacked network. The goal of the defense process is to fine-tune the parameters of the defogging network through adversarial training without changing the model structure, so as to obtain a more secure and robust defogging model, thereby improving the perturbation defogging output. The quality of the overall defense process is as follows Figure 1 shown.

[0046] Secondly, the attack process is implemented using a gradient optimization algorithm, with the following steps:

[0047] Step S4: Set a unified attack method and obtain adversarial samples I according to the attack method during the attack. δ , attack target function L att is calculated as follows:

[0048]

[0049] In the above formula, X represents the reference image used by the attacker; R represents the quality metric used to measure The distance from X.

[0050] Third, during the attack process, the perturbation information δ is dynamically updated and constrained through iteration. The steps are as follows:

[0051] Step S5: The perturbation information δ used to attack the defogging model is initialized by uniform distribution. The dimension of the perturbation information δ is consistent with the original foggy image I. In order to ensure that the perturbation information is not easily discovered after being added to the foggy image, the initial value range is [-∈,∈], and the perturbation information δ needs to satisfy ‖δ‖ ∞ <ε, where ∈ represents the numerical boundary. The update process of the perturbation information δ requires calculating the gradient information of the back propagation process and updating the perturbation information δ as follows:

[0052]

[0053] In the above formula, t represents the number of iterations of the update process, α is the step size of the update process, sgn is the sign function, represents the gradient calculation process. Furthermore, to further constrain δ, it is necessary to ensure that the sum of the perturbation information δ and the foggy image I is constrained to the maximum value range of the eight-bit image, that is, within the interval I + δ [0, 255]. In practical applications, the value of I is fixed, and the numerical clipping function is used to clip the value range of δ to the interval [0-I, 255-I].

[0054] Fourthly, calculate the three attack functions as follows:

[0055] Step S6: Calculate the original dehazed image J p , using the model's original dehazed image J p As the reference image X, the attacker’s goal is to increase the perturbation dehazing output F θ (I δ ) and the original dehazed image J p , calculate the corresponding attack target function:

[0056] L P =R(F θ (I δ ),J p )

[0057] Step S7: The attacker can attack a part of the image. In this case, the attack mask can be used to control the area of action of the perturbation information δ and select some pixels from the image to attack. For the image data with three color channels of size H×W×C, where H, W and C represent the length, width and number of channels of the image respectively, first, the original foggy image I and the original defogging image J are used to compare the original foggy image I with the original defogging image J. p The mean u is calculated as follows:

[0058]

[0059] Then, for each pixel position (m,n) in the original foggy image I, if there is a pixel value on any channel k at this pixel position that satisfies I(m,n,k)-J p If (m,n,k)>u, the value of the mask at that position is 1, otherwise the value of the mask at that position is 0. During the perturbation calculation process, the element-by-element multiplication operation "*" is used to calculate the mask-based attack target function:

[0060] L M =R(F θ (I+δ*mask),J p )

[0061] Step S8: Sample a true fog-free image J from the dataset and use the true fog-free image J as the reference image X. The goal of the attack is to increase the perturbation dehazing output F. θ (I δ ) and the distance between the real fog-free image J, and calculate the attack target function:

[0062] L G =R(F θ (I δ ),J)

[0063] During the attack, the attacker does not update the dehazing model parameters, but only needs to calculate the gradient information to maximize the objective function.

[0064] Fifth, the defense process of the dehazing model is implemented using adversarial training. The dehazing model to be defended is initialized with pre-trained weights. The defense steps are as follows:

[0065] Step S9: When defense training starts, the defogging model is initialized by the model that has completed regular defogging training. During defense training, it is necessary to improve the disturbance defogging output. The quality of the image is divided into two defense cases. The first case is that the reference image used by the attacker is J p , using the guided network Guide the defense process to generate guided dehazed images The second case is that the reference image used by the attacker is the real fog-free image J, and there is no need to generate a guided defogging image. The three attack functions are L P 、L M With L G , where L P With L M The original dehazed image J p As the reference image, the defense method of the two is the same, both using the guided defogging image Conduct defensive training; L G Using the real fog-free image J as the reference image, there is no need to guide the defogging image As a guide for the defense training process, the adversarial loss is calculated for the two defense cases. and as follows:

[0066]

[0067]

[0068] Step S10: Adversarial Loss and The calculation uses the adversarial sample Iδ In order to ensure the defogging effect of the defogging model on the original foggy image I, the original foggy image I is used as the defogging network F θ Input and calculate the original image restoration loss R(F θ (I), J), add this loss term as an auxiliary term to the adversarial loss to calculate the defensive adversarial loss and as follows:

[0069]

[0070]

[0071] Step S11: During defensive training, defensive loss and The parameter λ in is used to balance the weight between the adversarial loss and the original image restoration loss.

[0072] After completing the above steps, the attack and defense indicators are sorted out. Table 1, Table 2 and Table 3 respectively show the attack algorithm L P 、L M With L G The impact on the dehazing model performance. The attack parameter ε is set to 2, 4, 6, and 8, respectively. As ε increases, the attack amplitude gradually increases; the update step size α is set to 2. After calculation, the original dehazing model obtains a PSNR value of 31.461 and a corresponding SSIM value of 0.974. The results in Tables 1, 2, and 3 show that as the attack amplitude increases, the PSNR and SSIM values decrease, indicating that the attack effect continues to increase, proving the effectiveness of the proposed attack algorithm.

[0073] Table 1: Attack algorithm L P Attack results obtained

[0074] Parameter ε 2 4 6 8 PSNR 15.626 13.538 12.768 12.355 SSIM 0.817 0.739 0.683 0.634

[0075] Table 2: Attack algorithm L M Attack results obtained

[0076] Parameter ε 2 4 6 8 PSNR 16.743 14.901 14.238 13.845 SSIM 0.858 0.820 0.798 0.780

[0077] Table 3: Attack algorithm L G Attack results obtained

[0078] Parameter ε 2 4 6 8 PSNR 15.273 13.142 12.567 12.250 SSIM 0.792 0.711 0.667 0.625

[0079] Tables 4 and 5 show and The defense results obtained show that after the dehazing model is trained for defense, when the attack algorithm is used to attack the dehazing model again, the decrease in the PSNR and SSIM values is significantly lower than that without defense training (i.e., Table 1, Table 2, and Table 3). The attack capability of the attack algorithm is significantly reduced, which proves the effectiveness of the proposed defense algorithm.

[0080] Table 4: Defense Algorithms Defense results obtained

[0081] Parameter ε 2 4 6 8 PSNR 24.413 23.918 23.193 22.501 SSIM 0.906 0.884 0.850 0.803

[0082] Table 5: Defense Algorithms Defense results obtained

[0083] Parameter ε 2 4 6 8 PSNR 23.390 22.227 21.168 20.256 SSIM 0.893 0.868 0.837 0.800

[0084] The technical means disclosed in the solution of the present invention are not limited to the technical means disclosed in the above-mentioned embodiment, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A gradient information-based image defogging model attack and defense method, characterized in that: The following steps are involved: Step 1: Train the dehazing model F based on convolutional neural network θ , where θ represents the parameters of the defogging model; the foggy image I is used as the defogging model F θ Input, get the original dehazed image J p =F θ (I); Step 2: The goal of the attack process is to reduce the quality of the defogging image. During the attack process, the disturbance information δ is first added to the foggy image I to obtain the adversarial sample I δ =I+δ, and then the adversarial sample is used as the input of the dehazing model, and the obtained perturbation dehazing output is In step 2: The attack process is implemented using a gradient optimization algorithm, and the attack objective function L att is calculated as follows: In the above formula, X represents the reference image used by the attacker; R represents the image quality measurement method used to measure The distance between X and the target function L att , three attack methods can be designed; the first method is to convert the original defogging image J obtained by the defogging network p As the reference image X, the second way is to use the original dehazed image J p As the reference image X and using the mask to perform partial occlusion attack, the third way is to use the real fog-free image J as the reference image X; Step 3: The goal of the defense process is to fine-tune the parameters of the dehazing network through adversarial training without changing the model structure, thereby obtaining a more secure and robust dehazing model and improving the perturbation dehazing output. quality; The defense process of the defogging model is implemented by adversarial training. The defense training does not need to start from the randomly initialized defogging model again. At the beginning of the defense training, the defogging model is initialized by the model that has completed the regular defogging training. During the defense training process, it is necessary to improve the perturbation defogging output. The quality of the defense is divided into two types; The first case is that the reference image used by the attacker is J p , using guided network Guide the defense process to generate guided dehazed images The second case is that the reference image used by the attacker is the real fog-free image J, and there is no need to generate a guided defogging image. The three attack functions are L P , L and L G , where L P With L M The original dehazed image J p As the reference image, the defense method of the two is the same, both using the guided defogging image Conduct defensive training; L G Using the real fog-free image J as the reference image, there is no need to guide the defogging image As a guide for the defense training process; for two defense cases, the designed adversarial loss and as follows: Adversarial Loss and The calculation uses the adversarial sample I δ In order to ensure the defogging effect of the defogging model on the original foggy image I, the original foggy image I is used as the defogging network F θ Input and calculate the original image restoration loss R(F θ (I), J), add this loss term as an auxiliary term to the adversarial loss to obtain the defensive adversarial loss and as follows: When conducting defensive training, defensive losses and The parameter λ in is used to balance the weight between the adversarial loss and the original image restoration loss.

2. The image defogging model attack and defense method based on gradient information according to claim 1 is characterized in that: In step 2, during the attack process, the disturbance information δ is dynamically updated and constrained in an iterative manner. The disturbance information δ used to attack the defogging model is initialized by uniform distribution. In order to ensure that the disturbance information is not easily discovered after being added to the foggy image, the initialization value range is [-∈,∈], and the disturbance information δ needs to satisfy ‖δ‖ ∞ <ε, where ∈ represents the numerical boundary; the update process of the perturbation information δ requires calculating the gradient information of the back propagation process and updating the perturbation information δ as follows: In the above formula, t represents the number of iterations of the update process, α is the step size of the update process, sgn is the sign function, represents the gradient calculation process; in addition, in order to further constrain δ, it is necessary to ensure that the sum of the disturbance information δ and the foggy image I is constrained to be within the maximum value range of the eight-bit image, that is, within the interval [0,255].

3. The gradient information-based image defogging model attack and defense method according to claim 1, characterized in that: Design three different attack target functions; First, the original dehazed image J of the model is used p When X is used as the reference image, the attacker's goal is to increase the perturbation dehazing output F θ (I δ ) and the original dehazed image J p The attack target function is: L P =R(F θ (I δ ),J p ) Second, the attacker can attack part of the image. In this case, the attack mask can be used to control the area of action of the perturbation information δ and select some pixels from the image for attack. For image data with three color channels of size H×W×C, where H, W, and C represent the length, width, and number of channels of the image, respectively, first, the original foggy image I and the original defogging image J are used to compare the original foggy image I with the original defogging image J. p The mean u is calculated as follows: Then, for each pixel position (m,n) in the original foggy image I, if there is a pixel value on any channel k at this pixel position that satisfies I(m,n,k)-J p If (m,n,k)>u, the value of the mask at that position is 1, otherwise the value of the mask at that position is 0. The element-by-element multiplication operation "*" is used in the perturbation calculation process, and the mask-based attack target function can be obtained as follows: L M =R(F θ (I+δ*mask),J p ) Third, when the real haze-free image J is used as the reference image X, the goal of the attack is to increase the perturbation dehazing output F θ (I δ ) and the distance between the real fog-free image J, the attack objective function is: L G =R(F θ (I δ ),J) During the attack, the attacker does not update the dehazing model parameters, but only needs to calculate the gradient information to maximize the objective function.

Images