Adversarial patch generation method, electronic device and computer readable storage medium

Abstract

This application discloses an adversarial patch generation method, an electronic device, and a computer-readable storage medium. The method includes: acquiring a noisy image of the patch region of an original image and a target image; obtaining a first gradient based on the noisy image and the target image; obtaining an adversarial image based on the noisy image and obtaining a second gradient based on the adversarial image; obtaining the adversarial image based on the noisy image and obtaining the second gradient based on the adversarial image; and obtaining an adversarial patch image using a diffusion model based on the first gradient, the second gradient, and the noisy image. Through the above method, this application utilizes the first and second gradients to guide a diffusion model to obtain an adversarial patch image, resulting in a higher image quality that closely resembles real-world images and contains target information, making it highly aggressive against recognition models in subsequent adversarial face recognition.

Description

Technical Field

[0001] This application relates to the field of counter-attack technology, specifically to a method for generating counter-patches, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In the past, Generative Adversarial Networks (GANs) have dominated the field of image generation, generating images with high realism, comparable to real-world images. Their structure mainly consists of a generator and a discriminator. The generator produces images, and the discriminator monitors the similarity between the generated and original images. However, this type of method has significant drawbacks, such as low diversity of generated adversarial images, limited to generating only similar samples already in the training set. Furthermore, training a GAN is a complex and difficult process; the balance between the adversarial forces between the generator and discriminator is difficult to measure well, requiring continuous adjustments and iterations during training to develop a satisfactory model. Other image generation methods have similar structures, such as Variational Autoencoders (VAEs), which generate a constrained latent space distribution through an encoder, and then use a decoder to generate adversarial images from the sampled latent vectors. However, these methods generate adversarial images of poor quality and low accuracy, showing a significant difference from the original real-world images. Summary of the Invention

[0003] This application proposes an adversarial patch generation method, electronic device, and computer-readable storage medium to enhance the offensiveness of the generated adversarial patch images.

[0004] To address the aforementioned technical problems, this application provides a method for generating adversarial patches, comprising: acquiring a noisy image of the patch region of the original image and a target image; acquiring a first gradient based on the noisy image and the target image; acquiring an adversarial image based on the noisy image and acquiring a second gradient based on the adversarial image; and acquiring an adversarial patch image using a diffusion model based on the first gradient, the second gradient, and the noisy image.

[0005] The step of obtaining an adversarial patch image using a diffusion model based on a first gradient, a second gradient, and a noisy image includes: obtaining a first embedding vector based on the noisy image; obtaining a temporal embedding vector for the diffusion time step of the diffusion model; using the first gradient and the second gradient as guiding conditions, using the first embedding vector and the temporal embedding vector to predict and obtain an unnoisy image of the patch region of the original image based on the diffusion model; and sampling the unnoisy image to obtain the adversarial patch image.

[0006] The step of obtaining the first embedding vector based on the noisy image includes: processing the noisy image using a first encoder to obtain the first embedding vector.

[0007] The noise in the noisy image is standard Gaussian distributed noise. The steps of using the first gradient and the second gradient as guiding conditions, and using the diffusion model to predict and obtain the unnoisy image of the original image patch region based on the first embedding vector and the temporal embedding vector, include: obtaining the first perturbation mean and the first perturbation variance of the noise predicted by the diffusion model; obtaining the guided diffusion model predicted noise based on the first perturbation mean, the first perturbation variance, the first gradient and the second gradient; and obtaining the unnoisy image based on the first embedding vector, the temporal embedding vector and the guided diffusion model predicted noise.

[0008] The step of obtaining the guided post-diffusion model prediction noise based on the first perturbation mean, the first perturbation variance, the first gradient, and the second gradient includes: obtaining the second perturbation mean of the diffusion model prediction noise based on the first gradient, the second gradient, the first perturbation variance, and the first perturbation mean; obtaining the second perturbation variance of the guided post-diffusion model prediction noise using a network prediction model; and obtaining the guided post-diffusion model prediction noise based on the second perturbation mean and the second perturbation variance.

[0009] The step of obtaining the first gradient based on the noisy image and the target image includes: inputting the noisy image of the patched region of the original image into a first encoder to obtain a first embedding vector; inputting the target image into a second encoder to obtain a second embedding vector; and obtaining the first gradient based on the first embedding vector and the second embedding vector.

[0010] The step of obtaining the first gradient based on the first embedding vector and the second embedding vector includes: obtaining a first guiding scaling factor; calculating the third gradient of the dot product of the first embedding vector and the second embedding vector using a gradient function; and calculating the first gradient based on the third gradient and the first guiding scaling factor.

[0011] The step of obtaining the second gradient based on the adversarial image includes: obtaining a first guiding scale factor and a second guiding scale factor; inputting the adversarial image into the target recognition model to obtain the target's cross-entropy; calculating the fourth gradient based on the target's cross-entropy using a gradient function; and obtaining the second gradient based on the first guiding scale factor, the second guiding scale factor, and the fourth gradient.

[0012] To solve the above-mentioned technical problems, one technical solution adopted in this application is to provide an electronic device, which includes a processor and a memory connected to the processor. The memory stores program data, and the processor executes the program data stored in the memory to execute the anti-patch generation method that implements any of the above-mentioned methods.

[0013] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a computer-readable storage medium that stores program instructions internally, which are executed to implement the anti-patch generation method of any of the above-mentioned methods.

[0014] The beneficial effects of this application are as follows: Unlike existing technologies, the adversarial patch generation method of this application obtains a first gradient by comparing the noisy image of the patch region in the original image with the target image; it obtains a second gradient by using the adversarial image; and it uses the first and second gradients to guide the generation of the adversarial patch image using a diffusion mode, so that the generated adversarial patch image contains as much target information as possible, making the adversarial patch image highly offensive to the recognition model. Secondly, while preserving the image features of the original image, the adversarial patch image with target features is generated by guiding the generation through the first and second gradients. Such an adversarial patch image has good generalization and transferability, and also has a good attack effect on black-box recognition models. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating an embodiment of the anti-patch generation method of this application;

[0016] Figure 2 This is a schematic diagram of an embodiment of an original human face image;

[0017] Figure 3 This is a training diagram of an embodiment of the ef-clip model of this application;

[0018] Figure 4 yes Figure 1 A flowchart illustrating a specific embodiment of step S104;

[0019] Figure 5 yes Figure 4 A flowchart illustrating a specific embodiment of step S203;

[0020] Figure 6 yes Figure 5 A flowchart illustrating a specific embodiment of step S302;

[0021] Figure 7 yes Figure 6 A flowchart illustrating a specific embodiment of step S401;

[0022] Figure 8 yes Figure 1 A flowchart illustrating a specific embodiment of step S102;

[0023] Figure 9 yes Figure 8 A flowchart illustrating a specific implementation method for step S603;

[0024] Figure 10 yes Figure 1 A flowchart illustrating a specific embodiment of step S103;

[0025] Figure 11This is a schematic diagram of an embodiment of generating adversarial patch images according to this application;

[0026] Figure 12 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0027] Figure 13 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0029] In the past, Generative Adversarial Networks (GANs) have dominated the field of image generation, generating images with high realism, comparable to real-world images. Their structure mainly consists of a generator and a discriminator. The generator produces images, and the discriminator monitors the similarity between the generated and original images. However, this type of method has significant drawbacks, such as low diversity of generated adversarial images, limited to generating only similar samples already in the training set. Furthermore, training a GAN is a complex and difficult process; the balance between the adversarial forces between the generator and discriminator is difficult to measure well, requiring continuous adjustments and iterations during training to develop a satisfactory model. Other image generation methods have similar structures, such as Variational Autoencoders (VAEs), which generate a constrained latent space distribution through an encoder, and then use a decoder to generate adversarial images from the sampled latent vectors. However, these methods generate adversarial images of poor quality and low accuracy, showing a significant difference from the original real-world images.

[0030] Before introducing the scheme of this application, it is necessary to first introduce the diffusion model used in this application.

[0031] The term diffusion originates from thermodynamics and is inspired by non-equilibrium thermodynamics. It defines a Markov chain of diffusion steps, gradually adding random noise to the data, and then learning the reverse diffusion process to construct the desired data sample from the noise. Unlike VAEs or flow models, diffusion models are learned using a fixed procedure, and the latent variables have high dimensionality (the same as the original data). The forward process of a diffusion model is to continuously add noise to the image, eventually approximating a Gaussian noise distribution. The reverse process is the denoising process, gradually restoring the image from the noise.

[0032] In this application, the generation of adversarial patch images of face images is used as an example to introduce the adversarial patch generation method of this application.

[0033] In adversarial attacks on faces, physical patching attacks can also be categorized as a type of image generation. This primarily involves replacing a region of a face image, causing the face recognition model to incorrectly identify it as another target. This is a crucial area for face security. Previous methods for generating patch images mostly relied on a generator. First, a patch image with a noisy distribution was initialized as input. Then, gradients were backpropagated through multiple supervised losses to update this noisy input. Finally, the generated patch image was pasted onto the face image, forming the adversarial image.

[0034] Generally, the generated adversarial patch image needs to be as consistent as possible with the patched region of the original image, meaning it should be indistinguishable to the naked eye but able to fool the recognition model into identifying it as the target person. However, in current technologies, the adversarial images generated by the methods described above are of poor quality and low accuracy, showing a significant difference from the original images in the real world. Furthermore, to achieve higher quality adversarial images, each adversarial image requires separate supervised training, resulting in high training costs. Moreover, the generated adversarial patch images are constrained by multiple loss functions, lacking diversity. Additionally, the training process often requires a recognition model as the optimization objective, so the training results may perform well with that particular recognition model but cannot be easily transferred to other recognition models.

[0035] Therefore, to solve the above problems, this application first proposes an adversarial patch generation method, please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the anti-patch generation method of this application. Figure 1 As shown, the anti-patch generation method of this embodiment specifically includes steps S101 to S104:

[0036] Step S101: Obtain the noisy image of the patched region of the original image and the target image.

[0037] Taking the generation of adversarial patch images of faces as an example, in this embodiment, the electronic device first needs to obtain the noisy image of the patch area of ​​the original face image and the target image of the face to be identified in the recognition model. The target image of the face in the recognition model can be obtained in any way, and there are no restrictions here.

[0038] Obtaining a noisy image of the original image patch region requires obtaining a noisy image of the original image patch region. In this embodiment, the original image patch region of the face can be the forehead region, eye region, nose region, or mouth region, etc., but is not limited to these. The original image patch region of the face in this embodiment can be any face region, and the shape and position of the patch can be set. The size of the patch region of the original face image is generally limited to no more than 10% to 20% of the original face image.

[0039] Please see Figure 2 , Figure 2 This is a schematic diagram of an embodiment of an original human face image. (See diagram below.) Figure 2 As shown, this embodiment can obtain an un-noiseed image of the patched area in the forehead region of the original face image. In addition, in this embodiment, the un-noiseed image of the patched area of ​​the original face image can be denoised to obtain a denoised image of the patched area of ​​the original image.

[0040] Step S102: Obtain the first gradient based on the noisy image and the target image.

[0041] After obtaining the noisy image of the original image patch region and the target image of the recognition model, they are respectively input into an ef-clip model trained in this embodiment. In this embodiment, the model consists of a first encoder and a second encoder.

[0042] Before inputting the noisy image and the target image, this embodiment first trains the ef-clip model, and the training method is as follows:

[0043] Please see Figure 3 , Figure 3 This is a training diagram of an embodiment of the ef-clip model of this application. For example... Figure 3 As shown, in this embodiment, the ef-clip model can be trained using a contrastive learning method. The structures of the first and second encoders are not strictly limited and can be common image coding models in existing technologies, such as Transform or U-Net networks. Assuming that in a batch of training, the noisy image of the original image patch region is input into the first encoder to obtain the first embedding vector e, and the target image is input into the second encoder to obtain the second embedding vector f. The final model output is: (e1, f1)(e2, f2)…(e...). n ,f n Let (e) i ,f i ) is a pair of positive samples (e i It is the first embedding vector, f i (These are the second embedding vectors), their labels are 1, ei with f j,j≠i It is a negative sample, labeled 0. Similarly, (f i ,e i ) constitute a pair of positive samples, f i With e i≠j These constitute negative samples. In this embodiment, cross-entropy can be used as the loss function, as shown in equation (1):

[0044]

[0045] in, To minimize the output value of the loss function; n is the number of sample pairs in a batch, and log(.) is the logarithmic function. and It is the probability distribution function of sample similarity. Specifically, in this embodiment, e is calculated using the softmax function. i with f j The similarity probability distribution is shown in Equations (2) to (3):

[0046]

[0047]

[0048] In the above calculation process, the first embedding vector e and the second embedding vector f can be normalized so that their magnitude is equal to 1.

[0049] In this embodiment, the loss function described above is minimized. After training the ef-clip model, the first embedding vector e and the second embedding vector f output by the trained ef-clip model will have similar representations in the feature space.

[0050] In the process of generating adversarial patch images, the electronic device can input the noisy image of the patched region of the original image into the first encoder of the trained ef-clip model to obtain the first embedding vector, input the target image into the second encoder of the trained ef-clip model to obtain the second embedding vector, calculate the dot product between the two, and finally obtain the first gradient based on the dot product, the preset gradient function, and the first guiding scaling factor. The specific calculation method is described below.

[0051] Step S103: Obtain an adversarial image based on the noisy image, and obtain a second gradient based on the adversarial image.

[0052] In this embodiment, we will first introduce the meaning of adversarial patching attacks. Adversarial patching attacks refer to replacing pixels in certain areas of the original face image to obtain an adversarial image. This adversarial image differs from the original image only in the patched areas, but the target recognition model incorrectly identifies it as another person or a specific target we want to attack. For example... Figure 2 As shown, Figure 2 This is an example of generating a patch in the forehead area of ​​a face.

[0053] In this embodiment, in order to obtain the second gradient, the noisy image patch of the original image patch region is patched to the corresponding position of the original image to obtain the adversarial image. Then, the adversarial sample is input into the target recognition model, the target recognition model outputs the cross-entropy of the target, and finally, the second gradient can be obtained based on the target's cross-entropy, the first guiding scaling factor, and the second guiding scaling factor.

[0054] Step S104: Obtain the adversarial patch image based on the first gradient, the second gradient, and the noisy image using a diffusion model.

[0055] Based on the foregoing, the goal of this embodiment is to obtain an adversarial patch image that can mislead the results of the target recognition model after replacing certain regions of the original image.

[0056] In order to make the obtained adversarial patch image include information of the target image and be more offensive to the target recognition model, this embodiment first uses the first gradient and the second gradient as guiding conditions to obtain the denoised image of the patch region of the original image using a diffusion model. Finally, the denoised image of the patch region of the original image is sampled and processed to obtain the adversarial patch image with target image information.

[0057] Unlike existing technologies, the adversarial patch generation method of this application obtains a first gradient from the noisy image of the patch region of the original image and the target image; it obtains a second gradient from the adversarial image, and uses the first and second gradients to guide the generation of the adversarial patch image using a diffusion mode, so that the generated adversarial patch image contains as much target information as possible, making the adversarial patch image highly offensive to the recognition model. Furthermore, while preserving the image features of the original image, the adversarial patch image with target features is generated by guiding the generation through the first and second gradients. Such an adversarial patch image has good generalization and transferability, and also has a good attack effect on black-box recognition models.

[0058] Optionally, the method for obtaining the adversarial patch image is as follows: Figure 4 As shown, please refer to Figure 4 , Figure 4 yes Figure 1A flowchart illustrating a specific embodiment of step S104. This embodiment can be achieved through, as follows: Figure 4 The method shown implements step S104, specifically including steps S201 to S204:

[0059] Step S201: Obtain the first embedding vector based on the noisy image.

[0060] Based on the above embodiments, in this embodiment, the electronic device uses a first encoder to process the noisy image to obtain a first embedding vector, that is, the noisy image of the patched region of the original image is input into the first encoder of the trained ef-clip model.

[0061] Step S202: Obtain the temporal embedding vector of the diffusion time step of the diffusion model.

[0062] In this embodiment, the diffusion model is a diffusion generation model. Before using the diffusion model, we will first briefly introduce the working principle of the diffusion model.

[0063] The forward process of the diffusion model involves continuously adding noise to the image, eventually resulting in an image distribution that approximates Gaussian noise. The reverse process of the diffusion model, on the other hand, is the process of denoising the image, gradually restoring the image from the noise.

[0064] First, we introduce the forward process of the diffusion model. Assume we have an original, unnoised image x0, which can be considered as being sampled from a q-distribution. Then, we can continuously add variance β... t The noise is Gaussian, thus obtaining a series of potential Markov chain samples, where the expression for each step is shown in Equation (4):

[0065]

[0066] Where, β t Given a geometric sequence of numbers from 0 to 1, with the initial time β0 = 0 representing the original image, at step t, β... t =1. At this time, the graph x t This is from a standard Gaussian noise The noisy image is obtained by sampling. In fact, the diffusion process is the process of transforming an original image into a standard Gaussian distribution. Noise addition and Gaussian sampling are equivalent, that is, noise addition is changing the mean center point of Gaussian sampling. When training the diffusion model of this embodiment, this embodiment can use formula (5) to obtain the noisy image as the training set, without using formula (4) to iteratively obtain the noisy image step by step. Formula (5) is as follows:

[0067]

[0068] Where, α t =1-βt , It is a standard Gaussian noise distribution.

[0069] The denoising process of the diffusion model is the reverse of the forward process described above. It can gradually denoise a randomly selected Gaussian noise image to obtain the original image. The expressions for its denoising process are shown in formulas (6) and (7):

[0070]

[0071]

[0072] Where t∈[1,2,…,T], z=0 when t=0, otherwise For a standard Gaussian distribution, σ t Let Variance be the variance.

[0073] The denoising process of the diffusion model requires predicting the added Gaussian noise using the diffusion model to obtain the original noise-free image. Equation (7) indicates that the original un-noised image of step n is calculated using the diffusion model. In essence, it is only necessary to calculate the mean μ and variance, and then perform a sampling to obtain the original image. In order to calculate the mean, it is necessary to predict the noise and deduce the mean center of the original image. Through extensive experiments by the inventors, it was found that the variance in the original image can be obtained by network prediction or can be a constant.

[0074] In this embodiment, the diffusion model is to input the noisy image of the original image patch region, subtract the noise predicted by the model, and thus obtain the adversarial patch image. In this process, the diffusion model needs to input the temporal embedding vector of the diffusion time step to know which step of random sampling the noisy image of the original image patch region input into the diffusion model is in.

[0075] In this embodiment, the diffusion model needs to be trained in advance. During the training process, a noisy image at step t is randomly sampled (this image can be obtained from the original image via formula (5)). The noisy image at step t and the step number t are then input into the diffusion model. The diffusion model predicts noise ∈ . The training objective of the diffusion model is to minimize the error between the predicted noise and the actual added noise. The sampling process is to subtract the noise predicted by the model from the noisy image at step t, and continuously remove the noise to recover the original image.

[0076] The optimization objective of the diffusion model is shown in equation (8):

[0077]

[0078] In the above formula (8), x tFor the noise sample obtained by formula (5), ∈ θ It is a diffusion model. It is real noise and conforms to a standard Gaussian distribution. Equation (8) means that the noise predicted by the model should be as close as possible to the actual noise added.

[0079] Step S203: Using the first gradient and the second gradient as guiding conditions, the original image patch region is obtained by using a diffusion model to predict based on the first embedding vector and the temporal embedding vector.

[0080] In this embodiment, in order to make the generated adversarial patch image carry the feature information of the target image, it is necessary to use the first gradient and the second gradient as guiding conditions. The diffusion model uses the first embedding vector and the temporal embedding vector to predict and obtain the denoised image of the patch region of the original image.

[0081] Step S204: Sample the un-noiseed image to obtain the adversarial patch image.

[0082] When the diffusion model obtains the denoised image of the original image patch region at the t-th sampling step, and then calculates the denoised image of the original image patch region at the t-1-th step of the previous time step based on formula (7), and repeats this calculation until t=0, the adversarial patch image can be obtained.

[0083] Optionally, please refer to Figure 5 , Figure 5 yes Figure 4 A flowchart illustrating a specific embodiment of step S203. The noise in the noisy image can be set to standard Gaussian distributed noise. This embodiment can be achieved through, as shown in... Figure 5 The method shown implements step S203, which specifically includes steps S301 to S303:

[0084] Step S301: Obtain the first perturbation mean and the first perturbation variance of the noise predicted by the diffusion model.

[0085] This embodiment uses a conditionally guided diffusion model, assuming that the un-noiseed image of the original image patch region at time t is x. t At this point, before calculating the adversarial patch image, this application needs to obtain the first perturbation mean μ of the predicted noise of the diffusion model. θ (x t ,t) and the first disturbance variance ∑ θ (x t ,t).

[0086] Step S302: Obtain the guided post-diffusion model prediction noise based on the first perturbation mean, the first perturbation variance, the first gradient, and the second gradient.

[0087] Furthermore, in order to ensure that the generated adversarial patch image possesses the feature information of the target image, this application also requires calculating the guided diffusion model prediction noise using the first gradient and second gradient described above. Since the guided diffusion model prediction noise also follows a Gaussian distribution, the second perturbation mean and second perturbation variance of the guided diffusion model prediction noise are calculated respectively.

[0088] Step S303: Obtain the noisy image based on the first embedding vector, the temporal embedding vector, and the guided post-diffusion model to predict noise.

[0089] In this embodiment, based on the first embedding vector of the noisy image of the original patch region and the time embedding vector input to the diffusion model, the noise is predicted by subtracting the guided diffusion model from the diffusion model to obtain the noisy image of the original image patch region.

[0090] Optionally, please refer to Figure 6 , Figure 6 yes Figure 5 A flowchart illustrating a specific embodiment of step S302. The steps for obtaining the predicted noise from the guided diffusion model are as follows: Figure 6 As shown, this embodiment can be achieved through, as... Figure 6 The method shown implements step S302, specifically including steps S401 to S403:

[0091] Step S401: Obtain the second perturbation mean of the diffusion model prediction noise based on the first gradient, the second gradient, the first perturbation variance, and the first perturbation mean.

[0092] In this embodiment, the second perturbation mean of the guided diffusion model prediction noise is calculated by using the first gradient obtained based on the ef-clip model, the second gradient obtained based on the target recognition mode, the first perturbation variance and the first perturbation mean of the diffusion model prediction noise. The specific calculation method is as follows.

[0093] Specifically, please refer to Figure 7 , Figure 7 yes Figure 6 A flowchart illustrating a specific embodiment of step S401. The method for calculating the second perturbation mean of the diffusion model predicted noise is as follows: Figure 7 As shown, this embodiment can be achieved through, as... Figure 7 The method shown implements step S401, specifically including steps S501 to S502:

[0094] Step S501: Calculate the first sum of the first gradient and the second gradient, and calculate the product of the first sum and the first perturbation variance.

[0095] The electronic device first calculates the first sum (f1+f2) of the first gradient f1 and the second gradient f2, and then calculates the product ∑ of the first sum (f1+f2) and the first disturbance variance. θ (x t ,t)(f1+f2).

[0096] Step S502: Calculate the second sum of the product and the first perturbation mean to obtain the second perturbation mean of the noise predicted by the guided post-diffusion model.

[0097] Finally, the electronic device recalculates the product value ∑ θ (x t ,t)(f1+f2) and the mean value of the first disturbance μ θ (x t The second sum of ,t) is used as the second perturbation mean of the noise predicted by the guided post-diffusion model.

[0098] Based on the foregoing, this embodiment can predict the second perturbation mean of the diffusion model prediction noise based on formula (9), as shown in formula (9):

[0099]

[0100] in, Let μ be the mean of the second perturbation of the noise predicted by the guided post-diffusion model. θ (x t ,t) represents the mean of the first disturbance of the noise predicted by the diffusion model, ∑ θ (x t ,t) represents the first perturbation variance of the noise predicted by the diffusion model, f1 represents the first gradient, and f2 represents the second gradient.

[0101] Step S402: Use the network prediction model to obtain the second perturbation variance of the predicted noise of the guided post-diffusion model.

[0102] In this embodiment, the electronic device can use a network prediction model to obtain the second perturbation variance of the noise predicted by the guided diffusion model. In other embodiments, after extensive testing by the inventors, the second perturbation variance of the noise predicted by the guided diffusion model can also be taken as a constant.

[0103] Step S403: Obtain the guided post-diffusion model prediction noise based on the second perturbation mean and the second perturbation variance.

[0104] The noisy image in the patch region of the original image is composed of a Gaussian distribution. Therefore, the prediction noise of the guided post-diffusion model is also composed of a Gaussian distribution. When obtaining the prediction noise of the guided post-diffusion model, the second perturbation mean and the second perturbation variance are obtained based on the above steps, and the prediction noise of the guided post-diffusion model can be obtained.

[0105] Optionally, the method for obtaining the first gradient is as follows: Figure 8 As shown, please refer to Figure 8 , Figure 8 yes Figure 1 A flowchart illustrating a specific embodiment of step S102. This embodiment can be achieved through, as shown below... Figure 8 The method shown implements step S102, specifically including steps S601 to S603:

[0106] Step S601: Input the noisy image of the original image patch region into the first encoder to obtain the first embedding vector.

[0107] When the electronic device acquires the first gradient f1, it first uses the first encoder in the ef-clip model mentioned earlier to input the noisy image of the original image patch region into the first encoder to obtain the first embedding vector e(x). t ).

[0108] Step S602: Input the target image into the second encoder to obtain the second embedding vector.

[0109] The electronic device will use the second encoder in the ef-clip model mentioned above to input the target image into the second encoder to obtain the first embedding vector f(c).

[0110] Step S603: Obtain the first gradient based on the first embedding vector and the second embedding vector.

[0111] Finally, when calculating the first gradient, the first embedding vector e(x) is used. t The first gradient f1 is obtained from the second embedding vector f(c) and the second embedding vector f(c). The calculation method is shown below.

[0112] Optionally, the method for obtaining the first gradient based on the first embedding vector and the second embedding vector is as follows: Figure 9 As shown, please refer to Figure 9 , Figure 9 yes Figure 8 A flowchart illustrating a specific implementation method for step S603. This embodiment can be achieved through, as follows: Figure 9 The method shown implements step S603, which specifically includes steps S701 to S703:

[0113] Step S701: Obtain the first guiding scaling factor.

[0114] When calculating the first gradient f1, the electronic device needs to obtain the first guiding scaling factor s. The first guiding scaling factor s controls the guiding strength; the larger the value of s, the higher the accuracy. The value of s can be set based on actual conditions, and will not be elaborated here.

[0115] Step S702: Calculate the third gradient of the dot product of the first embedding vector and the second embedding vector using the gradient function.

[0116] The electronic device first needs to calculate the first embedding vector e(x) t The dot product of the second embedding vector f(c) and e(x) t Then, using the gradient function, calculate the third gradient of the dot product of the first and second embedding vectors.

[0117] Step S703: Calculate the first gradient based on the third gradient and the first guiding scaling factor.

[0118] Finally, the electronic device can calculate the first gradient f1 based on the third gradient and the first guiding scaling factor according to formula (10), as shown below:

[0119]

[0120] Optionally, the method for obtaining the second gradient is as follows: Figure 10 As shown, please refer to Figure 10 , Figure 10 yes Figure 1 A flowchart illustrating a specific embodiment of step S103. This embodiment can be achieved through, as shown below... Figure 10 The method shown implements step S103, specifically including steps S801 to S804:

[0121] Step S801: Obtain the first guiding scaling factor and the second guiding scaling factor.

[0122] When calculating the second gradient f2, the electronic device needs to obtain a first guiding scale factor s and a second guiding factor m. These factors control the guiding strength; the larger the values ​​of s and m, the higher the accuracy. The values ​​of s and m can be set based on actual conditions, and will not be elaborated upon here.

[0123] Step S802: Input the adversarial image into the target recognition model to obtain the target's cross-entropy.

[0124] The electronic device inputs the adversarial image into the target recognition model described in the previous embodiment. The adversarial image is obtained by patching the original image patched regions with a noisy image and then patching them to the corresponding positions in the original image. By inputting the adversarial image into the target recognition model, the cross-entropy of the target in the adversarial image can be obtained.

[0125] Step S803: Calculate the fourth gradient based on the cross-entropy of the target using the gradient function.

[0126] Electronic devices acquire the cross-entropy of the target The fourth gradient can be obtained using the same gradient function as before, that is, the fourth gradient is...

[0127] Step S804: Obtain the second gradient based on the first guiding scaling factor, the second guiding scaling factor, and the fourth gradient.

[0128] Finally, the electronic device can calculate the second gradient f2 based on the fourth gradient, the first guiding scaling factor, and the second guiding scaling factor according to formula (11), as shown below:

[0129]

[0130] In one application scenario, please refer to Figure 11 , Figure 11 This is a schematic diagram of an embodiment of generating adversarial patch images according to this application. Figure 11 As shown, the noisy image of the original image patch region is encoded by the first encoder of the ef-clip model to generate the first embedding vector, and the target image is encoded by the first encoder of the ef-clip model to generate the second embedding vector. The first embedding vector and the second embedding vector are calculated to form a dot product. The first embedding vector of the noisy image of the original image patch region and the temporal embedding vector of the diffusion time step are input into the diffusion model for noise reduction. The temporal embedding vector can be linearly encoded or cosine encoded, such as the positional encoding method in the existing transform model. The structure of the diffusion model can be the most advanced network structure, such as the typical U-net network or the transform model. In addition, the generated first embedding vector and the second embedding vector are calculated to form a dot product, and the first gradient can be calculated using the dot product. After the noisy image of the original image patch region is patched into an adversarial image, it is also sent to the target recognition model to calculate the cross-entropy of the target face to obtain the second gradient. Finally, the first gradient and the second gradient are used as guiding conditions to sample the noise of the current time step, so as to obtain the original image patch region without noise in the previous time step. Finally, the adversarial patch image can be obtained by sampling using formula (7) until t=0.

[0131] Unlike existing technologies, the adversarial patch generation method of this application obtains a first gradient from the noisy image of the patch region of the original image and the target image; it obtains a second gradient from the adversarial image, and uses the first and second gradients to guide the generation of the adversarial patch image using a diffusion model, ensuring that the generated adversarial patch image contains as much target information as possible, making it highly offensive against the recognition model. Secondly, while preserving the image features of the original image, the adversarial patch image with target features is generated by guiding the generation using the first and second gradients. Such adversarial patch images exhibit good generalization and transferability, and also show good attack effectiveness against black-box recognition models. This application also introduces the cross-entropy information of the target recognition model as a guiding condition for the diffusion model, guiding the generation direction of the adversarial patch image towards the target of the actual task, further increasing the success rate of the adversarial patch image in breaking the target recognition model.

[0132] Secondly, traditional adversarial attack methods optimize the input as trainable parameters, requiring separate model training for each sample and attack target. However, the diffusion model used in this application, as the image generation model, generates images from a distribution guided by the target, making the sampling differentiable. Therefore, for different adversarial patch images and different attack targets, there is no need to retrain the generator; only the input and target optimization are required, thus reducing training costs.

[0133] Optionally, this application further proposes an electronic device, please refer to [link to relevant documentation]. Figure 12 , Figure 12 This is a schematic diagram of an embodiment of the electronic device of this application. The electronic device 200 includes a processor 201 and a memory 202 connected to the processor 201.

[0134] Processor 201 can also be referred to as CPU (Central Processing Unit). Processor 201 may be an integrated circuit chip with signal processing capabilities. Processor 201 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor can be a microprocessor or any conventional processor.

[0135] The memory 202 is used to store program data required for the processor 201 to run.

[0136] The processor 201 is also used to execute program data stored in the memory 202 to implement any of the above-mentioned methods for generating anti-patch devices.

[0137] Optionally, this application further proposes a computer-readable storage medium. See also... Figure 13 , Figure 13 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application.

[0138] The computer-readable storage medium 300 of this application embodiment internally stores program instructions 310, which are executed to implement the anti-patch generation method of any of the above claims.

[0139] Specifically, program instructions 310 can be formed into a program file and stored in the aforementioned storage medium in the form of a software product, so that an electronic device (which may be a personal computer, server, or network device, etc.) or processor can execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0140] In this embodiment, the computer-readable storage medium 300 may be, but is not limited to, a USB flash drive, SD card, PD optical drive, portable hard drive, large-capacity floppy drive, flash memory, multimedia memory card, server, etc.

[0141] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the electronic device to perform the steps in the above-described method embodiments.

[0142] Furthermore, if the aforementioned functions are implemented as software functions and sold or used as independent products, they can be stored in a mobile terminal-readable storage medium. That is, this application also provides a storage device storing program data, which can be executed to implement the methods of the above embodiments. This storage device can be, for example, a USB flash drive, an optical disc, or a server. In other words, this application can be embodied in the form of a software product, which includes several instructions to cause a smart terminal to execute all or part of the steps of the methods described in the various embodiments.

[0143] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0144] Any process or method description in the flowchart or otherwise herein can be understood as representing an apparatus, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0145] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (which may be a personal computer, server, network device, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0146] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A countermeasure patch generation method characterized by comprising: include: Obtain the noisy image of the patched region of the original image and the target image; A first gradient is obtained based on the noisy image and the target image; An adversarial image is obtained based on the noisy image, and a second gradient is obtained based on the adversarial image; Based on the first gradient, the second gradient, and the noisy image, an adversarial patch image is obtained using a diffusion model. The step of obtaining the first gradient based on the noisy image and the target image includes: inputting the noisy image of the patched region of the original image into a first encoder to obtain a first embedding vector; inputting the target image into a second encoder to obtain a second embedding vector; obtaining a first guiding scaling factor; calculating the third gradient of the dot product of the first embedding vector and the second embedding vector using a gradient function; and calculating the first gradient based on the third gradient and the first guiding scaling factor. The step of obtaining the second gradient based on the adversarial image includes: obtaining a first guiding scale factor and a second guiding scale factor; inputting the adversarial image into a target recognition model to obtain the cross-entropy of the target; calculating a fourth gradient based on the cross-entropy of the target using a gradient function; and obtaining the second gradient based on the first guiding scale factor, the second guiding scale factor, and the fourth gradient.

2. The adversarial patch generation method of claim 1, wherein, The step of obtaining an adversarial patch image using a diffusion model based on the first gradient, the second gradient, and the noisy image includes: The first embedding vector is obtained based on the noisy image; Obtain the temporal embedding vector of the diffusion time step of the diffusion model; Using the first gradient and the second gradient as guiding conditions, the diffusion model is used to predict and obtain the noisy image of the original image patch region based on the first embedding vector and the temporal embedding vector. The adversarial patch image is obtained by sampling the un-noiseed image.

3. The method for generating adversarial patches according to claim 2, characterized in that, The step of obtaining the first embedding vector based on the noisy image includes: The noisy image is processed using a first encoder to obtain the first embedding vector.

4. The method for generating adversarial patches according to claim 2, characterized in that, The noise in the noisy image is standard Gaussian distributed noise. The step of using the first gradient and the second gradient as guiding conditions, and using the diffusion model to predict and obtain the denoised image of the original image patch region based on the first embedding vector and the temporal embedding vector, includes: Obtain the first perturbation mean and the first perturbation variance of the noise predicted by the diffusion model; The guided diffusion model predicts noise based on the first perturbation mean, the first perturbation variance, the first gradient, and the second gradient. The noise-free image is obtained based on the first embedding vector, the temporal embedding vector, and the noise predicted by the guided diffusion model.

5. The method for generating adversarial patches according to claim 4, characterized in that, The step of obtaining the guided diffusion model to predict noise based on the first perturbation mean, the first perturbation variance, the first gradient, and the second gradient includes: The second perturbation mean of the noise predicted by the diffusion model is obtained based on the first gradient, the second gradient, the first perturbation variance, and the first perturbation mean. The second perturbation variance of the noise predicted by the diffusion model after guidance is obtained using a network prediction model. The diffusion model predicts noise based on the second perturbation mean and the second perturbation variance.

6. An electronic device, characterized in that, The electronic device includes a processor and a memory connected to the processor, wherein the memory stores program data, and the processor executes the program data stored in the memory to perform the anti-patch generation method according to any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, It internally stores program instructions that are executed to implement the anti-patch generation method according to any one of claims 1-5.

Images