A non-restrictive adversarial sample generation method and system based on GAN

Abstract

The present invention provides a GAN-based non-restrictive adversarial sample generation method and system, relating to the field of artificial intelligence security technology, comprising: obtaining an initial input image and a target image and performing preprocessing; inputting the preprocessed initial input image into a GAN neural network model for training, encoding the preprocessed initial input image and the target image into a latent space using an encoder, and regularizing the obtained latent vector; using the latent vector of the regularized target image as the initial value of the latent vector of the adversarial sample, constructing an adversarial sample similarity objective function and iteratively updating it in the latent space to obtain the latent vector of the adversarial sample to be generated; inputting the latent vector of the adversarial sample to be generated into an optimized GAN neural network model to generate the adversarial sample; the present invention utilizes the latent vector of the updated latent space and a generator network G to generate the non-restrictive adversarial sample, thereby ensuring the naturalness of the adversarial sample and improving the generation efficiency and attack success rate of the non-restrictive adversarial sample.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence security technology, and in particular, to a GAN-based non-restrictive adversarial sample generation method and system. Background Art

[0002] In recent years, adversarial attacks have become an increasingly serious threat to deep learning security, making them a hot topic in AI security research. Adversarial examples are samples created by artificially adding subtle perturbations to the original dataset that are invisible to the naked eye or visible to the naked eye but have no effect on the overall outcome after processing. These examples can cause a trained model to produce a classification output that differs from the original sample with high confidence.

[0003] Adversarial attacks can be categorized into white-box and black-box attacks based on their cost. A white-box attack involves the attacker gaining complete access to the target model, understanding its structure and the specific parameters of each layer. This allows them to control the model's inputs and even modify them at the bit level. A black-box attack involves the attacker being completely unaware of the target model's internal information, treating it as a black box and only being able to control its inputs. A major branch of black-box attacks is decision-boundary-based attacks. These attacks rely neither on surrogate models nor on confidence scores; they only require the class labels output by the black-box classifier to succeed.

[0004] Adversarial attacks can be categorized into targeted and untargeted attacks based on their target. The former aims to mislead the network into assigning a specific class, while the latter minimizes the probability of the true class without assigning a specific class to the network output. Untargeted attacks are easier to implement than targeted attacks because the victim model's output can be anything other than the true class.

[0005] Adversarial attacks are currently widely used in artificial intelligence (AI) fields such as autonomous driving, face recognition, and image classification. Image classification is a very active research area within adversarial attacks, and a variety of adversarial attack algorithms have been proposed, such as FGSM, PGD, and the C&W algorithm, all of which have achieved promising results. Among adversarial attack methods, decision boundary-based attacks are more relevant to real-world scenarios and therefore hold greater research value. However, they are more difficult to execute and typically require a higher number of queries. While the Boundary Attack and HSJA algorithms have significantly improved the success rate and query efficiency of decision boundary-based attacks, they are restricted adversarial attacks. Numerous defenses have been proposed against this type of attack, such as adversarial training, image enhancement, and input denoising. This results in a relatively low success rate for restricted adversarial examples.

[0006] The current prior art discloses a method for generating semantic adversarial samples based on GAN, including: S102. Pre-training a WGAN_GP neural network model using a local training set to obtain a generator and a discriminator; S104. Initializing noise data in a low-dimensional submanifold space; S106. Obtaining a generated image adversarial sample through the generator mapping; S108. Using the image adversarial sample to perform an adversarial attack on a target classifier including attack terms and semantic terms; S110. Determining whether the image classification output by the target classifier is the original classification category; if not, outputting a semantic adversarial sample; if so, increasing the weight parameter of the attack term, and updating the initial value of the noise data, and iteratively repeating steps S106 to S110; the method in the prior art directly updates the adversarial sample in the sense of the image space, and based on the restricted adversarial attack, there is a problem that a large number of sample queries are required, and the success rate and generation efficiency of the adversarial attack are low. Summary of the Invention

[0007] In order to overcome the defects of the above-mentioned prior art in the success rate and generation efficiency of restrictive adversarial sample attacks, the present invention provides a non-restrictive adversarial sample generation method and system based on GAN, which can generate non-restrictive samples, improve the success rate of adversarial attacks while improving the generation efficiency.

[0008] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0009] A non-restrictive adversarial sample generation method based on GAN, comprising the following steps:

[0010] S1: Obtain an initial input image and a target image, and preprocess the initial input image to obtain a preprocessed initial input image;

[0011] S2: Input the preprocessed initial input image into the preset GAN neural network model for training to obtain an optimized GAN neural network model;

[0012] S3: Encode the preprocessed initial input image and target image into the latent space using an encoder to obtain the latent vector of the initial input image and the latent vector of the target image, and regularize the latent vector of the initial input image and the latent vector of the target image;

[0013] S4: Use the latent vector of the regularized target image as the initial value of the latent vector of the adversarial sample, construct the adversarial sample similarity objective function, and iteratively update the latent vector of the adversarial sample using the latent vector of the regularized initial input image in the latent space to obtain the latent vector of the adversarial sample to be generated;

[0014] S5: Input the latent vector of the adversarial sample to be generated into the optimized GAN neural network model to obtain the adversarial sample.

[0015] Preferably, in step S2, the preset GAN neural network model is specifically a StyleGAN2 neural network model, including a generator network G, a discriminator network D and a pSp encoder.

[0016] Preferably, the adversarial sample similarity objective function constructed in step S4 is specifically:

[0017] The Euclidean distance is used to measure the similarity between two vectors. The goal is to minimize the Euclidean distance between the latent vector of the regularized initial input image and the latent vector of the adversarial sample while making the adversarial sample predicted as the target category. The following adversarial sample similarity objective function is constructed:

[0018]

[0019] Among them, d(w ori , w adv ) represents the latent vector w of the initial input image after regularization ori and the latent vector w of the adversarial sample adv The Euclidean distance between the hidden vector w of the adversarial sample adv The initial value is the latent vector w of the target image trg ;constraint Indicates that the adversarial example is predicted to be the target class.

[0020] Preferably, in the adversarial sample similarity objective function, is a Boolean function, specifically:

[0021]

[0022] in, Indicates that the adversarial sample is predicted to be the target category, Indicates that the adversarial sample is not predicted as the target category;

[0023] It indicates the function of using the preset classification model to determine whether the adversarial sample is predicted as the target category. , it means that the adversarial sample is predicted as the target category, otherwise it means that the adversarial sample is not predicted as the target category. The function is specifically:

[0024]

[0025] Among them, y trg Indicates the target category represented by the target image, F c () is the probability output of the preset classification model for the category label c.

[0026] Preferably, in step S4, the latent vector of the adversarial sample is iteratively updated in the latent space using the latent vector of the regularized initial input image to obtain the latent vector of the adversarial sample to be generated. The specific method is:

[0027] S4.1: In the tth iteration, the latent vector w of the initial input image after regularization is used ori , in the latent space, the latent vector w of the adversarial sample is searched by binary search adv Perform boundary search to obtain the hidden vector w of the adversarial sample that reaches the boundary in the tth iteration t , and calculate the perturbation size δ of the tth iteration t ;

[0028] S4.2: Use the Monte Carlo method to estimate the gradient direction and iteratively update the step size ξ of the tth iteration along the gradient direction t ;

[0029] S4.3: According to the perturbation size δ of the tth iteration t and the step size δ of the tth iteration t Update the hidden vector w of the adversarial sample that reaches the boundary at the tth iteration t , obtain the hidden vector of the adversarial sample after the tth iteration update And calculate the hidden vector of the adversarial sample after the tth iteration update and the latent vector w of the initial input image after regularization ori The Euclidean distance d between t ;

[0030] S4.4: Determine whether the number of iterations t is less than a preset threshold T. If so, repeat steps S4.1 to S4.3. Otherwise, output the latent vector of the updated adversarial sample with the smallest Euclidean distance as the latent vector of the adversarial sample to be generated.

[0031] Preferably, in step S4.1, the latent vector w of the regularized initial input image is used ori , in the latent space, the latent vector w of the adversarial sample is searched by binary search adv Perform boundary search to obtain the hidden vector w of the adversarial sample that reaches the boundary in the tth iteration t , and calculate the perturbation size δ of the tth iteration t , the specific method is:

[0032] Using the latent vector w of the regularized initial input image ori , in the latent space, the latent vector w of the adversarial sample is searched by binary search abv Perform boundary search to obtain the hidden vector w of the adversarial sample that reaches the boundary in the tth iteration t, the specific formula is:

[0033]

[0034] Among them, Bin Search() is a binary search, θ is the parameter of the binary search, It is w t-2 Obtained after updating the gradient direction;

[0035] Calculate the perturbation size δ for the tth iteration t The specific formula is:

[0036]

[0037] Where d is the size of the initial input image after preprocessing.

[0038] Preferably, in step S4.2, the Monte Carlo method is used to estimate the gradient direction, and the step size ξ of the tth iteration is iteratively updated along the gradient direction. t , the specific method is:

[0039] The Monte Carlo method is used to estimate the gradient direction. The specific formula is:

[0040]

[0041] in, Indicates that w is t Find the gradient, is a uniform distribution in the d-dimensional mapping space, To control the baseline of the deviation boundary error, it is specifically expressed as:

[0042] Iteratively update the step size ξ of the tth iteration along the gradient direction t , the specific method is:

[0043] Iteratively update the step size ξ of the tth iteration along the gradient direction t , decreases at a geometric rate until it crosses the boundary, and the step size ξ of the tth iteration t The initial value is

[0044] Preferably, in step S4.3, according to the disturbance size δ of the tth iteration t and the step size ξ of the tth iteration t Update the hidden vector w of the adversarial sample that reaches the boundary at the tth iteration t , obtain the hidden vector of the adversarial sample after the tth iteration update And calculate the hidden vector of the adversarial sample after the tth iteration update and the latent vector w of the initial input image after regularizationori The Euclidean distance d between t , the specific method is:

[0045] According to the perturbation size δ of the tth iteration t and the step size ξ of the tth iteration t Update the hidden vector w of the adversarial sample that reaches the boundary at the tth iteration t , obtain the hidden vector of the adversarial sample after the tth iteration update The specific formula is:

[0046] w t nes =w t +δ t μg t

[0047]

[0048]

[0049] Among them, g t is the cumulative gradient of the tth iteration, g0=0; μ is g t The attenuation factor, ξ t Indicates the step size of the tth iteration;

[0050] Calculate the hidden vector of the adversarial sample after the tth iteration update and the latent vector w of the initial input image after regularization ori The Euclidean distance d between t , the specific formula is:

[0051]

[0052] Among them, d t is the hidden vector of the adversarial sample after the t-th iteration update and the latent vector w of the initial input image after regularization ori The Euclidean distance between .

[0053] Preferably, in step S5, the latent vector of the adversarial sample to be generated is input into the optimized GAN neural network model to obtain the adversarial sample, and the specific method is:

[0054] The latent vector of the adversarial sample to be generated is input into the generator network G in the optimized GAN neural network model to generate the adversarial sample. The objective function of the generator network G is specifically:

[0055]

[0056] Among them, G(w′ adv) represents the generator network G using the latent vector w′ of the adversarial sample to be generated adv Generate adversarial examples.

[0057] The present invention also provides a GAN-based non-restrictive adversarial sample generation system, which applies the above-mentioned GAN-based non-restrictive adversarial sample generation method, including:

[0058] Initialization unit: used to obtain an initial input image and a target image, and preprocess the initial input image to obtain a preprocessed initial input image;

[0059] Model training unit: used to input the preprocessed initial input image into the preset GAN neural network model for training to obtain an optimized GAN neural network model;

[0060] Latent vector encoding unit: used to encode the preprocessed initial input image and target image into the latent space using the encoder, obtain the latent vector of the initial input image and the latent vector of the target image, and regularize the latent vector of the initial input image and the latent vector of the target image;

[0061] Latent vector iteration unit: used to use the latent vector of the regularized target image as the initial value of the latent vector of the adversarial sample, construct the adversarial sample similarity objective function, and iteratively update the latent vector of the adversarial sample using the latent vector of the regularized initial input image in the latent space to obtain the latent vector of the adversarial sample to be generated;

[0062] Adversarial sample generation unit: used to input the latent vector of the adversarial sample to be generated into the optimized GAN neural network model to obtain the adversarial sample.

[0063] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0064] The present invention provides a non-restrictive adversarial sample generation method and system based on GAN. The method obtains an initial input image and a target image, and preprocesses the initial input image to obtain a preprocessed initial input image; inputs the preprocessed initial input image into a preset GAN neural network model for training to obtain an optimized GAN neural network model; uses an encoder to encode the preprocessed initial input image and the target image into a latent space to obtain a latent vector of the initial input image and a latent vector of the target image, and regularizes the latent vector of the initial input image and the latent vector of the target image; uses the latent vector of the regularized target image as the initial value of the latent vector of the adversarial sample, constructs an adversarial sample similarity objective function, iteratively updates the latent vector of the adversarial sample using the latent vector of the regularized initial input image in the latent space to obtain the latent vector of the adversarial sample to be generated; and inputs the latent vector of the adversarial sample to the optimized GAN neural network model to obtain the adversarial sample;

[0065] Unlike traditional methods that directly update adversarial samples in the image space, this paper uses the latent vectors of the updated latent space and the generator network G to generate unrestricted adversarial samples. The generator network G can well map the low-dimensional submanifold space data to the image space, making the generated image adversarial samples as similar as possible to the original image samples and retaining accurate semantic information, thereby improving the attack success rate of the adversarial samples.

[0066] In addition, the present invention uses latent vectors in the latent space to generate adversarial samples, performs semantic constraints in the latent space, ensures the naturalness of adversarial samples, and improves the efficiency of generating non-restrictive adversarial samples. At the same time, by utilizing GAN's ability to approximate data distribution, the attack capability of adversarial samples is effectively guaranteed, so that the generated adversarial samples have both high query efficiency and attack success rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Flowchart of a non-restrictive adversarial sample generation method based on GAN provided in Example 1.

[0068] Figure 2 Flowchart of a non-restrictive adversarial sample generation method based on GAN provided in Example 2.

[0069] Figure 3 This is a structural diagram of a non-restrictive adversarial sample generation system based on GAN provided in Example 3. DETAILED DESCRIPTION

[0070] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0071] In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size;

[0072] It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0073] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0074] Example 1

[0075] like Figure 1 As shown, this embodiment provides a non-restrictive adversarial sample generation method based on GAN, including the following steps:

[0076] S1: Obtain an initial input image and a target image, and preprocess the initial input image to obtain a preprocessed initial input image;

[0077] S2: Input the preprocessed initial input image into the preset GAN neural network model for training to obtain an optimized GAN neural network model;

[0078] S3: Encode the preprocessed initial input image and target image into the latent space using an encoder to obtain the latent vector of the initial input image and the latent vector of the target image, and regularize the latent vector of the initial input image and the latent vector of the target image;

[0079] S4: Use the latent vector of the regularized target image as the initial value of the latent vector of the adversarial sample, construct the adversarial sample similarity objective function, and iteratively update the latent vector of the adversarial sample using the latent vector of the regularized initial input image in the latent space to obtain the latent vector of the adversarial sample to be generated;

[0080] S5: Input the latent vector of the adversarial sample to be generated into the optimized GAN neural network model to obtain the adversarial sample.

[0081] In the specific implementation process, we first obtain the initial input image and target image from the public dataset, and determine the target category to be identified based on the target image;

[0082] Preprocess the initial input image, adjust the size and resolution of the initial input image, and divide it into training set and validation set according to a certain ratio;

[0083] Input the preprocessed initial input image into the preset GAN neural network model for training to obtain an optimized GAN neural network model;

[0084] The encoder corresponding to the GAN neural network model is used to encode the preprocessed initial input image and the target image into a latent space. In this embodiment, the latent space is w + space, obtain the latent vector of the initial input image and the latent vector of the target image, and regularize the latent vector of the initial input image and the latent vector of the target image;

[0085] The latent vector of the regularized target image is used as the initial value of the latent vector of the adversarial sample, and the adversarial sample similarity objective function is constructed. The latent vector of the adversarial sample is iteratively updated in the latent space using the latent vector of the regularized initial input image to obtain the latent vector of the adversarial sample to be generated.

[0086] Finally, the latent vector of the adversarial sample to be generated is input into the optimized GAN neural network model, and the generator network G is used to generate the adversarial sample;

[0087] Unlike traditional methods that directly update adversarial samples in the image space, this paper uses the latent vectors of the updated latent space and the generator network G to generate unrestricted adversarial samples. The generator network G can well map the low-dimensional submanifold space data to the image space, making the generated image adversarial samples as similar as possible to the original image samples and retaining accurate semantic information, thereby improving the attack success rate of the adversarial samples.

[0088] In addition, the present invention uses latent vectors in the latent space to generate adversarial samples, performs semantic constraints in the latent space, ensures the naturalness of adversarial samples, and improves the efficiency of generating non-restrictive adversarial samples. At the same time, by utilizing GAN's ability to approximate data distribution, the attack capability of adversarial samples is effectively guaranteed, so that the generated adversarial samples have both high query efficiency and attack success rate.

[0089] Example 2

[0090] like Figure 2 As shown, this embodiment provides a non-restrictive adversarial sample generation method based on GAN, including the following steps:

[0091] S1: Obtain an initial input image and a target image, and preprocess the initial input image to obtain a preprocessed initial input image;

[0092] S2: Input the preprocessed initial input image into the preset GAN neural network model for training to obtain an optimized GAN neural network model;

[0093] The preset GAN neural network model is specifically a StyleGAN2 neural network model, including a generator network G, a discriminator network D and a pSp encoder;

[0094] S3: Use the pSp encoder to encode the preprocessed initial input image and target image into the latent space, obtain the latent vector of the initial input image and the latent vector of the target image, and regularize the latent vector of the initial input image and the latent vector of the target image;

[0095] S4: Use the latent vector of the regularized target image as the initial value of the latent vector of the adversarial sample, construct the adversarial sample similarity objective function, and iteratively update the latent vector of the adversarial sample using the latent vector of the regularized initial input image in the latent space to obtain the latent vector of the adversarial sample to be generated;

[0096] S5: Input the latent vector of the adversarial sample to be generated into the optimized GAN neural network model to obtain the adversarial sample;

[0097] The adversarial sample similarity objective function constructed in step S4 is specifically:

[0098] The Euclidean distance is used to measure the similarity between two vectors. The goal is to minimize the Euclidean distance between the latent vector of the regularized initial input image and the latent vector of the adversarial sample while making the adversarial sample predicted as the target category. The following adversarial sample similarity objective function is constructed:

[0099]

[0100] Among them, d(w ori , w adv ) represents the latent vector w of the initial input image after regularization ori and the latent vector w of the adversarial sample adv The Euclidean distance between the hidden vector w of the adversarial sample adv The initial value is the latent vector w of the target image trg ;constraint Indicates that the adversarial sample is predicted to be the target category;

[0101] In the adversarial sample similarity objective function, is a Boolean function, specifically:

[0102]

[0103] in, Indicates that the adversarial sample is predicted to be the target category, Indicates that the adversarial sample is not predicted as the target category;

[0104] It indicates the function of using the preset classification model to determine whether the adversarial sample is predicted as the target category. , it means that the adversarial sample is predicted as the target category, otherwise it means that the adversarial sample is not predicted as the target category. The function is specifically:

[0105]

[0106] Among them, y trg Indicates the target category represented by the target image, F c () is the probability output of the preset classification model for the category label c;

[0107] In step S4, the latent vector of the adversarial sample is iteratively updated in the latent space using the latent vector of the regularized initial input image to obtain the latent vector of the adversarial sample to be generated. The specific method is:

[0108] S4.1: In the tth iteration, the latent vector w of the initial input image after regularization is used ori , in the latent space, the latent vector w of the adversarial sample is searched by binary search advPerform boundary search to obtain the hidden vector w of the adversarial sample that reaches the boundary in the tth iteration t , and calculate the perturbation size δ of the tth iteration t ;

[0109] S4.2: Use the Monte Carlo method to estimate the gradient direction and iteratively update the step size ξ of the tth iteration along the gradient direction t ;

[0110] S4.3: According to the perturbation size δ of the tth iteration t and the step size ξ of the tth iteration t Update the hidden vector w of the adversarial sample that reaches the boundary at the tth iteration t , obtain the hidden vector of the adversarial sample after the tth iteration update And calculate the hidden vector of the adversarial sample after the tth iteration update and the latent vector w of the initial input image after regularization ori The Euclidean distance d between t ;

[0111] S4.4: Determine whether the number of iterations t is less than a preset threshold T. If so, repeat steps S4.1 to S4.3. Otherwise, output the latent vector of the updated adversarial example with the smallest Euclidean distance as the latent vector of the adversarial example to be generated.

[0112] In step S4.1, the latent vector w of the regularized initial input image is used ori , in the latent space, the latent vector w of the adversarial sample is searched by binary search adv Perform boundary search to obtain the hidden vector w of the adversarial sample that reaches the boundary in the tth iteration t , and calculate the perturbation size δ of the tth iteration t , the specific method is:

[0113] Using the latent vector w of the regularized initial input image ori , in the latent space, the latent vector w of the adversarial sample is searched by binary search adv Perform boundary search to obtain the hidden vector w of the adversarial sample that reaches the boundary in the tth iteration t , the specific formula is:

[0114]

[0115] Among them, Bin Search() is a binary search, θ is the parameter of the binary search, It is w t-2 Obtained after updating the gradient direction;

[0116] Calculate the perturbation size δ for the tth iterationt The specific formula is:

[0117]

[0118] Where d is the size of the initial input image after preprocessing;

[0119] In step S4.2, the Monte Carlo method is used to estimate the gradient direction, and the step size ξ of the tth iteration is iteratively updated along the gradient direction. t , the specific method is:

[0120] The Monte Carlo method is used to estimate the gradient direction. The specific formula is:

[0121]

[0122] in, Indicates that w is t Find the gradient, is a uniform distribution in the d-dimensional mapping space, To control the baseline of the deviation boundary error, it is specifically expressed as:

[0123] Iteratively update the step size ξ of the tth iteration along the gradient direction t , the specific method is:

[0124] Iteratively update the step size ξ of the tth iteration along the gradient direction t , decreases at a geometric rate until it crosses the boundary, and the step size ξ of the tth iteration t The initial value is

[0125] In step S4.3, according to the disturbance size δ of the tth iteration t and the step size ξ of the tth iteration t Update the hidden vector w of the adversarial sample that reaches the boundary at the tth iteration t , obtain the hidden vector of the adversarial sample after the tth iteration update And calculate the hidden vector of the adversarial sample after the tth iteration update and the latent vector w of the initial input image after regularization ori The Euclidean distance d between t , the specific method is:

[0126] According to the perturbation size δ of the tth iteration t and the step size ξ of the tth iteration t Update the hidden vector w of the adversarial sample that reaches the boundary at the tth iteration t , obtain the hidden vector of the adversarial sample after the tth iteration update The specific formula is:

[0127] w t nes =w t +δ t μg t

[0128]

[0129]

[0130] Among them, g t is the cumulative gradient of the tth iteration, g0=0; μ is g t The attenuation factor, ξ t Indicates the step size of the tth iteration;

[0131] Calculate the hidden vector of the adversarial sample after the tth iteration update and the latent vector w of the initial input image after regularization ori The Euclidean distance d between t , the specific formula is:

[0132]

[0133] Among them, d t is the hidden vector of the adversarial sample after the t-th iteration update and the latent vector w of the initial input image after regularization ori The Euclidean distance between

[0134] In step S5, the latent vector of the adversarial sample to be generated is input into the optimized GAN neural network model to obtain the adversarial sample. The specific method is:

[0135] The latent vector of the adversarial sample to be generated is input into the generator network G in the optimized GAN neural network model to generate the adversarial sample. The objective function of the generator network G is specifically:

[0136]

[0137] Among them, G(w′ adv ) represents the generator network G using the latent vector w′ of the adversarial sample to be generated adv Generate adversarial examples.

[0138] In the specific implementation process, we first obtain the initial input image and target image from the public dataset, and determine the target category to be identified based on the target image;

[0139] The dataset used in this example is the CelebA-HQ dataset, which is a facial image dataset with 30,000 face images, including 6,217 celebrity identities. The resolution of each image is 1024*1024, and it includes information such as facial features and facial attributes of each celebrity.

[0140] The obtained initial input images are preprocessed as follows: each initial input image is resized to 256*256, and then divided into a training set and a validation set at a ratio of 4:1;

[0141] Input the preprocessed initial input image into the StyleGAN2 neural network model for training to obtain the optimized StyleGAN2 neural network model;

[0142] The StyleGAN2 neural network model used in this example is a generative adversarial network that can be used to generate facial images. It consists of two sub-networks: a generator network G and a discriminator network D, as well as a pSp encoder. During GAN training, the generator network G is responsible for generating forged images, while the discriminator network D is responsible for distinguishing between forged images generated by the generator network G and real images. The StyleGAN2 architecture combines the concepts of progressive learning and style transfer to synthesize high-resolution images.

[0143] Then, the pSp encoder is used to encode the preprocessed initial input image and target image into the latent space to obtain the latent vector of the initial input image and the latent vector of the target image, and the latent vector of the initial input image and the latent vector of the target image are regularized;

[0144] The pSp encoder in this embodiment is an encoder designed for the StyleGAN neural network, through which the preprocessed initial input image and target image are encoded into the w of StyleGAN2. + The latent vector is generated in the space (dimension is 18x512). Based on the feature pyramid network structure, the style vector can be extracted from different pyramid scales, which can more accurately reconstruct the input image without the need for additional optimization steps.

[0145] The preprocessed initial input image x ori and the target image x trg Input to the pSp encoder to get the latent vector w of the corresponding initial input image ori and the latent vector w of the target image trg , and regularize the two latent vectors;

[0146] In this embodiment, a loss function of the pSp encoder is also provided. The loss function of the pSp encoder can be expressed as:

[0147] L2(x)=x-pPp(x)||2

[0148] L LPIPS (x)=||F(x)-F(pSp(x))||2

[0149]

[0150] L ID (x) = 1-<R(x),R(pSp(x))>

[0151] L(x)=λ1L2(x)+λ2L LPIPS (x)+λ3L ID (x)+λ4L reg (x)

[0152] Among them, L2(x) is the L2 loss at the pixel level, that is, the mean square error between the target pixel and the training pixel;

[0153] L LPIPS (x) is the perceptual loss, which is trained by the classification function F() and is used to calculate the similarity between two images;

[0154] L reg (x) is the generation loss, that is, the latent vector generated by the pSp encoder E(x) and Here, refers to the average of all vectors;

[0155] L ID (x) is the face loss, using the optimized StyleGAN2 neural network model to calculate the cosine similarity between the input image and the final generated image;

[0156] L(x) is the total loss function of the entire pSp encoder;

[0157] The latent vector w of the regularized target image trg As the initial value of the latent vector of the adversarial sample, the adversarial sample similarity objective function is constructed, specifically:

[0158] The Euclidean distance is used to measure the similarity between two vectors. The goal is to minimize the Euclidean distance between the latent vector of the regularized initial input image and the latent vector of the adversarial sample while making the adversarial sample predicted as the target category. The following adversarial sample similarity objective function is constructed:

[0159]

[0160] Among them, d(w ori , w adv ) represents the latent vector w of the initial input image after regularizationori and the latent vector w of the adversarial sample adv The Euclidean distance between the hidden vector w of the adversarial sample adv The initial value is the latent vector w of the target image trg ;constraint Indicates that the adversarial sample is predicted to be the target category;

[0161] In the adversarial sample similarity objective function, is a Boolean function, specifically:

[0162]

[0163] in, Indicates that the adversarial sample is predicted to be the target category, Indicates that the adversarial sample is not predicted as the target category;

[0164] It indicates the function of using the preset classification model to determine whether the adversarial sample is predicted as the target category. , it means that the adversarial sample is predicted as the target category, otherwise it means that the adversarial sample is not predicted as the target category. The function is specifically:

[0165]

[0166] Among them, y trg Indicates the target category represented by the target image, F c () is the probability output of the preset classification model for the category label c;

[0167] In the latent space, the latent vector of the initial input image after regularization is used to iteratively update the latent vector of the adversarial sample to obtain the latent vector of the adversarial sample to be generated. Specifically,

[0168] S4.1: In the tth iteration, the latent vector w of the initial input image after regularization is used ori , in the latent space, the latent vector w of the adversarial sample is searched by binary search adv Perform boundary search to obtain the hidden vector w of the adversarial sample that reaches the boundary in the tth iteration t , the specific formula is:

[0169]

[0170] Among them, Bin Search() is a binary search, θ is the parameter of the binary search, It is w t-2 Obtained after updating the gradient direction; in the tth iteration, (located in the target class side area) through binary search to continuously approach the boundary until it reaches the boundary and obtains w t ;

[0171] And calculate the perturbation size δ of the tth iteration t , the specific formula is:

[0172]

[0173] Where d is the size of the initial input image after preprocessing;

[0174] S4.2: Monte Carlo method is used to estimate the gradient direction. The specific formula is:

[0175]

[0176] in, Indicates that w is t Find the gradient, is a uniform distribution in the d-dimensional mapping space, To control the baseline of the deviation boundary error, it is specifically expressed as:

[0177] Iteratively update the step size ξ of the tth iteration along the gradient direction t , decreases at a geometric rate until it crosses the boundary, and the step size ξ of the tth iteration t The initial value is

[0178] S4.3: According to the perturbation size δ of the tth iteration t and the step size ξ of the tth iteration t Update the hidden vector w of the adversarial sample that reaches the boundary at the tth iteration t , obtain the hidden vector of the adversarial sample after the tth iteration update The specific formula is:

[0179] w t nes =w t +δ t μg t

[0180]

[0181]

[0182] Among them, g t is the cumulative gradient of the tth iteration, g0=0; μ is g t The attenuation factor, ξ t Indicates the step size of the tth iteration;

[0183] Calculate the hidden vector of the adversarial sample after the tth iteration update and the latent vector w of the initial input image after regularization ori The Euclidean distance d between t , the specific formula is:

[0184]

[0185] Among them, d t is the hidden vector of the adversarial sample after the t-th iteration update and the latent vector w of the initial input image after regularization ori The Euclidean distance between

[0186] S4.4: Determine whether the number of iterations t is less than a preset threshold T. If so, repeat steps S4.1 to S4.3. Otherwise, output the latent vector of the updated adversarial example with the smallest Euclidean distance as the latent vector of the adversarial example to be generated.

[0187] Finally, the latent vector of the adversarial sample to be generated is input into the optimized StyleGAN2 neural network model, and the generator network G is used to generate the adversarial sample;

[0188] The objective function of the generator network G is specifically:

[0189]

[0190] Among them, G(w′ adv ) represents the generator network G using the latent vector w′ of the adversarial sample to be generated adv Generate adversarial examples;

[0191] Unlike traditional methods that directly update adversarial samples in the image space, this paper uses the latent vectors of the updated latent space and the generator network G to generate unrestricted adversarial samples. The generator network G can well map the low-dimensional submanifold space data to the image space, making the generated image adversarial samples as similar as possible to the original image samples and retaining accurate semantic information, thereby improving the attack success rate of the adversarial samples.

[0192] In addition, the present invention uses latent vectors in the latent space to generate adversarial samples, performs semantic constraints in the latent space, ensures the naturalness of adversarial samples, and improves the efficiency of generating non-restrictive adversarial samples. At the same time, by utilizing GAN's ability to approximate data distribution, the attack capability of adversarial samples is effectively guaranteed, so that the generated adversarial samples have both high query efficiency and attack success rate.

[0193] Example 3

[0194] like Figure 3As shown, this embodiment provides a non-restrictive adversarial sample generation system based on GAN, applying the non-restrictive adversarial sample generation method based on GAN described in Example 1 or 2, including:

[0195] Initialization unit 301: used to obtain an initial input image and a target image, and preprocess the initial input image to obtain a preprocessed initial input image;

[0196] Model training unit 302: used to input the pre-processed initial input image into a preset GAN neural network model for training to obtain an optimized GAN neural network model;

[0197] Latent vector encoding unit 303: used to encode the preprocessed initial input image and target image into a latent space using an encoder, obtain the latent vector of the initial input image and the latent vector of the target image, and regularize the latent vector of the initial input image and the latent vector of the target image;

[0198] Latent vector iteration unit 304: used to use the latent vector of the regularized target image as the initial value of the latent vector of the adversarial sample, construct the adversarial sample similarity objective function, and iteratively update the latent vector of the adversarial sample using the latent vector of the regularized initial input image in the latent space to obtain the latent vector of the adversarial sample to be generated;

[0199] Adversarial sample generation unit 305: used to input the latent vector of the adversarial sample to be generated into the optimized GAN neural network model to obtain the adversarial sample.

[0200] In the specific implementation process, the initialization unit 301 first obtains the initial input image and the target image from the public dataset, determines the target category to be recognized based on the target image, preprocesses the initial input image, adjusts the size and resolution of the initial input image, and divides it into a training set and a validation set according to a certain ratio;

[0201] The model training unit 302 inputs the preprocessed initial input image into a preset GAN neural network model for training to obtain an optimized GAN neural network model;

[0202] The latent vector encoding unit 303 uses an encoder to encode the pre-processed initial input image and the target image into a latent space. In this embodiment, the latent space is w + space, obtain the latent vector of the initial input image and the latent vector of the target image, and regularize the latent vector of the initial input image and the latent vector of the target image;

[0203] The latent vector iteration unit 304 uses the latent vector of the regularized target image as the initial value of the latent vector of the adversarial sample, constructs an adversarial sample similarity objective function, and iteratively updates the latent vector of the adversarial sample using the latent vector of the regularized initial input image in the latent space to obtain the latent vector of the adversarial sample to be generated;

[0204] Finally, the adversarial sample generation unit 305 inputs the latent vector of the adversarial sample to be generated into the optimized GAN neural network model and generates the adversarial sample using the generator network G;

[0205] Unlike traditional methods that directly update adversarial samples in the image space, this paper uses the latent vectors of the updated latent space and the generator network G to generate unrestricted adversarial samples. The generator network G can well map the low-dimensional submanifold space data to the image space, making the generated image adversarial samples as similar as possible to the original image samples and retaining accurate semantic information, thereby improving the attack success rate of the adversarial samples.

[0206] In addition, the present invention uses latent vectors in the latent space to generate adversarial samples, performs semantic constraints in the latent space, ensures the naturalness of adversarial samples, and improves the efficiency of generating non-restrictive adversarial samples. At the same time, by utilizing GAN's ability to approximate data distribution, the attack capability of adversarial samples is effectively guaranteed, so that the generated adversarial samples have both high query efficiency and attack success rate.

[0207] The same or similar reference numerals correspond to the same or similar components;

[0208] The terms used in the drawings to describe positional relationships are for illustrative purposes only and should not be construed as limiting this patent;

[0209] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A non-restrictive adversarial sample generation method based on GAN, characterized in that: The following steps are involved: S1: Obtain an initial input image and a target image, and preprocess the initial input image to obtain a preprocessed initial input image; S2: Input the preprocessed initial input image into the preset GAN neural network model for training to obtain an optimized GAN neural network model; S3: Encode the preprocessed initial input image and target image into the latent space using an encoder to obtain the latent vector of the initial input image and the latent vector of the target image, and regularize the latent vector of the initial input image and the latent vector of the target image; S4: Use the latent vector of the regularized target image as the initial value of the latent vector of the adversarial sample, construct the adversarial sample similarity objective function, and iteratively update the latent vector of the adversarial sample using the latent vector of the regularized initial input image in the latent space to obtain the latent vector of the adversarial sample to be generated; The constructed adversarial sample similarity objective function is specifically: The Euclidean distance is used to measure the similarity between two vectors. The goal is to minimize the Euclidean distance between the latent vector of the regularized initial input image and the latent vector of the adversarial sample while making the adversarial sample predicted as the target category. The following adversarial sample similarity objective function is constructed: in, represents the latent vector of the initial input image after regularization and the latent vector of the adversarial sample The Euclidean distance between the hidden vectors of the adversarial samples The initial value is the latent vector of the target image ;constraint Indicates that the adversarial sample is predicted to be the target category; In the latent space, the latent vector of the regularized initial input image is used to iteratively update the latent vector of the adversarial sample to obtain the latent vector of the adversarial sample to be generated. The specific method is: S4.1: In the tth iteration, the latent vector of the initial input image after regularization is used , in the latent space, the latent vector of the adversarial sample is searched by binary search Perform boundary search to obtain the hidden vector of the adversarial sample that reaches the boundary at the tth iteration , and calculate the perturbation size of the tth iteration ; S4.2: Use the Monte Carlo method to estimate the gradient direction and iteratively update the step size of the tth iteration along the gradient direction ; S4.3: According to the perturbation size of the tth iteration and the step size of the tth iteration Update the hidden vector of the adversarial sample that reaches the boundary at the tth iteration , obtain the hidden vector of the adversarial sample after the tth iteration update , and calculate the hidden vector of the adversarial sample after the t-th iteration update and the latent vector of the initial input image after regularization Euclidean distance between ; S4.4: Determine whether the number of iterations t is less than a preset threshold T. If so, repeat steps S4.1 to S4.

3. Otherwise, output the latent vector of the updated adversarial example with the smallest Euclidean distance as the latent vector of the adversarial example to be generated. S5: Input the latent vector of the adversarial sample to be generated into the optimized GAN neural network model to obtain the adversarial sample.

2. A non-restrictive adversarial sample generation method based on GAN according to claim 1, characterized in that: In step S2, the preset GAN neural network model is specifically a StyleGAN2 neural network model, which includes a generator network G, a discriminator network D and a pSp encoder.

3. The non-restrictive adversarial sample generation method based on GAN according to claim 1, characterized in that: In the adversarial sample similarity objective function, is a Boolean function, specifically: in, Indicates that the adversarial sample is predicted to be the target category, Indicates that the adversarial sample is not predicted as the target category; It indicates the function of using the preset classification model to determine whether the adversarial sample is predicted as the target category. , it means that the adversarial sample is predicted as the target category, otherwise it means that the adversarial sample is not predicted as the target category. The function is specifically: in, Indicates the target category represented by the target image, It is the probability output of the preset classification model for the category label c.

4. The non-restrictive adversarial sample generation method based on GAN according to claim 3, characterized in that: In step S4.1, the latent vector of the regularized initial input image is used , in the latent space, the latent vector of the adversarial sample is searched by binary search Perform boundary search to obtain the hidden vector of the adversarial sample that reaches the boundary at the tth iteration , and calculate the perturbation size of the tth iteration , the specific method is: Using the latent vector of the regularized initial input image , in the latent space, the latent vector of the adversarial sample is searched by binary search Perform boundary search to obtain the hidden vector of the adversarial sample that reaches the boundary at the tth iteration , the specific formula is: in, For binary search, is the parameter for binary search, Yes Obtained after updating the gradient direction; Calculate the perturbation size at the tth iteration The specific formula is: in, is the size of the initial input image after preprocessing.

5. The non-restrictive adversarial sample generation method based on GAN according to claim 4, characterized in that: In step S4.2, the Monte Carlo method is used to estimate the gradient direction, and the step size of the tth iteration is iteratively updated along the gradient direction. , the specific method is: The Monte Carlo method is used to estimate the gradient direction. The specific formula is: in, Indicates that at the boundary Find the gradient, is a uniform distribution in the d-dimensional mapping space, To control the baseline of the deviation boundary error, it is specifically expressed as: ; Iteratively update the step size of the tth iteration along the gradient direction , the specific method is: Iteratively update the step size of the tth iteration along the gradient direction , decreases at a geometric rate until it crosses the boundary, and the step size of the tth iteration is The initial value is .

6. The non-restrictive adversarial sample generation method based on GAN according to claim 5, characterized in that: In step S4.3, according to the disturbance size of the tth iteration and the step size of the tth iteration Update the hidden vector of the adversarial sample that reaches the boundary at the tth iteration , obtain the hidden vector of the adversarial sample after the tth iteration update , and calculate the hidden vector of the adversarial sample after the t-th iteration update and the latent vector of the initial input image after regularization Euclidean distance between , the specific method is: According to the perturbation size of the tth iteration and the step size of the tth iteration Update the hidden vector of the adversarial sample that reaches the boundary at the tth iteration , obtain the hidden vector of the adversarial sample after the tth iteration update , the specific formula is: in, is the cumulative gradient of the t-th iteration, =0; for The attenuation factor, Indicates the step size of the tth iteration; Calculate the hidden vector of the adversarial sample after the tth iteration update and the latent vector of the initial input image after regularization Euclidean distance between , the specific formula is: in, is the hidden vector of the adversarial sample after the t-th iteration update and the latent vector of the initial input image after regularization The Euclidean distance between .

7. A non-restrictive adversarial sample generation method based on GAN according to claim 1 or 6, characterized in that: In step S5, the latent vector of the adversarial sample to be generated is input into the optimized GAN neural network model to obtain the adversarial sample. The specific method is: The latent vector of the adversarial sample to be generated is input into the generator network G in the optimized GAN neural network model to generate the adversarial sample. The objective function of the generator network G is specifically: in, Represents the generator network G using the hidden vector of the adversarial sample to be generated Generate adversarial examples.

8. A non-restrictive adversarial sample generation system based on GAN, applying the non-restrictive adversarial sample generation method based on GAN according to any one of claims 1 to 7, characterized in that: include: Initialization unit: used to obtain an initial input image and a target image, and preprocess the initial input image to obtain a preprocessed initial input image; Model training unit: used to input the preprocessed initial input image into the preset GAN neural network model for training to obtain an optimized GAN neural network model; Latent vector encoding unit: used to encode the preprocessed initial input image and target image into the latent space using the encoder, obtain the latent vector of the initial input image and the latent vector of the target image, and regularize the latent vector of the initial input image and the latent vector of the target image; Latent vector iteration unit: used to use the latent vector of the regularized target image as the initial value of the latent vector of the adversarial sample, construct the adversarial sample similarity objective function, and iteratively update the latent vector of the adversarial sample using the latent vector of the regularized initial input image in the latent space to obtain the latent vector of the adversarial sample to be generated; Adversarial sample generation unit: used to input the latent vector of the adversarial sample to be generated into the optimized GAN neural network model to obtain the adversarial sample.

Images