A content-based adversarial patch camouflage generation method

Abstract

This invention discloses a method for generating adversarial patch camouflage based on content features. This method utilizes high-dimensional feature vectors representing content features extracted from images by a neural network. It innovatively uses feature vector distance to constrain the generation of adversarial patches. This allows adversarial patches to generate content close to the reference image, thus enabling them to be disguised as arbitrary image content, enhancing their concealment. Compared to existing camouflage methods that use pixel distance to constrain adversarial patch generation, the adversarial patches generated by this method appear more natural to the human senses and do not negatively impact the original adversarial effect of the adversarial patches.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and relates to a method for generating an adversarial patch camouflage based on content features. Background Art

[0002] With the continuous development of artificial intelligence technology, machine vision is widely used in modern intelligent systems, such as smart robots and self-driving cars. Machine vision uses cameras to capture information about the environment surrounding the intelligent system and employs deep learning algorithms to detect and identify objects within the images, thereby achieving environmental perception. Because machine vision perception results often serve as a source of information for subsequent decision-making in intelligent systems, the security of these results is paramount.

[0003] In recent years, the research on image adversarial samples has been increasing. Image adversarial samples refer to samples that can interfere with the results of machine vision perception. The study of image adversarial samples has important guiding significance for ensuring the security of machine systems and intelligent systems. At present, the research on image adversarial samples is mainly concentrated in the digital domain, that is, directly modifying the pixel values ​​of digital images to construct image adversarial samples. Its implementation in the physical domain is mainly in the form of patches, which have a universal adversarial effect. Although the image adversarial patches constructed by this method usually have a good adversarial effect, the generated patterns are often eye-catching and weird, which makes it difficult for previous adversarial patches to be concealed. Therefore, the present invention proposes a content-based adversarial patch camouflage generation method, which can make the generated adversarial patches have familiar patterns, characters and content, enhance the practicality and concealment of the adversarial patches, provide new ideas for the construction of image adversarial samples, and provide new guidance for machine learning security analysis and protection. Summary of the Invention

[0004] The present invention provides a content-based feature-based adversarial patch camouflage generation method. This method uses the distance of high-dimensional feature vectors representing image content as a loss function to constrain the generation of adversarial patches, making the generated adversarial patches look close to any camouflaged reference image, thereby enhancing the naturalness and concealment of the adversarial patches.

[0005] The present invention is achieved by adopting the following technical solutions:

[0006] A method for generating adversarial patch camouflage based on content features includes the following steps:

[0007] 1) Design an adversarial loss function for the target neural network, which is used to optimize the adversarial patch. This step specifically includes the following two steps:

[0008] (1) Determine the design size and adversarial target of the adversarial patch and initialize the adversarial patch with random pixel values;

[0009] (2) Use the adversarial target in step (1) to replace the label term in the training loss function of the target neural network to obtain the adversarial loss function L adv The adversarial loss function should be determined according to the target neural network. For example, for an image classifier, its adversarial loss function can be expressed as:

[0010] L adv =L CE (p, t) = -log(pt)

[0011] Among them, L CE represents the cross entropy loss, p represents the confidence vector output by the classifier, t represents the target category, and p t Represents the confidence of the output t-th category.

[0012] 2) Using a convolutional neural network model pre-trained on a public image dataset, perform high-dimensional feature extraction on a camouflaged reference image to extract a high-dimensional feature vector. This step specifically includes the following two steps:

[0013] (1) Determine a reference image for camouflage and resize it to the legal input size of the pre-trained convolutional neural network model;

[0014] (2) Input the reference image in step (1) into the pre-trained convolutional neural network model. After several layers of convolution feature extraction, a high-dimensional feature vector β1 is obtained and saved. The extraction process is expressed as:

[0015] β1=Φ j (x1)

[0016] Among them, Φ j represents the j-th convolution operation of the pre-trained convolutional neural network model, x1 represents the reference image, and β1 represents the feature vector extracted from the reference image.

[0017] 3) In the adversarial patch generation phase, the smaller adversarial patch is upsampled and then fed into the convolutional neural network model described in step 2) to extract a feature vector of the same dimension as in step 2). This step specifically includes the following two steps:

[0018] (1) Upsample the adversarial patch and adjust it to the same size as the resized reference image in step 2) to obtain a large-size adversarial patch;

[0019] (2) The large-size adversarial patch in step (1) is input into the pre-trained convolutional neural network model. After the convolution feature extraction with the same number of layers as in step 2), a high-dimensional feature vector is obtained and saved. The extraction process is expressed as:

[0020] β2=Φ j (x2)

[0021] Among them, Φ j represents the j-th convolution operation of the pre-trained convolutional neural network model, x2 represents the large-size adversarial patch, and β2 represents the feature vector extracted from the large-size adversarial patch.

[0022] 4) Calculate the Euclidean distance between the two feature vectors extracted in steps 2) and 3) respectively, use it as the content loss function, and add it to the adversarial loss function designed in step 1) with a certain weight to form a joint loss function. This step specifically includes the following two steps:

[0023] (1) Based on the high-dimensional feature vector β1 of the reference image obtained in step 2) and the high-dimensional feature vector β2 of the adversarial patch obtained in step 3), the Euclidean distance δ between the two feature vectors is calculated and used as a loss function;

[0024] δ=||β1-β2||2

[0025] (2) Add the loss function δ obtained in step (1) with a certain weight α to the adversarial loss function L designed in step 1) adv In the above example, the two constitute a joint loss function L all .

[0026] L all =L adv +α*δ

[0027] 5) Based on the joint loss function obtained in step 4), a gradient optimization method is used to optimize the generated adversarial patch. This step specifically includes the following five steps:

[0028] (1) Randomly select a background image from the standard image dataset, place the adversarial patch at a random position in the background image, and use the adversarial patch to replace the original pixels in the background image;

[0029] (2) Input the image with the adversarial patch in step (1) into the target neural network to obtain an output vector representing the recognition result;

[0030] (3) Calculate the joint loss value of the output vector obtained in step (2) according to the joint loss function obtained in step 4);

[0031] (4) Back-propagate the joint loss value obtained in step (3) to the adversarial patch to form a gradient, and use optimizers such as SGD and Adam to optimize the adversarial patch.

[0032] (5) Repeat steps (1)-(4) until the adversarial patch converges to the desired effect.

[0033] The beneficial effects of the present invention are:

[0034] The adversarial patch camouflage method proposed in the present invention utilizes high-dimensional content feature vectors extracted from images by a neural network, and innovatively proposes using feature vector distance to constrain the generation of adversarial patches, which can make the image content of the adversarial patch close to any reference image. Therefore, selecting a suitable reference image can make the camouflaged adversarial patch more natural to the human eye and have stronger concealment, providing new ideas for the construction of image adversarial samples and new guidance for machine learning security analysis and protection.

[0035] When generating adversarial patches, constraints based on content feature distances produce better camouflage results than conventional pixel value constraints. This is because conventional pixel value constraints fail to reflect the focus on key features of the reference image, leading to suboptimal effects such as excessive focus on the background. However, constraints based on the distance between content features extracted by a neural network focus more on foreground objects, ensuring that the adversarial patch resembles the reference image in key features, which is more consistent with the human eye's perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a method for calculating content loss according to an embodiment of the present invention.

[0037] Figure 2 This is a generation effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to facilitate ordinary technicians in this field to understand and implement this invention, the present invention is further elaborated in detail below with reference to the accompanying drawings and embodiments. It should be understood that the implementation examples described here are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0039] The content loss calculation method of the embodiment of the present invention is as follows: Figure 1 shown.

[0040] The generation effect of the embodiment of the present invention is as follows: Figure 2 shown.

[0041] The content feature-resistant patch camouflage generation method of the present invention comprises the following steps:

[0042] 1) Design an adversarial loss function for the target neural network to counter the patch. This step specifically includes the following two steps:

[0043] (1) Determine the design size and adversarial target of the adversarial patch. In this embodiment, the adversarial target is a traffic sign, and use random pixel values ​​to initialize the adversarial patch.

[0044] (2) Use the adversarial target in step (1) to replace the label term in the training loss function of the target neural network to obtain the adversarial loss function L adv The adversarial loss function should be determined according to the target neural network. In the embodiment, the target neural network is an image classifier, and its adversarial loss function can be expressed as:

[0045] L adv =L CE (p, t) = -log(p t )

[0046] Among them, L CE represents the cross entropy loss, p represents the confidence vector output by the classifier, t represents the target category, and p t Represents the confidence of the output t-th category.

[0047] 2) Use a convolutional neural network model pre-trained on a public image dataset to perform feature extraction on a camouflaged reference image, extracting a high-dimensional feature vector. This step specifically includes the following two steps:

[0048] (1) Determine a reference image for camouflage, here a rose sketch, and resize it to the legal input size of the pre-trained convolutional neural network model;

[0049] (2) Input the reference image in step (1) into the VGG convolutional neural network model pre-trained with the ImageNet dataset. After several layers of convolutional feature extraction, a high-dimensional feature vector β1 is obtained and saved. The extraction process is expressed as:

[0050] β1=Φ j (x1)

[0051] Among them, Φ j represents the j-th convolution operation of the pre-trained convolutional neural network model, x1 represents the reference image, and β1 represents the feature vector extracted from the reference image.

[0052] 3) In the adversarial patch generation phase, the smaller adversarial patch is upsampled and then fed into the convolutional neural network model in step 2) to extract a feature vector of the same dimension as in step 2). This step specifically includes the following two steps:

[0053] (1) Upsample the adversarial patch and adjust it to the same size as the resized reference image in step 2) to obtain a large-size adversarial patch;

[0054] (2) Input the large-size adversarial patch in step (1) into the pre-trained convolutional neural network model. After the convolution feature extraction with the same number of layers as in step 2), a feature vector is obtained and saved. The extraction process is expressed as:

[0055] β2=Φ j (x2)

[0056] Among them, Φ j represents the j-th convolution operation of the pre-trained convolutional neural network, x2 represents the large-size adversarial patch, and β2 represents the feature vector extracted from the large-size adversarial patch.

[0057] 4) Calculate the Euclidean distance between the two feature vectors extracted in steps 2) and 3) respectively, use it as the content loss function, and add it to the adversarial loss function designed in step 1) with a certain weight to form a joint loss function. This step specifically includes the following two steps:

[0058] (1) Based on the feature vector β1 of the reference image obtained in step 2) and the feature vector β2 of the adversarial patch obtained in step 3), the Euclidean distance δ between the two feature vectors is calculated and used as a loss function;

[0059] δ=||β1-β2||2

[0060] (2) Add the loss function δ obtained in step (1) with a certain weight α to the adversarial loss function L designed in step 1) adv In the above example, the two constitute a joint loss function L all .

[0061] L all =L adv +α*δ

[0062] 5) Based on the joint loss function obtained in step 4), a gradient optimization method is used to optimize the generated adversarial patch. This step specifically includes the following five steps:

[0063] (1) Randomly select a background image from the standard image dataset ImageNet, place the adversarial patch at a random position in the background image, and use the adversarial patch to replace the original pixels in the background image;

[0064] (2) Input the image with the adversarial patch in step (1) into the target neural network model to obtain an output vector representing the recognition result;

[0065] (3) Calculate the joint loss value of the output vector obtained in step (2) according to the joint loss function obtained in step 4);

[0066] (4) Back-propagate the joint loss value obtained in step (3) to the adversarial patch to form a gradient, and use optimizers such as SGD and Adam to optimize the adversarial patch.

[0067] (5) Repeat steps (1)-(4) until the adversarial patch converges to the desired effect.

[0068] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0069] It should be understood that the above description of the embodiments is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary persons in this field can make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which shall fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A content-based adversarial patch camouflage generation method, characterized in that: The method comprises the following steps: 1) Designing an adversarial loss function for the target neural network, wherein the adversarial loss function is used to optimize the adversarial patch; 2) Using a convolutional neural network model pre-trained on a public image dataset, perform high-dimensional feature extraction on a camouflaged reference image to extract a high-dimensional feature vector. The specific process is as follows: 21) Determine a camouflaged reference image and resize it to the legal input size of the pre-trained convolutional neural network model; 22) Input the reference image in step 21) into the pre-trained convolutional neural network model. After several layers of convolution feature extraction, a high-dimensional feature vector is obtained. And save, the extraction process is expressed as: ， in, Represents the first layer of the pre-trained convolutional neural network model Layer convolution operation, represents the reference image, represents the feature vector extracted from the reference image; 3) After upsampling, the adversarial patch is input into the convolutional neural network model described in step 2) to extract the feature vector of the same dimension as in step 2). The specific process is as follows: 31) Upsample the adversarial patch and adjust it to the same size as the resized reference image in step 2) to obtain a large-size adversarial patch; 32) Input the large-size adversarial patch from step 31) into the pre-trained convolutional neural network model. After the same number of convolutional feature extraction layers as in step 2), a high-dimensional feature vector is obtained and saved. The extraction process is expressed as: ， in, Represents the first layer of the pre-trained convolutional neural network model Layer convolution operation, represents a large-size adversarial patch, represents the feature vector extracted from the large-sized adversarial patch; 4) Calculate the Euclidean distance between the two feature vectors extracted in steps 2) and 3) respectively, use it as the content loss function, and add it to the adversarial loss function designed in step 1) with a certain weight to form a joint loss function. The specific process is as follows: 41) Based on the high-dimensional feature vector of the reference image obtained in step 2) The high-dimensional feature vector of the adversarial patch obtained in step 3) , calculate the Euclidean distance between two eigenvectors , using it as a loss function: ； 42) The loss function obtained in step 41) With a certain weight Add the adversarial loss function designed in step 1) In the above example, the two constitute a joint loss function : ； 5) Based on the joint loss function obtained in step 4), a gradient optimization method is used to optimize the generated adversarial patch.

2. The content feature-based anti-patch disguise generation method according to claim 1, characterized in that: Step 1) specifically includes the following steps: (1) Determine the design size and adversarial target of the adversarial patch and initialize the adversarial patch with random pixel values; (2) Use the adversarial target in step (1) to replace the label term in the training loss function of the target neural network to obtain the adversarial loss function .

3. The content feature-based anti-patch disguise generation method according to claim 1, characterized in that: Step 5) includes the following steps: (1) Randomly select a background image from a standard image dataset and place the adversarial patch at a random position in the background image; (2) Input the image with the adversarial patch in step (1) into the target neural network to obtain an output vector representing the recognition result; (3) Calculate the joint loss value of the output vector obtained in step (2) according to the joint loss function obtained in step 4); (4) Back-propagate the joint loss value obtained in step (3) to the adversarial patch to form a gradient and optimize the adversarial patch; (5) Repeat steps (1)-(4) until the adversarial patch converges to the desired effect.

Images