Method and system for generating confrontation camouflage pattern based on stable diffusion model

Abstract

The invention relates to the technical field of target detection confrontation, and discloses a confrontation camouflage pattern generation method and system based on a stable diffusion model.The method comprises the steps that an encoder is used for encoding a preset reference camouflage picture to obtain original hidden space representation, the original hidden space representation is gradually diffused and gradually denoised, and target hidden space representation is obtained; decoding is carried out to obtain an intermediate adversarial patch, and meanwhile, the Euclidean distance is obtained based on the self-attention score graph; adding the intermediate adversarial patch into the data set to obtain a confidence score of the detection model; and adding the Euclidean distance and the confidence score to obtain a target loss function, correcting original hidden space Gaussian noise, substituting the steps for repeated iteration, taking the intermediate adversarial patch corresponding to the last iteration as a target adversarial patch, and ensuring the semantic consistency of the adversarial camouflage pattern and the original reference camouflage picture. The problem that reference image semantics are damaged in the adversarial patch generation process is avoided, and the adversarial patch keeps visual naturalness and practicability.

Description

Technical Field

[0001] The present invention relates to the field of target detection confrontation technology, and in particular to a method and system for generating a confrontation camouflage pattern based on a stable diffusion model. Background Art

[0002] With the widespread application of deep neural networks in fields such as autonomous driving and intelligent security, their shortcomings of being susceptible to adversarial noise interference have gradually become apparent, affecting their reliability in actual deployment. In order to improve the robustness of the detection model, exploring the security vulnerabilities of existing deep networks is of great significance for subsequent model design and robustness enhancement.

[0003] Adversarial camouflage, as a technology that enables targets to evade detection model recognition, can maximize the exploration of blind spots of existing models and also has significant application value in privacy protection.

[0004] However, most existing adversarial camouflage pattern generation techniques do not consider color constraints. The generated adversarial patches are usually brightly colored, easily detected by the human eye, and lack practical concealment. In order to ensure the naturalness of the adversarial patches, existing methods usually directly modify the "reference camouflage image" in the RGB pixel space to generate adversarial patches. However, when trying to enhance the deceptiveness of the adversarial patches, this method often significantly changes the semantic information of the "reference camouflage image", which seriously damages the naturalness, practicality, and adversarial ability of the adversarial patches. Summary of the invention

[0005] In view of this, the present invention provides a method and system for generating an adversarial camouflage pattern based on a stable diffusion model to solve the problem of how to improve the naturalness, practicality and adversarial ability of adversarial patches.

[0006] In a first aspect, the present invention provides a method for generating an anti-camouflage pattern based on a stable diffusion model, the method comprising:

[0007] Step S101, obtaining a data set, a detection model, a stable diffusion model, and a preset reference camouflage picture corresponding to a target detector;

[0008] Step S102, encoding the preset reference disguised picture using an encoder to obtain an original latent space representation; wherein the preset reference disguised picture represents an initial adversarial patch;

[0009] Step S103, using the DDIM inverse process to gradually diffuse the original latent space representation into the original latent space Gaussian noise, wherein each step of the gradual diffusion process generates a corresponding first intermediate representation;

[0010] Step S104, based on the stable diffusion model, the original latent space Gaussian noise is gradually denoised to obtain a target latent space representation; wherein each step of the gradual denoising process generates a corresponding second intermediate representation;

[0011] Step S105, decoding the target latent space representation to obtain an intermediate adversarial patch, and at the same time, obtaining a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation, and obtaining a Euclidean distance between the first self-attention score map and the second self-attention score map; wherein the intermediate adversarial patch represents an image of the same dimension as the preset reference disguised image, and belongs to an adversarial patch that appears in each repetition process;

[0012] Step S106, adding the intermediate adversarial patch to the data set as input data of the detection model, and obtaining a confidence score of the detection model;

[0013] Step S107, adding the Euclidean distance and the confidence score to obtain a target loss function; when the target loss function reaches a preset standard value, the corresponding intermediate adversarial patch in step S105 is used as the target adversarial patch, and when the target loss function does not reach the preset standard value, step S108 is executed;

[0014] Step S108, perform gradient calculation based on the target loss function to correct the original latent space Gaussian noise, substitute the corrected original latent space Gaussian noise into step S104 and continue to repeat subsequent steps until the number of repetitions reaches a preset number of iterations, and use the intermediate adversarial patch in step S105 corresponding to the last iteration as the target adversarial patch, which is used to counter the target detector.

[0015] The embodiment of the present invention encodes a preset reference camouflage picture, diffuses it to Gaussian noise using the DDIM inverse process of the stable diffusion model, and then gradually denoises to generate a target latent space representation, so that the generated adversarial camouflage pattern is more natural visually, avoiding the disadvantages of traditional adversarial patches being too bright and conspicuous. Compared with the situation in which the patches generated in the prior art method are abrupt in color and easily perceived by the human eye, the adversarial camouflage pattern is more concealed in the real environment through the gradual optimization of the diffusion model; by generating a first self-attention score map and a second self-attention score map, the Euclidean distance is calculated to ensure a high degree of semantic consistency between the adversarial camouflage pattern and the original reference camouflage picture, avoiding the problem of destroying the semantic information of the reference image in the adversarial patch generation process in the prior art, and using the Euclidean distance of the self-attention score map as the loss The generated intermediate adversarial patch is added to the data set as the input of the detection model to obtain the confidence score of the detection model, and then the Euclidean distance is added to the confidence score to obtain the target loss function, so that the adversarial camouflage pattern can minimize the detection confidence of the target detection model, which solves the problem that the adversarial patch in the prior art is easily recognized by the detection model, improves the aggressiveness and concealment of the adversarial camouflage pattern, and through multiple rounds of iterations, adjusts the Gaussian noise in the latent space and optimizes the loss function. The generated adversarial patch maintains the naturalness of the vision, and maintains the consistency of the image semantics under the optimization of the self-attention score map. The concealment of the finally generated adversarial patch in the physical world is significantly enhanced.

[0016] In an optional implementation, the step S103, using the DDIM inverse process to gradually diffuse the original latent space representation into the original latent space Gaussian noise, specifically includes:

[0017] Use the inverse process of DDIM to add noise to the original latent space representation x0 T times to obtain the original latent space Gaussian noise x T , where each noise addition generates the corresponding first intermediate representation x t , the first intermediate representation x corresponding to the t+1th noise addition process t+1 The generation of depends on the first intermediate representation x corresponding to the t-th noise addition process t .

[0018] The embodiment of the present invention gradually converts the original latent space representation into Gaussian noise by adding noise T times, generates a first intermediate representation at each step, and the generation of the intermediate representation at each step depends on the intermediate representation at the previous step. The structural information of the original latent space representation is better maintained by a step-by-step diffusion method, and the DDIM inverse process is introduced to carefully modify the reference camouflage image in the latent space, which can ensure semantic consistency and ensure that the finally generated adversarial patch maintains a high degree of naturalness.

[0019] In an optional implementation, the DDIM inverse process is used to perform T noise additions on the original latent space representation x0 to obtain the original latent space Gaussian noise x T , where each noise addition generates the corresponding first intermediate representation x t , including:

[0020] Obtain a first calculation formula corresponding to the DDIM original process, where the first calculation formula is:

[0021]

[0022] in, α s Represents the mean coefficient of the normal distribution in each denoising process, ∈ θ (x t ,t) represents the residual predicted by the stable diffusion model at time t; σ t represents the variance of the normal distribution during the denoising process at time t, which is set to 0; z is random noise, which obeys the normal distribution with a mean of 0 and a variance of 1;

[0023] Determine a first intermediate representation x based on the first calculation formula t-1 About the first intermediate representation x t The denoised difference term is obtained by a second calculation formula:

[0024]

[0025] Reverse the second calculation formula to obtain t Generate x t+1 The noise-added difference term is expressed by a third calculation formula, and the third calculation formula is:

[0026]

[0027] Determine x based on the third calculation formula t Generate x t+1 The one-time noise adding process corresponding to the time is represented by the following mathematical formula:

[0028] x t+1 =Inverse(x t );

[0029] According to each noise addition process, the original latent space Gaussian noise x is obtained by performing T noise additions from the original latent space representation x0. T The fourth calculation formula is:

[0030]

[0031] The embodiment of the present invention further refines the generation process of the original latent space Gaussian noise. Through multiple calculation formulas, the noise addition is reversed, and the noise addition in each step is more accurately controlled to ensure the generation accuracy of Gaussian noise in each step. The continuous noise addition process also avoids sudden changes in the noise addition process, further improving the semantic consistency and naturalness of the final adversarial patch.

[0032] In an optional implementation, the first calculation formula is also used to characterize any denoising process in the whole process of denoising the original latent space Gaussian noise through the stable diffusion model;

[0033] The step S104, based on the stable diffusion model, gradually denoises the original latent space Gaussian noise to obtain the target latent space representation, specifically includes:

[0034] Determine x based on the first calculation formula t+1 Generate x t The first denoising process corresponding to the time is represented by the following mathematical formula:

[0035] x t =Denoise(x t+1 );

[0036] According to each denoising process, the Gaussian noise x T Perform T denoising to get the target latent space representation x ′ 0, the fifth calculation formula is:

[0037]

[0038] The embodiment of the present invention characterizes each denoising process through the first calculation formula, and determines the target latent space representation obtained by denoising the original latent space Gaussian noise T times through the fifth calculation formula, thereby ensuring that the generated target latent space representation is more in line with the expected goal, further improving the naturalness of the final adversarial patch, and effectively increasing the concealment in the physical world. It solves the dual problems of practicality and naturalness of adversarial patches in the prior art, and maintains effective deception to target detectors.

[0039] In an optional implementation, the first self-attention score map is represented by the following mathematical formula:

[0040]

[0041] The second self-attention score map is represented by the following mathematical formula:

[0042]

[0043] Where N represents the size of the self-attention score map;

[0044] The step S105, decoding the target latent space representation to obtain an intermediate adversarial patch, and at the same time, obtaining a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation, and obtaining a Euclidean distance between the first self-attention score map and the second self-attention score map based on the first self-attention score map and the second self-attention score map, specifically includes:

[0045] The target latent space representation x is represented by using a decoder corresponding to the stable diffusion model ′ 0 is decoded to obtain the intermediate adversarial patch x ′ ;

[0046] At the same time, a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation are obtained, and a Euclidean distance between the first self-attention score map and the second self-attention score map is obtained based on the first self-attention score map and the second self-attention score map, wherein the Euclidean distance between the two constitutes the first part of the target loss function It is expressed by the sixth calculation formula:

[0047]

[0048] The embodiment of the present invention obtains the Euclidean distance by calculating the first self-attention score map and the second self-attention score map, so that the generated intermediate adversarial patch is highly consistent with the original reference camouflage image in semantics. By calculating the Euclidean distance between the two, the similarity between the adversarial patch pattern and the reference original image at the semantic level is quantified as part of the target loss function, which can effectively reduce the detection confidence of the target detector, making the generated camouflage pattern more aggressive and more difficult to be recognized by the detection system.

[0049] In an optional implementation, the step S106, adding the intermediate adversarial patch to the data set as input data of the detection model, and obtaining a confidence score of the detection model, specifically includes:

[0050] According to the location information of the target object P in each picture of the training set and the size ratio of the preset expected adversarial patch, the intermediate adversarial patch x is transformed using the affine transformation matrix. ′ Rotate and scale to generate image I with intermediate adversarial patches M ;

[0051] The picture I M With the image I in the training set r Fusion, get adversarial input data I A , as input data of the detection model;

[0052] Input the input data into the detection model to obtain the detection confidence score F of the detection model about the target object P obj F(I A ).

[0053] The embodiment of the present invention performs an affine transformation on the intermediate adversarial patch so that the size and angle of the adversarial patch better match the position and size of the target object in the training set, and generates more accurate adversarial input data by fusing it with the image in the training set, thereby improving the visual naturalness and authenticity of the camouflage. By inputting the fused image into the detection model, it is ensured that the generated adversarial camouflage pattern can reduce the confidence of the target detector under various conditions, further improving the aggressiveness and stability of the adversarial camouflage pattern, and having a higher anti-interference ability.

[0054] In an optional embodiment, the intermediate adversarial patch x is transformed using an affine transformation matrix according to the location information of the target object P in each picture of the training set and the size ratio of the preset desired adversarial patch. ′ Rotate and scale to generate image I with intermediate adversarial patches M , expressed by the seventh calculation formula:

[0055] I M =S*R(θ)*x ′ ;

[0056] Among them, R(θ) represents the rotation matrix, which is used to rotate the adversarial patch and is expressed by the following mathematical formula:

[0057]

[0058] S represents the scaling matrix, which is used to scale the adversarial patch and is expressed by the following mathematical formula:

[0059]

[0060] S x and S y Respectively represent the scaling factors corresponding to the x and y axes.

[0061] The embodiment of the present invention generates an image with an intermediate adversarial patch through a rotation matrix and a scaling matrix, which can still maintain visual naturalness at various viewing angles and distances, and is suitable for targets in actual environments, and can obviously mislead target detectors in different target scenes.

[0062] In an optional implementation, the image I M With the image I in the training set r Fusion, get adversarial input data I A , as the input data of the detection model, is represented by the eighth calculation formula:

[0063]

[0064] The embodiment of the present invention fuses the generated camouflage picture with the picture in the training set to construct adversarial input data as the input of the detection model, so that the camouflage pattern is more integrated with the original image data in the training set, thereby reducing the recognition capability of the target detector.

[0065] In an optional implementation, the detection confidence score F obj F(I A ) above the preset standard value constitutes the second part of the target loss function It is expressed by the ninth calculation formula:

[0066]

[0067] In step S107, the Euclidean distance is added to the confidence score to obtain a target loss function, which is represented by the tenth calculation formula:

[0068]

[0069] Among them, α and β represent the weight corresponding to the Euclidean distance and the weight corresponding to the confidence score, respectively.

[0070] The embodiment of the present invention controls the composition of the target loss function more flexibly by assigning different weights to the Euclidean distance and the confidence score of the detection model respectively, and the loss function includes the Euclidean distance and the confidence score. By minimizing the Euclidean distance between the self-attention score map and the original self-attention score map in the process of generating the adversarial patch, and the confidence score of the pedestrian detection model, it is ensured that the generated adversarial patch can minimize the detection confidence of the model and maintain the visual naturalness of the image. The overall loss function composed of the Euclidean distance and the confidence score makes the adversarial patch more aggressive and concealed.

[0071] In a second aspect, the present invention provides a system for generating an anti-camouflage pattern based on a stable diffusion model, the device comprising:

[0072] Data preparation and model loading module, used to obtain the data set, detection model, stable diffusion model and preset reference camouflage picture corresponding to the target detector;

[0073] A latent space encoding module, configured to encode the preset reference disguised image using an encoder to obtain an original latent space representation; wherein the preset reference disguised image represents an initial adversarial patch;

[0074] A diffusion process simulation module, used for gradually diffusing the original latent space representation into the original latent space Gaussian noise by using the DDIM inverse process, wherein each step of the gradual diffusion process generates a corresponding first intermediate representation;

[0075] A denoising and reconstruction module, configured to perform step-by-step denoising on the original latent space Gaussian noise based on the stable diffusion model to obtain a target latent space representation; wherein each step of the step-by-step denoising process generates a corresponding second intermediate representation;

[0076] an adversarial patch generation and distance calculation module, configured to decode the target latent space representation to obtain an intermediate adversarial patch, and at the same time, obtain a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation, and obtain a Euclidean distance between the first self-attention score map and the second self-attention score map; wherein the intermediate adversarial patch represents an image of the same dimension as the preset reference disguised image, and belongs to an adversarial patch that appears in each repetition process;

[0077] an adversarial patch evaluation module, configured to add the intermediate adversarial patch to the data set as input data of the detection model and obtain a confidence score of the detection model;

[0078] A target loss calculation module, configured to add the Euclidean distance to the confidence score to obtain a target loss function; when the target loss function reaches a preset standard value, the corresponding intermediate adversarial patch in step S105 is used as a target adversarial patch; when the target loss function does not reach the preset standard value, step S108 is executed;

[0079] A gradient optimization module is used to perform gradient calculation based on the target loss function to correct the original latent space Gaussian noise, and to continue to repeat subsequent steps using the corrected original latent space Gaussian noise until the number of repetitions reaches a preset number of iterations, and to use the intermediate adversarial patch corresponding to the last iteration as the target adversarial patch, which is used to counter the target detector.

[0080] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the method for generating an anti-camouflage pattern based on a stable diffusion model according to the first aspect or any corresponding embodiment thereof.

[0081] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for generating an anti-camouflage pattern based on a stable diffusion model according to the first aspect or any corresponding embodiment thereof.

[0082] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the method for generating an anti-camouflage pattern based on a stable diffusion model according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0084] Figure 1 is a schematic flow chart of a method for generating an anti-camouflage pattern based on a stable diffusion model according to an embodiment of the present invention;

[0085] Figure 2 is a flow chart of another method for generating an anti-camouflage pattern based on a stable diffusion model according to an embodiment of the present invention;

[0086] Figure 3 is a schematic diagram of an adversarial patch of another adversarial camouflage pattern generation method based on a stable diffusion model according to an embodiment of the present invention;

[0087] Figure 4 is a schematic diagram of detection results of a target detector according to a method for generating an anti-camouflage pattern based on a stable diffusion model according to an embodiment of the present invention;

[0088] Figure 5 is a structural block diagram of a system for generating an anti-camouflage pattern based on a stable diffusion model according to an embodiment of the present invention;

[0089] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0091] With the widespread application of deep neural network detection models in autonomous driving and smart security, neural network detection models have been found to be vulnerable to adversarial camouflage, which allows the target to evade the recognition of the detection model.

[0092] Adversarial camouflage is a technology that allows evading the recognition of target detection models. It can maximize the exploration of blind spots of existing models and can be used for privacy protection, with obvious application value.

[0093] Existing adversarial camouflage pattern generation techniques usually do not consider the color restrictions of adversarial patches. The generated adversarial patches are usually very bright in color, easily detected by the human eye, and lack the concealment required in practical applications. For practical application scenarios, such as privacy protection or avoiding detection, overly obvious adversarial patches may attract unnecessary attention, thus affecting their practicality.

[0094] In order to improve the deceptiveness of adversarial patches, traditional methods will significantly modify the reference camouflage image in the RGB pixel space. Although this can improve the interference ability of adversarial patches on the detection model, it also destroys the semantic information of the reference image, causing the generated camouflage pattern to lose its original naturalness.

[0095] Traditional adversarial patch generation technology often focuses on how to deceive the detection model to the greatest extent, but ignores the sensitivity of the human visual system to color and shape. When the model generates adversarial camouflage patterns, although it can confuse the detection model, it ignores the human eye's ability to recognize colors, which makes the patch very conspicuous. In other words, if the color and shape of the adversarial patch are too conspicuous, its intention will be clearly exposed in actual application scenarios, thereby affecting the actual application effect of the patch.

[0096] Therefore, an embodiment of the present invention provides an adversarial camouflage pattern generation method based on a stable diffusion model, which is applied in public places, surveillance or scenes with target detectors, that is, individuals may want to avoid being recognized by automatic detection systems to protect their privacy. By attacking the latent space of the stable diffusion model, the generated adversarial camouflage pattern is more natural, avoiding the problem of overly bright colors of adversarial patches in traditional methods, improving the concealment and practicality in real environments, while also maintaining effective deception of target detection models. By minimizing the Euclidean distance between the self-attention score map and the original self-attention score map in the adversarial patch generation process, as well as the confidence score of the pedestrian detection model, it is ensured that the generated adversarial patch can minimize the detection confidence of the model while maintaining the visual naturalness of the image. The loss function makes the adversarial patch more aggressive and concealed.

[0097] According to an embodiment of the present invention, an embodiment of a method for generating an anti-camouflage pattern based on a stable diffusion model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0098] In this embodiment, a method for generating an anti-camouflage pattern based on a stable diffusion model is provided, which can be used in the above-mentioned computer. Figure 1 FIG. 1 is a flow chart of a method for generating an anti-camouflage pattern based on a stable diffusion model according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0099] Step S101, obtaining a data set, a detection model, a stable diffusion model, and a preset reference camouflage picture corresponding to a target detector;

[0100] It should be noted that the target detector refers to a detection model that can identify target objects in an image, and the data set refers to a data set corresponding to the target detector, which is used to train the target detector. The stable diffusion model refers to a model that converts random noise into a specific target image through a step-by-step denoising process. Preset reference camouflage image: an initial camouflage pattern, used as a starting point for generating adversarial camouflage patterns, such as some inconspicuous images used for adversarial recognition.

[0101] Specifically, if the target detector is a pedestrian recognition detector, this embodiment needs to attack the pedestrian target detector, and needs to collect a pedestrian data set, a pedestrian detection model, a stable diffusion model and a reference camouflage picture, including: (1) Download the INRIA pedestrian data set, which is divided into a training set and a test set. Among them, the training set is used as the base map for pasting adversarial patches to train adversarial patches, including 614 pictures. The test set is used to evaluate the attack effect of the adversarial patch, including 288 pictures, each of which is 416*416*3 in size. (2) Download the target detection model YOLOv5 pre-trained on the COCO data set to detect the INRIA pedestrian data training set after pasting the adversarial patch. (3) Download the stable diffusion model Stablediffusion model to encode the reference picture, decode the generated latent space representation, and gradually denoise the latent space Gaussian noise. (4) Collect a reference camouflage image x∈(0,1)^(W×H×3) as the initial adversarial patch, where W=288, H=288 are the width and height of the reference camouflage image, respectively, and 3 is the number of image channels. In the present invention, a green camouflage pattern is selected as the reference camouflage image.

[0102] Step S102, encoding the preset reference disguised picture using an encoder to obtain an original latent space representation; wherein the preset reference disguised picture represents an initial adversarial patch;

[0103] Step S103, using the DDIM inverse process to gradually diffuse the original latent space representation into the original latent space Gaussian noise, wherein each step of the gradual diffusion process generates a corresponding first intermediate representation;

[0104] Step S104, based on the stable diffusion model, the original latent space Gaussian noise is gradually denoised to obtain a target latent space representation; wherein each step of the gradual denoising process generates a corresponding second intermediate representation;

[0105] It should be noted that the encoder is used to convert the input image into a latent space representation. The latent space representation refers to the high-dimensional representation of the image in the latent space, which contains the feature information of the image. The adversarial patch refers to a camouflage pattern that is less visually conspicuous but can confuse the detector. The DDIM inverse process refers to a method for gradual diffusion or inverse diffusion of image representation. Gaussian noise refers to random noise with zero mean, which is used as the starting point for generating images in the diffusion model. The target latent space representation refers to the final latent space representation after denoising, which is used to generate the target camouflage pattern.

[0106] It can be understood that in step S104, the reference camouflage image is converted into a latent space representation through an encoder, the purpose of which is to compress the image information into the latent space to facilitate the subsequent generation of the target latent space. From step S103 to step S104, the original latent space representation is gradually diffused into Gaussian noise, and the Gaussian noise is gradually denoised using a stable diffusion model until the latent space representation of the target is obtained, which is to gradually generate an adversarial camouflage pattern in the latent space. Through multiple denoising, the generated adversarial image is guaranteed to be more natural.

[0107] Exemplarily, the encoder of the Stable diffusion model is used to encode the reference disguised image into the original latent space representation, and then the original latent space representation of the reference disguised image is gradually diffused into the original latent space Gaussian noise using the DDIM inverse process. At the same time, an intermediate representation belonging to each step will be generated in each diffusion process.

[0108] Step S105, decoding the target latent space representation to obtain an intermediate adversarial patch, and at the same time, obtaining a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation, and obtaining a Euclidean distance between the first self-attention score map and the second self-attention score map; wherein the intermediate adversarial patch represents an image of the same dimension as the preset reference disguised image, and belongs to an adversarial patch that appears in each repetition process;

[0109] It can be understood that the self-attention score map refers to a score map generated by the self-attention mechanism of the model, which indicates the model's attention to different areas in the image. This embodiment decodes the target latent space representation to obtain an intermediate adversarial patch. At the same time, a first self-attention score map (intermediate representation in the diffusion process) and a second self-attention score map (intermediate representation after denoising) are generated, and their Euclidean distance is calculated. By using this distance as part of the loss function, it is ensured that the adversarial patch is semantically consistent with the initial reference camouflage image.

[0110] Exemplarily, the latent space Gaussian noise is gradually denoised using the stable diffusion model to obtain the latent space representation, and then the decoder of the stable diffusion model is used to decode the obtained latent space representation into an image of the same dimension as the reference disguised image, that is, the intermediate adversarial patch. At the same time, the Euclidean distance between the self-attention score map of the newly generated intermediate representation in each denoising process and the corresponding self-attention score map of the original intermediate representation is calculated as the first part of the loss function.

[0111] Step S106, adding the intermediate adversarial patch to the data set as input data of the detection model, and obtaining a confidence score of the detection model;

[0112] It can be understood that the generated intermediate adversarial patch is added to the dataset and input into the detection model to check the output confidence of the detection model. The lower the confidence, the higher the degree of adversarial effect of the patch on the detector.

[0113] Step S107, adding the Euclidean distance and the confidence score to obtain a target loss function; when the target loss function reaches a preset standard value, the corresponding intermediate adversarial patch in step S105 is used as the target adversarial patch, and when the target loss function does not reach the preset standard value, step S108 is executed;

[0114] It can be understood that the Euclidean distance between the self-attention score maps and the confidence score of the detection model are added as the target loss function. The smaller the loss function, the more successful the generated adversarial patch. In this embodiment, the loss function not only ensures the aggressiveness of the adversarial patch (low confidence), but also maintains the semantic consistency between the adversarial patch and the reference disguise map.

[0115] Step S108, perform gradient calculation based on the target loss function to correct the original latent space Gaussian noise, substitute the corrected original latent space Gaussian noise into step S104 and continue to repeat subsequent steps until the number of repetitions reaches a preset number of iterations, and use the intermediate adversarial patch in step S105 corresponding to the last iteration as the target adversarial patch, which is used to counter the target detector.

[0116] It can be understood that based on the loss function, the Gaussian noise in the original latent space is corrected by gradient calculation, and adjusted so that the generated adversarial patch is closer to the target. After multiple rounds of iterations, the adversarial camouflage pattern with the strongest concealment in the detector is finally obtained.

[0117] The embodiment of the present invention encodes a preset reference camouflage picture, diffuses it to Gaussian noise using the DDIM inverse process of the stable diffusion model, and then gradually denoises to generate a target latent space representation, so that the generated adversarial camouflage pattern is more natural visually, avoiding the disadvantages of traditional adversarial patches being too bright and conspicuous. Compared with the situation in which the patches generated in the prior art method are abrupt in color and easily perceived by the human eye, the adversarial camouflage pattern is more concealed in the real environment through the gradual optimization of the diffusion model; by generating a first self-attention score map and a second self-attention score map, the Euclidean distance is calculated to ensure a high degree of semantic consistency between the adversarial camouflage pattern and the original reference camouflage picture, avoiding the problem of destroying the semantic information of the reference image in the adversarial patch generation process in the prior art, and using the Euclidean distance of the self-attention score map as the loss The generated intermediate adversarial patch is added to the data set as the input of the detection model to obtain the confidence score of the detection model, and then the Euclidean distance is added to the confidence score to obtain the target loss function, so that the adversarial camouflage pattern can minimize the detection confidence of the target detection model, which solves the problem that the adversarial patch in the prior art is easily recognized by the detection model, improves the aggressiveness and concealment of the adversarial camouflage pattern, and through multiple rounds of iterations, adjusts the Gaussian noise in the latent space and optimizes the loss function. The generated adversarial patch maintains the naturalness of the vision, and maintains the consistency of the image semantics under the optimization of the self-attention score map. The concealment of the finally generated adversarial patch in the physical world is significantly enhanced.

[0118] In an optional implementation, the step S103, using the DDIM inverse process to gradually diffuse the original latent space representation into the original latent space Gaussian noise, specifically includes:

[0119] Use the inverse process of DDIM to add noise to the original latent space representation x0 T times to obtain the original latent space Gaussian noise x T , where each noise addition generates the corresponding first intermediate representation x t , the first intermediate representation x corresponding to the t+1th noise addition process t+1 The generation of depends on the first intermediate representation x corresponding to the t-th noise addition process t .

[0120] For example, the encoder of the Stable diffusion model is used to encode the reference disguised image into the original latent space representation x0, and the DDIM inverse process is used to add noise to x0 T times, where T is 20, to obtain the latent space Gaussian noise x T And a series of intermediate latent space representations x1, x2…x T-1, where the generation of x1 depends on x0, the generation of x2 depends on x1, and so on.

[0121] Specifically, the original latent space representation x0 is denoised T times using the inverse process of DDIM to obtain the original latent space Gaussian noise x T , where each noise addition generates the corresponding first intermediate representation x t ,include:

[0122] Step A1, obtaining a first calculation formula corresponding to the original DDIM process, wherein the first calculation formula is:

[0123]

[0124] in, α s Represents the mean coefficient of the normal distribution in each denoising process, ∈ θ (x t ,t) represents the residual predicted by the stable diffusion model at time t; σ t represents the variance of the normal distribution during the denoising process at time t, which is set to 0; z is random noise, which obeys the normal distribution with a mean of 0 and a variance of 1;

[0125] For example, x0 to x T The size of is 64*64*4, where α s It is obtained by linear interpolation from 0.9999 to 0.98; ∈ θ (x t ,t) is the residual predicted by the stable diffusion model at time t, with a size of 64*64*4; σ t is the variance of the normal distribution during the denoising process at time t, which is generally set to 0; z is random noise, which obeys a normal distribution with a mean of 0 and a variance of 1, and has a size of 64*64*4.

[0126] Step A2: determining a first intermediate representation x based on the first calculation formula t-1 About the first intermediate representation x t The denoised difference term is obtained by a second calculation formula:

[0127]

[0128] Step A3, performing a reverse operation on the second calculation formula to obtain t Generate x t+1 The noise-added difference term is expressed by a third calculation formula, and the third calculation formula is:

[0129]

[0130] Step A4, determining x based on the third calculation formula t Generate x t+1 The one-time noise adding process corresponding to the time is represented by the following mathematical formula:

[0131] x t+1 =Inverse(x t );

[0132] Step A5: Determine the original latent space Gaussian noise x obtained by performing T noise additions on the original latent space representation x0 according to each noise addition process. T The fourth calculation formula is:

[0133]

[0134] The embodiment of the present invention further refines the generation process of the original latent space Gaussian noise. Through multiple calculation formulas, the noise addition is reversed, and the noise addition in each step is more accurately controlled to ensure the generation accuracy of Gaussian noise in each step. The continuous noise addition process also avoids sudden changes in the noise addition process, further improving the semantic consistency and naturalness of the final adversarial patch.

[0135] The embodiment of the present invention gradually converts the original latent space representation into Gaussian noise by adding noise T times, generates a first intermediate representation at each step, and the generation of the intermediate representation at each step depends on the intermediate representation at the previous step. The structural information of the original latent space representation is better maintained by a step-by-step diffusion method, and the DDIM inverse process is introduced to carefully modify the reference camouflage image in the latent space, which can ensure semantic consistency and ensure that the finally generated adversarial patch maintains a high degree of naturalness.

[0136] In an optional implementation, the first calculation formula is also used to characterize any denoising process in the whole process of denoising the original latent space Gaussian noise through the stable diffusion model;

[0137] The step S104, based on the stable diffusion model, gradually denoises the original latent space Gaussian noise to obtain the target latent space representation, specifically includes:

[0138] Step B1, determining x based on the first calculation formula t+1 Generate x t The first denoising process corresponding to the time is represented by the following mathematical formula:

[0139] x t =Denoise(x t+1 );

[0140] Step B2: Determine the Gaussian noise x from the original latent space according to each denoising process. T Perform T denoising to get the target latent space representation x ′ 0, the fifth calculation formula is:

[0141]

[0142] The embodiment of the present invention characterizes each denoising process through the first calculation formula, and determines the target latent space representation obtained by denoising the original latent space Gaussian noise T times through the fifth calculation formula, thereby ensuring that the generated target latent space representation is more in line with the expected goal, further improving the naturalness of the final adversarial patch, and effectively increasing the concealment in the physical world. It solves the dual problems of practicality and naturalness of adversarial patches in the prior art, and maintains effective deception to target detectors.

[0143] In an optional implementation, the first self-attention score map is represented by the following mathematical formula:

[0144]

[0145] The second self-attention score map is represented by the following mathematical formula:

[0146]

[0147] Where N represents the size of the self-attention score map;

[0148] The step S105, decoding the target latent space representation to obtain an intermediate adversarial patch, and at the same time, obtaining a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation, and obtaining a Euclidean distance between the first self-attention score map and the second self-attention score map based on the first self-attention score map and the second self-attention score map, specifically includes:

[0149] Step C1, using a decoder corresponding to the stable diffusion model to represent the target latent space x ′ 0 is decoded to obtain the intermediate adversarial patch x ′ ;

[0150] Step C2: At the same time, a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation are obtained, and a Euclidean distance between the first self-attention score map and the second self-attention score map is obtained based on the first self-attention score map and the second self-attention score map, wherein the Euclidean distance between the two constitutes the first part of the target loss function. It is expressed by the sixth calculation formula:

[0151]

[0152] For example, the decoder of the Stable diffusion model is used to represent the latent space x ′ 0 decoded into an image x of the same dimension as the reference disguised image ′ , calculate the self-attention score map of the newly generated intermediate representation in each step of denoising Compared with the original self-attention score map The Euclidean distance of is used as the first part of the loss function, where N is the size of the self-attention score map, including three scales {36, 18, 9}. This loss function is named The goal is to maintain x by minimizing the Euclidean distance between the two ′ Similarity to x0.

[0153] The embodiment of the present invention obtains the Euclidean distance by calculating the first self-attention score map and the second self-attention score map, so that the generated intermediate adversarial patch is highly consistent with the original reference camouflage image in semantics. By calculating the Euclidean distance between the two, the similarity between the adversarial patch pattern and the reference original image at the semantic level is quantified as part of the target loss function, which can effectively reduce the detection confidence of the target detector, making the generated camouflage pattern more aggressive and more difficult to be recognized by the detection system.

[0154] In an optional implementation, the step S106, adding the intermediate adversarial patch to the data set as input data of the detection model, and obtaining a confidence score of the detection model, specifically includes:

[0155] Step D1: According to the location information of the target object P in each picture of the training set and the size ratio of the preset expected adversarial patch, the intermediate adversarial patch x is transformed using an affine transformation matrix. ′ Rotate and scale to generate image I with intermediate adversarial patches M ;

[0156] For example, the adversarial patch x ′ According to the location information of pedestrian P in each picture <x l ,y l ,x r ,y r > and the desired size ratio of the adversarial patch, use the affine transformation matrix to transform x ′ Rotate and scale to generate image I with adversarial patches M .

[0157] Specifically, step D1 uses an affine transformation matrix to transform the intermediate adversarial patch x into ′Rotate and scale to generate image I with intermediate adversarial patches M , expressed by the seventh calculation formula:

[0158] I M =S*R(θ)*x ′ ;

[0159] Among them, R(θ) represents the rotation matrix, which is used to rotate the adversarial patch and is expressed by the following mathematical formula:

[0160]

[0161] S represents the scaling matrix, which is used to scale the adversarial patch and is expressed by the following mathematical formula:

[0162]

[0163] S x and S y Respectively represent the scaling factors corresponding to the x and y axes.

[0164] For example, the value range of θ is [-20°, 20°), S x and S y The values ​​are 0.25 respectively.

[0165] The embodiment of the present invention generates an image with an intermediate adversarial patch through a rotation matrix and a scaling matrix, which can still maintain visual naturalness at various viewing angles and distances, and is suitable for targets in actual environments, and can obviously mislead target detectors in different target scenes.

[0166] Step D2: the image I M With the image I in the training set r Fusion, get adversarial input data I A , as input data of the detection model;

[0167] Specifically, in step D2, the image I M With the image I in the training set r Fusion, get adversarial input data I A , as the input data of the detection model, is represented by the eighth calculation formula:

[0168]

[0169] For example, the image I M INRIA pedestrian dataset r Fusion, forming the adversarial input I for the target detection model YOLOv5 A .

[0170] The embodiment of the present invention fuses the generated camouflage picture with the picture in the training set to construct adversarial input data as the input of the detection model, so that the camouflage pattern is more integrated with the original image data in the training set, thereby reducing the recognition capability of the target detector.

[0171] Step D3: input the input data into the detection model to obtain the detection confidence score F of the detection model for the target object P. obj F(I A ).

[0172] For example, I A Input into the target detection model YOLOv5 to get the model F about I A The detection confidence score F of all pedestrian regions in obj F(I A ).

[0173] The embodiment of the present invention performs an affine transformation on the intermediate adversarial patch so that the size and angle of the adversarial patch better match the position and size of the target object in the training set, and generates more accurate adversarial input data by fusing it with the image in the training set, thereby improving the visual naturalness and authenticity of the camouflage. By inputting the fused image into the detection model, it is ensured that the generated adversarial camouflage pattern can reduce the confidence of the target detector under various conditions, further improving the aggressiveness and stability of the adversarial camouflage pattern, and having a higher anti-interference ability.

[0174] In an optional implementation, the detection confidence score F obj F(I A ) above the preset standard value constitutes the second part of the target loss function It is expressed by the ninth calculation formula:

[0175]

[0176] In step S107, the Euclidean distance is added to the confidence score to obtain a target loss function, which is represented by the tenth calculation formula:

[0177]

[0178] Among them, α and β represent the weight corresponding to the Euclidean distance and the weight corresponding to the confidence score, respectively.

[0179] Specifically, take F obj F(I A ) is used as the second part of the loss function, and the loss function is named Will and The weighted combination is performed with appropriate parameters α and β. In this embodiment, α and β are set to 6 and 1 respectively. Using the total loss function Perform gradient backpropagation calculation to optimize the latent space Gaussian noise x T , adversarial camouflage is achieved by minimizing the maximum confidence score.

[0180] The embodiment of the present invention controls the composition of the target loss function more flexibly by assigning different weights to the Euclidean distance and the confidence score of the detection model respectively, and the loss function includes the Euclidean distance and the confidence score. By minimizing the Euclidean distance between the self-attention score map and the original self-attention score map in the process of generating the adversarial patch, and the confidence score of the pedestrian detection model, it is ensured that the generated adversarial patch can minimize the detection confidence of the model and maintain the visual naturalness of the image. The overall loss function composed of the Euclidean distance and the confidence score makes the adversarial patch more aggressive and concealed.

[0181] In actual application, the process of using the method for generating an anti-camouflage pattern based on a stable diffusion model provided by this embodiment can be referred to as follows: Figure 2 Taking the attack on pedestrian target detector as an example, a pedestrian dataset, a pedestrian detection model, a stable diffusion model and a reference camouflage picture are collected; the encoder of the Stablediffusion model is used to encode the reference camouflage picture into the original latent space representation, and then the DDIM inverse process is used to gradually diffuse the original latent space representation of the reference camouflage picture into the original latent space Gaussian noise. At the same time, an intermediate representation belonging to each step will be generated in each diffusion process; the latent space Gaussian noise is gradually denoised by the stable diffusion model to obtain the latent space representation, and then the decoder of the Stablediffusion model is used to decode the obtained latent space representation into an image of the same dimension as the reference camouflage picture, which is called an adversarial patch. At the same time, the Euclidean distance between the self-attention score map of the newly generated intermediate representation in each denoising process and the corresponding self-attention score map of the original intermediate representation is calculated as the first part of the loss function; the adversarial patch is pasted to the pedestrian target area in each picture of the pedestrian dataset as the input of the pedestrian detection model using affine transformation, and the confidence score of the pedestrian detection model on the pedestrian class is used as the second part of the adversarial loss function. The two parts of the loss function, the Euclidean distance between the self-attention score maps and the confidence score, are added together as the final loss function, and the gradient calculation is performed to modify the Gaussian noise in the latent space. The above iterative steps are repeated to generate a natural and deceptive target adversarial patch, which can be printed and pasted in the middle of the front of the pedestrian's body to achieve the goal of evading the perception of the pedestrian detection model and complete the adversarial camouflage.

[0182] For example, Figure 3-3 01 is a reference camouflage picture. Figure 3-302 is the absolute difference between the original disguised image and the adversarial patch, Figure 3-3 03 is the target adversarial patch generated for this implementation. The target adversarial patch is printed and accurately pasted on the middle of the front of the pedestrian's body to complete the adversarial camouflage. Then, the camouflage effect is tested. The test results are as follows: Figure 4 As shown in the right picture in the middle, the pedestrian target detector did not detect the tester with the target adversarial patch of this embodiment. Figure 4 In the middle left picture, the test person has the original camouflage reference image on his body. The original reference camouflage image can still be detected by the pedestrian target detector and cannot escape the monitoring and recognition model.

[0183] The embodiment of the present invention provides a method for generating an anti-camouflage pattern based on a stable diffusion model, which has the following beneficial effects compared with the prior art:

[0184] (1) By attacking the latent space of the stable diffusion model, the present invention generates a more natural adversarial camouflage pattern, avoiding the problem of overly bright colors of adversarial patches in traditional methods, improving the concealment and practicality in real environments, while also maintaining effective deception of target detection models. Taking the attack on the pedestrian perception system as an example, the present invention aims to attack and mislead the pedestrian target detection model, so that pedestrians can evade the recognition of modern detection systems, thereby achieving the purpose of camouflage or evading detection.

[0185] (2) The present invention designs a pedestrian target attack algorithm for the pedestrian detection model, but the generated adversarial patch is not limited to the specific pedestrian detection model. It can also show good adversarial effect even on the black box model with unknown parameters. It can be widely used in single-stage or multi-stage target detection algorithms of different tasks, and has obvious attack effects in both the physical world and the digital world. The adversarial patches generated by traditional methods are too obvious in color and easily detected by the human eye. The present invention introduces the DDIM inverse process and the stable diffusion model, and carefully modifies the reference camouflage image in the latent space to ensure that the generated adversarial patches maintain a high degree of naturalness, effectively increasing the concealment in the physical world, and solving the dual problems of practicality and naturalness of adversarial patches.

[0186] (3) By minimizing the Euclidean distance between the self-attention score map and the original self-attention score map during the generation of the adversarial patch, as well as the confidence score of the pedestrian detection model, it is ensured that the generated adversarial patch can minimize the detection confidence of the model while maintaining the visual naturalness of the image. This loss function makes the adversarial patch more aggressive and concealed. In view of the problem of severe damage to the semantic information of the reference camouflage image during the generation of adversarial patches in the prior art, the present invention uses the Euclidean distance of the self-attention score map as part of the loss function to ensure that the generated adversarial patch maintains a high degree of semantic consistency with the original image, avoids the distortion of the original semantics of the image caused by adversarial enhancement, and improves the applicability and authenticity of the camouflage pattern.

[0187] In this embodiment, a system for generating an anti-camouflage pattern based on a stable diffusion model is also provided. The system is used to implement the above-mentioned embodiments and preferred implementation modes. The descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0188] This embodiment provides a system for generating an anti-camouflage pattern based on a stable diffusion model. Figure 5 As shown, including:

[0189] The data preparation and model loading module 501 is used to obtain a data set, a detection model, a stable diffusion model and a preset reference camouflage picture corresponding to the target detector;

[0190] A latent space encoding module 502 is used to encode the preset reference disguised picture using an encoder to obtain an original latent space representation; wherein the preset reference disguised picture represents an initial adversarial patch;

[0191] A diffusion process simulation module 503, used for gradually diffusing the original latent space representation into the original latent space Gaussian noise by using the DDIM inverse process, wherein each step of the gradual diffusion process generates a corresponding first intermediate representation;

[0192] A denoising and reconstruction module 504 is used to gradually denoise the original latent space Gaussian noise based on the stable diffusion model to obtain a target latent space representation; wherein each step of the gradual denoising process generates a corresponding second intermediate representation;

[0193] The adversarial patch generation and distance calculation module 505 is used to decode the target latent space representation to obtain an intermediate adversarial patch, and at the same time, obtain a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation, and obtain the Euclidean distance between the two based on the first self-attention score map and the second self-attention score map; wherein the intermediate adversarial patch represents an image of the same dimension as the preset reference disguised image, and belongs to an adversarial patch that appears in each repetition process;

[0194] an adversarial patch evaluation module 506, configured to add the intermediate adversarial patch to the data set as input data of the detection model and obtain a confidence score of the detection model;

[0195] A target loss calculation module 507 is used to add the Euclidean distance and the confidence score to obtain a target loss function; when the target loss function reaches a preset standard value, the corresponding intermediate adversarial patch in step S105 is used as a target adversarial patch, and when the target loss function does not reach the preset standard value, step S108 is executed;

[0196] The gradient optimization module 508 is used to perform gradient calculation based on the target loss function to correct the original latent space Gaussian noise, and continue to repeat subsequent steps using the corrected original latent space Gaussian noise until the number of repetitions reaches a preset number of iterations, and use the intermediate adversarial patch corresponding to the last iteration as the target adversarial patch, which is used to counter the target detector.

[0197] In an optional implementation, the diffusion process simulation module 503 specifically includes:

[0198] The noise diffusion unit is used to add noise to the original latent space representation x0 T times using the inverse process of DDIM to obtain the original latent space Gaussian noise x T , where each noise addition generates the corresponding first intermediate representation x t , the first intermediate representation x corresponding to the t+1th noise addition process t+1 The generation of depends on the first intermediate representation x corresponding to the t-th noise addition process t .

[0199] In an optional implementation manner, the noise diffusion unit specifically includes:

[0200] Obtain a first calculation formula corresponding to the DDIM original process, where the first calculation formula is:

[0201]

[0202] in, α sRepresents the mean coefficient of the normal distribution in each denoising process, ∈ θ (x t ,t) represents the residual predicted by the stable diffusion model at time t; σ t represents the variance of the normal distribution during the denoising process at time t, which is set to 0; z is random noise, which obeys the normal distribution with a mean of 0 and a variance of 1;

[0203] Determine a first intermediate representation x based on the first calculation formula t-1 About the first intermediate representation x t The denoised difference term is obtained by a second calculation formula:

[0204]

[0205] Reverse the second calculation formula to obtain t Generate x t+1 The noise-added difference term is expressed by a third calculation formula, and the third calculation formula is:

[0206]

[0207] Determine x based on the third calculation formula t Generate x t+1 The one-time noise adding process corresponding to the time is represented by the following mathematical formula:

[0208] x t+1 =Inverse(x t );

[0209] According to each noise addition process, the original latent space Gaussian noise x is obtained by performing T noise additions from the original latent space representation x0. T The fourth calculation formula is:

[0210]

[0211] In an optional implementation, the first calculation formula is also used to characterize any denoising process in the whole process of denoising the original latent space Gaussian noise through the stable diffusion model; the denoising and reconstruction module 504 specifically includes:

[0212] Determine x based on the first calculation formula t+1 Generate x t The first denoising process corresponding to the time is represented by the following mathematical formula:

[0213] x t =Denoise(x t+1 );

[0214] According to each denoising process, the Gaussian noise x T Perform T denoising to get the target latent space representation x ′ 0, the fifth calculation formula is:

[0215]

[0216] In an optional implementation, the first self-attention score map is represented by the following mathematical formula:

[0217]

[0218] The second self-attention score map is represented by the following mathematical formula:

[0219]

[0220] Where N represents the size of the self-attention score map;

[0221] The step S105, decoding the target latent space representation to obtain an intermediate adversarial patch, and at the same time, obtaining a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation, and obtaining a Euclidean distance between the first self-attention score map and the second self-attention score map based on the first self-attention score map and the second self-attention score map, specifically includes:

[0222] The target latent space representation x is represented by using a decoder corresponding to the stable diffusion model ′ 0 is decoded to obtain the intermediate adversarial patch x ′ ;

[0223] At the same time, a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation are obtained, and a Euclidean distance between the first self-attention score map and the second self-attention score map is obtained based on the first self-attention score map and the second self-attention score map, wherein the Euclidean distance between the two constitutes the first part of the target loss function It is expressed by the sixth calculation formula:

[0224]

[0225] In an optional implementation, the adversarial patch evaluation module 506 specifically includes:

[0226] According to the location information of the target object P in each picture of the training set and the size ratio of the preset expected adversarial patch, the intermediate adversarial patch x is transformed using the affine transformation matrix. ′ Rotate and scale to generate image I with intermediate adversarial patches M ;

[0227] The picture I M With the image I in the training set r Fusion, get adversarial input data I A , as input data of the detection model;

[0228] Input the input data into the detection model to obtain the detection confidence score F of the detection model about the target object P obj F(I A ).

[0229] In an optional embodiment, the intermediate adversarial patch x is transformed using an affine transformation matrix according to the location information of the target object P in each picture of the training set and the size ratio of the preset desired adversarial patch. ′ Rotate and scale to generate image I with intermediate adversarial patches M , expressed by the seventh calculation formula:

[0230] I M =S*R(θ)*x ′ ;

[0231] Among them, R(θ) represents the rotation matrix, which is used to rotate the adversarial patch and is expressed by the following mathematical formula:

[0232]

[0233] S represents the scaling matrix, which is used to scale the adversarial patch and is expressed by the following mathematical formula:

[0234]

[0235] S x and S y Respectively represent the scaling factors corresponding to the x and y axes.

[0236] In an optional implementation, the image I M With the image I in the training set r Fusion, get adversarial input data I A , as the input data of the detection model, is represented by the eighth calculation formula:

[0237]

[0238] In an optional implementation, the detection confidence score F obj F(I A ) above the preset standard value constitutes the second part of the target loss function It is expressed by the ninth calculation formula:

[0239]

[0240] The target loss calculation module 507 is represented by the tenth calculation formula:

[0241]

[0242] Among them, α and β represent the weight corresponding to the Euclidean distance and the weight corresponding to the confidence score, respectively.

[0243] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0244] The anti-camouflage pattern generation system based on the stable diffusion model in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0245] The embodiment of the present invention also provides a computer device having the above Figure 5 The adversarial camouflage pattern generation system based on the stable diffusion model is shown.

[0246] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0247] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0248] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0249] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0250] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0251] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0252] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0253] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0254] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the defined scope.

Claims

1. A method for generating an anti-camouflage pattern based on a stable diffusion model, characterized in that: The method comprises: Step S101, obtaining a data set, a detection model, a stable diffusion model, and a preset reference camouflage picture corresponding to a target detector; Step S102, encoding the preset reference disguised picture using an encoder to obtain an original latent space representation; wherein the preset reference disguised picture represents an initial adversarial patch; Step S103, using the DDIM inverse process to gradually diffuse the original latent space representation into the original latent space Gaussian noise, wherein each step of the gradual diffusion process generates a corresponding first intermediate representation; Step S104, based on the stable diffusion model, the original latent space Gaussian noise is gradually denoised to obtain a target latent space representation; wherein each step of the gradual denoising process generates a corresponding second intermediate representation; Step S105, decoding the target latent space representation to obtain an intermediate adversarial patch, and at the same time, obtaining a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation, and obtaining a Euclidean distance between the first self-attention score map and the second self-attention score map; wherein the intermediate adversarial patch represents an image of the same dimension as the preset reference disguised image, and belongs to an adversarial patch that appears in each repetition process; Step S106, adding the intermediate adversarial patch to the data set as input data of the detection model, and obtaining a confidence score of the detection model; Step S107, adding the Euclidean distance and the confidence score to obtain a target loss function; when the target loss function reaches a preset standard value, the corresponding intermediate adversarial patch in step S105 is used as the target adversarial patch, and when the target loss function does not reach the preset standard value, step S108 is executed; Step S108, perform gradient calculation based on the target loss function to correct the original latent space Gaussian noise, substitute the corrected original latent space Gaussian noise into step S104 and continue to repeat subsequent steps until the number of repetitions reaches a preset number of iterations, and use the intermediate adversarial patch in step S105 corresponding to the last iteration as the target adversarial patch, which is used to counter the target detector.

2. The method according to claim 1, characterized in that The step S103, using the DDIM inverse process to gradually diffuse the original latent space representation into the original latent space Gaussian noise, specifically includes: Use the inverse process of DDIM to add noise to the original latent space representation x0 T times to obtain the original latent space Gaussian noise x T , where each noise addition generates the corresponding first intermediate representation x t , the first intermediate representation x corresponding to the t+1th noise addition process t+1 The generation of depends on the first intermediate representation x corresponding to the t-th noise addition process t .

3. The method according to claim 2, characterized in that The DDIM inverse process is used to perform T noise additions on the original latent space representation x0 to obtain the original latent space Gaussian noise x T , where each noise addition generates the corresponding first intermediate representation x t , specifically including: Obtain a first calculation formula corresponding to the DDIM original process, where the first calculation formula is: in, α s Represents the mean coefficient of the normal distribution in each denoising process, ∈ θ (x t ,t) represents the residual predicted by the stable diffusion model at time t; σ t represents the variance of the normal distribution during the denoising process at time t, which is set to 0; z is random noise, which obeys the normal distribution with a mean of 0 and a variance of 1; Determine a first intermediate representation x based on the first calculation formula t-1 About the first intermediate representation x t The denoised difference term is obtained by a second calculation formula: Reverse the second calculation formula to obtain t Generate x t+1 The noise-added difference term is expressed by a third calculation formula, and the third calculation formula is: Determine x based on the third calculation formula t Generate x t+1 The one-time noise adding process corresponding to the time is represented by the following mathematical formula: x t+1 =Inverse(x t ); According to each noise addition process, the original latent space Gaussian noise x is obtained by performing T noise additions from the original latent space representation x0. T The fourth calculation formula is:

4. The method according to claim 3, characterized in that The first calculation formula is also used to characterize any denoising process in the whole process of denoising the original latent space Gaussian noise through the stable diffusion model; The step S104, based on the stable diffusion model, gradually denoises the original latent space Gaussian noise to obtain the target latent space representation, specifically includes: Determine x based on the first calculation formula t+1 Generate x t The first denoising process corresponding to the time is represented by the following mathematical formula: x t =Denoise(x t+1 ); According to each denoising process, the Gaussian noise x T Perform T times of denoising to obtain the fifth calculation formula for the target latent space representation x′0, which is:

5. The method according to claim 4, characterized in that The first self-attention score map is represented by the following mathematical formula: The second self-attention score map is represented by the following mathematical formula: Where N represents the size of the self-attention score map; The step S105, decoding the target latent space representation to obtain an intermediate adversarial patch, and at the same time, obtaining a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation, and obtaining a Euclidean distance between the first self-attention score map and the second self-attention score map based on the first self-attention score map and the second self-attention score map, specifically includes: Decoding the target latent space representation x′0 using a decoder corresponding to the stable diffusion model to obtain an intermediate adversarial patch x′; At the same time, a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation are obtained, and a Euclidean distance between the first self-attention score map and the second self-attention score map is obtained based on the first self-attention score map and the second self-attention score map, wherein the Euclidean distance between the two constitutes the first part of the target loss function It is expressed by the sixth calculation formula:

6. The method according to claim 5, characterized in that The step S106, adding the intermediate adversarial patch to the data set as input data of the detection model, and obtaining the confidence score of the detection model, specifically includes: According to the location information of the target object P in each picture of the training set and the size ratio of the preset expected adversarial patch, the intermediate adversarial patch x′ is rotated and scaled using the affine transformation matrix to generate an image I with the intermediate adversarial patch M ; The picture I M With the image I in the training set r Fusion, get adversarial input data I A , as input data of the detection model; The input data is input into the detection model to obtain the detection confidence score F of the detection model about the target object P obj F(I A ).

7. The method according to claim 6, characterized in that According to the location information of the target object P in each picture of the training set and the preset size ratio of the expected adversarial patch, the intermediate adversarial patch x′ is rotated and scaled using the affine transformation matrix to generate a picture I with the intermediate adversarial patch M , expressed by the seventh calculation formula: I M =S*R(θ)*x′; Among them, R(θ) represents the rotation matrix, which is used to rotate the adversarial patch and is expressed by the following mathematical formula: S represents the scaling matrix, which is used to scale the adversarial patch and is expressed by the following mathematical formula: S x and S y Respectively represent the scaling factors corresponding to the x and y axes.

8. The method according to claim 6, characterized in that The picture I M With the image I in the training set r Fusion, get adversarial input data I A , as the input data of the detection model, is represented by the eighth calculation formula:

9. The method according to claim 6, characterized in that The detection confidence score F obj F(I A ) above the preset standard value constitutes the second part of the target loss function It is expressed by the ninth calculation formula: In step S107, the Euclidean distance is added to the confidence score to obtain a target loss function, which is represented by the tenth calculation formula: Among them, α and β represent the weight corresponding to the Euclidean distance and the weight corresponding to the confidence score, respectively.

10. A system for generating countermeasure camouflage patterns based on a stable diffusion model, characterized in that: The system comprises: Data preparation and model loading module, used to obtain the data set, detection model, stable diffusion model and preset reference camouflage picture corresponding to the target detector; A latent space encoding module, configured to encode the preset reference disguised image using an encoder to obtain an original latent space representation; wherein the preset reference disguised image represents an initial adversarial patch; A diffusion process simulation module, used for gradually diffusing the original latent space representation into the original latent space Gaussian noise by using the DDIM inverse process, wherein each step of the gradual diffusion process generates a corresponding first intermediate representation; A denoising and reconstruction module, configured to perform step-by-step denoising on the original latent space Gaussian noise based on the stable diffusion model to obtain a target latent space representation; wherein each step of the step-by-step denoising process generates a corresponding second intermediate representation; an adversarial patch generation and distance calculation module, configured to decode the target latent space representation to obtain an intermediate adversarial patch, and at the same time, obtain a first self-attention score map corresponding to the first intermediate representation and a second self-attention score map corresponding to the second intermediate representation, and obtain a Euclidean distance between the first self-attention score map and the second self-attention score map; wherein the intermediate adversarial patch represents an image of the same dimension as the preset reference disguised image, and belongs to an adversarial patch that appears in each repetition process; an adversarial patch evaluation module, configured to add the intermediate adversarial patch to the data set as input data of the detection model and obtain a confidence score of the detection model; A target loss calculation module, configured to add the Euclidean distance to the confidence score to obtain a target loss function; when the target loss function reaches a preset standard value, the corresponding intermediate adversarial patch in step S105 is used as a target adversarial patch; when the target loss function does not reach the preset standard value, step S108 is executed; A gradient optimization module is used to perform gradient calculation based on the target loss function to correct the original latent space Gaussian noise, and to continue to repeat subsequent steps using the corrected original latent space Gaussian noise until the number of repetitions reaches a preset number of iterations, and to use the intermediate adversarial patch corresponding to the last iteration as the target adversarial patch, which is used to counter the target detector.

11. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for generating an anti-camouflage pattern based on a stable diffusion model according to any one of claims 1 to 9 by executing the computer instructions.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for generating an anti-camouflage pattern based on a stable diffusion model according to any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the method for generating an anti-camouflage pattern based on a stable diffusion model according to any one of claims 1 to 9.

Images