Image anti-generation method and device based on adversarial attack

Abstract

The invention provides an image anti-generation method based on adversarial attack, which comprises the following steps of: segmenting a target image to obtain a key mask of the target image; and adding antagonistic disturbance on the key mask to obtain an anti-generation protection image of the target image. According to the method, few unperceived adversarial noise is used for reversely generating important concepts, such as facial areas in human images, for protecting users, malicious personalization can be effectively prevented, and compared with MIST and other reverse generation protection methods for the whole images, the method provides better balance between safety and visual quality of noise. The invention further provides an anti-attack image anti-generation device, a computer readable storage medium and electronic equipment.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an image degeneration method and device based on counter-attack. Background Art

[0002] With the emergence of popular large-scale diffusion models such as Stablediffusion and DALL·E 2 / 3, people who lack professional knowledge in painting or photography can easily create realistic or artistic works through simple text descriptions. However, the success of these models has raised potential concerns about privacy and intellectual property rights. For example, attackers can easily imitate well-known artworks for commercial purposes, or through personalization techniques such as Textual Inversion or DreamBooth, using only a few reference images, they can create deceptive content containing sensitive private information of others.

[0003] A mainstream direction to mitigate these potential risks is to use adversarial attack techniques to transform original images into adversarial samples that resist personalization. MIST uses adversarial sample attacks in the reversal phase of stable diffusion to prevent malicious imitation of the artist's style. Similarly, Anti-DreamBooth destroys the quality of DreamBooth model generation and enhances privacy by adding subtle adversarial perturbations to user images before publishing.

[0004] However, existing adversarial attack methods are mainly designed to protect the entire image from generation, and usually show obvious adversarial noise. They include background regions and other non-essential information into the anti-generation protection range, resulting in consumption of the already limited adversarial noise budget. Summary of the invention

[0005] In order to solve the problem of excessive resource consumption in the data degeneration protection process in the personalized generation scenario, the present invention proposes an image degeneration method based on adversarial attack, including: segmenting the target image to obtain a key mask of the target image; adding adversarial perturbations to the key mask to obtain a degeneration protection image of the target image.

[0006] Furthermore, an adversarial perturbation δ is added to the key mask m to obtain the anti-generated protection image x' of the target image x, where x'=x+δ, and x' satisfies: is the training target of the latent diffusion model, which generates a synthetic image based on the output of x, y is the given condition for the generation of the latent diffusion model, and ∈ is the limiting threshold of δ.

[0007] Furthermore, when and When , the minimum value of the adversarial perturbation δ is obtained; D(·) is the distance metric defined by the p-norm, α is the adversarial threshold, when Image classification attack methods can successfully induce attacks on this latent diffusion model.

[0008] Preferably, a SAM image segmentation model is used to generate a key mask m in the target image x using user-verified sentences and / or pixels.

[0009] The present invention also proposes an image degeneration device based on adversarial attack, comprising: a segmentation module, used to segment the target image and obtain a key mask of the target image; an adversarial module, used to add adversarial perturbations to the key mask to obtain a degeneration protection image of the target image.

[0010] Furthermore, the adversarial module includes: a disturbance adding module, which is used to add an adversarial perturbation δ to the key mask m to obtain an anti-generated protection image x' of the target image x, where x'=x+δ, and x' satisfies: is the training target of the latent diffusion model, which generates a synthetic image based on the output of x, y is the given condition for the generation of the latent diffusion model, and ∈ is the limiting threshold of δ.

[0011] Furthermore, the adversarial module includes: a disturbance generation module, used to obtain the minimum adversarial disturbance δ that satisfies the generation of x'; including: when satisfying and When , the minimum value of the adversarial perturbation δ is obtained; D(·) is the distance metric defined by the p-norm, α is the adversarial threshold, when Image classification attack methods can successfully induce attacks on this latent diffusion model.

[0012] Furthermore, the segmentation module uses the SAM image segmentation model to generate a key mask m through user-verified sentences and / or pixels in the target image x.

[0013] The present invention also proposes a computer-readable storage medium storing computer-executable instructions, characterized in that when the computer-executable instructions are executed, the image degeneration method based on adversarial attack as described above is implemented.

[0014] The present invention also provides an electronic device, comprising the image anti-generation device based on adversarial attack as described above.

[0015] The method of the present invention uses less imperceptible adversarial noise to counter-generate and protect user-important concepts, such as facial areas in human images, which can effectively prevent malicious personalization. Compared with counter-generation protection methods such as MIST that target the entire image, it provides a better balance between security and the visual quality of noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the process of converting the original image into an adversarial sample using adversarial attack technology.

[0017] Figure 2 It is a flow chart of the image degeneration method based on adversarial attack of the present invention.

[0018] Figure 3 It is a schematic diagram of an image anti-generation device based on adversarial attack of the present invention.

[0019] Figure 4 It is a schematic diagram of the confrontation module of the present invention.

[0020] Figure 5 It is a schematic diagram of an electronic device of the present invention.

[0021] Figure 6 It is a schematic diagram of the hardware structure of an electronic device of the present invention.

[0022] Fig. 7A , 7B It is a comparative schematic diagram of the present invention and the prior art.

[0023] Wherein, the accompanying drawings are marked as follows:

[0024] S1, S2: Step 10: Split Module

[0025] 20: Adversarial module 21: Perturbation adding module

[0026] 22: Perturbation generation module DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation method described herein is only used to explain the present invention and is not used to limit the present invention.

[0028] In view of the problems existing in the prior art, this paper proposes a new framework (REFIT) for developing flexible and open methods to counter-generate and protect important image content. Specifically, REFIT uses the effective spatial information provided by the user as a prior guide and strategically focuses on hindering the personalization process in these specified areas rather than in the entire image. By leveraging the advances in open domain segmentation techniques such as SAM, users can easily obtain the mask of the specified area, which is added as prior information in the generation process of adversarial samples to achieve accurate counter-generative protection of the target concept.

[0029] Furthermore, given that the output can still be identified as synthetic as long as there are some clear signs of anti-generation protection for the generation task, the present invention transforms the optimization objective to minimize the perceptibility of adversarial noise that can successfully attack. In order to facilitate the solution, the present invention uses the Lagrange multiplier method combined with gradient descent for optimization.

[0030] Figure 1 The flow of the regional concept anti-generation protection framework of the present invention is shown. Accurately describing spatial locations through language is challenging for users, but can be accurately achieved using masks. Due to the development of image segmentation tasks in open domains, the present invention uses SAM to generate an important concept mask m through user-verified sentences and / or pixels in a given target image x. Finally, the mask and image provided by the user will be fed together into the adversarial sample generation module. This process produces the final anti-generation protected image, which contains the specified concepts and regions according to the user's input. The present invention directly uses the original Stable Diffusion v1.4 during the attack, unlike Anti-DreamBooth, which requires a proxy model to be trained by DreamBooth before the adversarial attack.

[0031] Figure 2 Flowchart of the image degeneration method based on counter-attack of the present invention. Figure 2 As shown, in a first embodiment of the present invention, an image degeneration method is proposed, comprising:

[0032] Step S1, segment the target image x and obtain the key mask m of the target image x; use SAM to generate an important concept mask m through user-verified sentences / pixels in the given target image x, so that in the subsequent steps, the most critical information in the target image can be preferentially de-generated and protected; X'

[0033] Step S2, adding adversarial perturbation δ to the key mask m to obtain the anti-generated protection image x' of the target image x;

[0034] Specifically, most personalization techniques are applied to latent diffusion models (LDM, parameterized as θ). A typical LDM consists of an image encoder εθ , an image decoder A conditional encoder τ θ and a conditional denoising UNetε θ The training of LDM includes diffusion process and denoising process, both of which occur in the latent space. For the target image x, z0 = ε θ (x) is the latent code of x, the diffusion process randomly samples time t and Gaussian noise g, the latent vector z0 is in z t Noise g is added. Then the denoising UNetε is assigned θ Remove the number of τ in each time step t θ (y) is the conditional latent code z t The added noise and ε θ The output of is set to g′. The training objective of the latent diffusion model is for:

[0035]

[0036] Where t is the time step of the diffusion process and y is the given condition for generation.

[0037] The anti-generative protection image x' (adversarial sample) attempts to find an imperceptible perturbation δ and add it to the target image x to invalidate the entire conditional generation process. The adversarial example x'(x'=x+δ) of the trained LDM can be formalized as:

[0038]

[0039]

[0040] Where ∈ is the perturbation size and limits the maximum range of adversarial perturbations. Adversarial attacks transform original images into special samples that are difficult to denoise. They enhance the optimization challenge of the latent diffusion model, thereby achieving image anti-generation protection.

[0041] The difference from the prior art is that the technical solution of the present invention aims to achieve precise concept anti-generation protection. The present invention uses a mask m to represent the spatial position of important information in the feature map, thereby achieving precise adversarial attack. This precise optimization allows the most critical information in the anti-generation protection image to be prioritized with a smaller adversarial noise intensity. The optimization goal of the adversarial example in the method of the present invention can be defined as:

[0042]

[0043]

[0044]

[0045] The adversarial example x' makes the output of the diffusion model far away from the true Gaussian noise g in a specific mask area, while the predictions in other areas are close.

[0046] Based on the above formula, the adversarial sample will destroy the target area as much as possible. However, for the generation task, as long as there are some obvious signs of anti-generation protection, the output can still be identified as synthetic. Therefore, the present invention extends the typical image classification attack method C&W to the diffusion model and proposes a loose optimization goal to try to find the minimum adversarial perturbation δ that can successfully cause the attack. For convenience, the present invention sets a heuristic method: if The attack is successful, where α is a constant, and through experiments, the preferred value of α is 0.1. This problem can be redefined as:

[0047]

[0048]

[0049] Where D(·) is the distance metric defined by the p-norm (p=2 in practice). To facilitate the solution, we use the Lagrange multiplier method to obtain the final optimization target and use the gradient descent method to update the optimization:

[0050]

[0051]

[0052] It should be noted that, in various embodiments of the present invention, the sequence numbers of the above steps do not mean the order of execution. The order of execution of the steps should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0053] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0054] Figure 3 Schematic diagram of the image anti-generation device based on adversarial attack of the present invention. Figure 3 As shown, in the second embodiment of the present invention, an image degeneration device is proposed, comprising:

[0055] A segmentation module 10 is used to segment the target image x to obtain a key mask m of the target image x;

[0056] An adversarial module 20, configured to add an adversarial perturbation δ to the key mask m to obtain an anti-generated protected image x' of the target image x; Figure 4 Schematic diagram of the confrontation module of the present invention. Figure 4 As shown, the confrontation module 20 includes:

[0057] The disturbance adding module 21 is used to add the adversarial disturbance δ to the key mask m to obtain the anti-generative protection image x' (x'=x+δ) of the target image x; wherein, and x' satisfies:

[0058]

[0059]

[0060] is the training target of the latent diffusion model LDM, y is the given condition for generating the latent diffusion model, and ∈ is the limiting threshold of δ.

[0061] A disturbance generation module 22, used to obtain the minimum adversarial disturbance that satisfies the generation of x';

[0062] This paper extends the typical image classification attack method C&W to the diffusion model and proposes a loose optimization goal to try to find the minimum adversarial perturbation δ that can successfully cause the attack. For convenience, this paper sets a heuristic method: if The attack is successful, where α is a constant, and through experiments, the preferred value of α is 0.1. This problem can be redefined as:

[0063]

[0064]

[0065] Where D(·) is the distance metric defined by the p-norm (p=2 in practice). To facilitate the solution, we use the Lagrange multiplier method to obtain the final optimization target and use the gradient descent method to update the optimization:

[0066]

[0067]

[0068] In the third embodiment of the present invention, a computer-readable storage medium is proposed. The image degeneration device based on adversarial attack of the present invention, if its function is implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a computer-readable storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. Therefore, in the third embodiment of the present invention, a computer-readable storage medium is provided for storing a computer program for executing an image degeneration method based on adversarial attack. It should be understood that the computer-readable storage medium in the embodiment of the present invention may be a volatile memory and / or a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct rambus RAM (DRRAM).

[0069] Figure 5 Schematic diagram of an electronic device of the present invention. Figure 5As shown, in the fourth embodiment of the present invention, an electronic device is proposed, including the image anti-generation device based on adversarial attack as described above. A person of ordinary skill in the art can understand that all or part of the steps in the above method can be completed by instructing related hardware (such as a processor, FPGA, ASIC, etc.) through a program. All or part of the steps of the above embodiment can also be implemented using one or more integrated circuits. Accordingly, each module in the above embodiment can be implemented in the form of hardware, such as implementing its corresponding functions through an integrated circuit, or in the form of a software function module, such as implementing its corresponding functions through a processor executing a program / instruction stored in a memory. The embodiments of the present invention are not limited to any specific form of combination of hardware and software.

[0070] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0071] The electronic device of the present invention may be any device with data processing capability, and the device with data processing capability may be a device or apparatus such as a computer. The device embodiment may be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the processor of any device with data processing capability reads the corresponding computer program instructions in the non-volatile memory into the memory and runs the instructions. Figure 6 This is a schematic diagram of the hardware structure of an electronic device of the present invention. Figure 6 As shown in the figure, from the hardware level, it is a hardware structure diagram of any device with data processing capability where the image anti-generation device based on the counter-attack of the present invention is located, except Figure 6 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0072] When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0073] The proposed concept-based image anti-generation protection framework uses less imperceptible adversarial noise to anti-generate user-important concepts, such as facial regions in human images. This approach effectively prevents malicious personalization and provides a better balance between security and visual quality of noise compared to indiscriminate anti-generation protection methods such as MIST.

[0074] like Fig. 7A As shown, the adversarial samples of MIST show obvious strange textures on the neck and background, and the anti-generation protection effect is also similar to the special style pictures of the target person.

[0075] like Figure 7B As shown, the present invention first applies SAM to locate the user anti-generation protection area in the image, and then combines it with the mask to generate targeted adversarial samples to achieve accurate concept anti-generation protection. Different from the prior art, the present invention focuses on how to use reduced adversarial noise to prioritize important information in the anti-generation protection image.

[0076] Table 1 shows the automatic evaluation using different reference-based image quality assessments. The evaluation obtains the average difference between all adversarial samples and clean images. The statistical results show that the adversarial examples generated by the method of the present invention are closer to the original images. Since there is currently no standard to evaluate whether the anti-generation protection is successful in a specific concept domain, the present invention further introduces human evaluation.

[0077] Nine identities were sampled in VGGFace2, and all nine objects were sampled in the TI dataset. Adversarial examples generated by different methods were used to train personalized models, including TI and DB. A total of 52 people participated in the survey. The manual evaluation was also divided into two parts: protection effect and adversarial noise visibility. In order to achieve the anti-generation protection effect, an image generated by a personalized model was randomly selected. And the corresponding clean image was given. The evaluators judged whether the anti-generation protection was successful. Table 1 shows that the participants believed that the average anti-generation protection success rate of our method REFIT was 88.10%, which was higher than that of MIST and Anti-DB. In terms of noise visibility, adversarial examples of three different anti-generation protection methods were shown to human evaluators, and the one with the least obvious adversarial noise was selected. As shown in Table 1, 82.22% of human evaluators believed that the adversarial noise of REFIT was the most imperceptible. The results prove that the anti-generation protection effect should not be pursued blindly, as long as the purpose of anti-generation protection is to make others realize that there is a problem with the generated image. After achieving the purpose of anti-generation protection, continuing to pursue the anti-generation protection effect will reduce the visual quality of the adversarial image, which is difficult for users to accept.

[0078] method MIST Anti-DB The present invention FID↓ 115.81 117.94 87.83 PSNR↑ 31.82 24.66 32.23 SSIM↑ 0.81 0.64 0.82 Noise level visual assessment (TOP1) ↑ 6.67％ 11.11％ 82.22％ Anti-generation protection success rate↑ 85.24％ 86.69％ 88.10％

[0079] Table 1

[0080] We conduct rigorous quantitative and qualitative experiments on popular personalized generative models such as TI and DreamBooth, confirming REFIT's proficiency in concept preservation, especially for face cases. Compared with full-image anti-generative protection baselines such as MIST and Anti-DreamBooth, our method produces more subtle adversarial noise, especially in FID.

[0081] The above implementation modes are only used to illustrate the present invention, but not to limit the present invention. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also belong to the scope of the present invention. The patent protection scope of the present invention should be defined by the claims.

Claims

1. An image degeneration method based on adversarial attack, characterized in that: include: Segment the target image to obtain the key mask of the target image; Adding adversarial perturbations on the key mask, we obtain the anti-generative protection image of the target image.

2. The image degeneration method according to claim 1, characterized in that: Adding adversarial perturbation δ to the key mask m, we get the anti-generated protected image x' of the target image x, where x'=x+δ, and x' satisfies: is the training target of the latent diffusion model, which generates a synthetic image based on the output of x, y is the given condition for the generation of the latent diffusion model, and ∈ is the limiting threshold of δ.

3. The image degeneration method according to claim 2, characterized in that: Also includes: When satisfied and When , the minimum value of the adversarial perturbation δ is obtained; D(·) is the distance metric defined by the p-norm, α is the adversarial threshold, when When , the image classification attack method can successfully cause an attack on the potential diffusion model.

4. The image degeneration method according to claim 1, characterized in that: Using the SAM image segmentation model, a key mask m is generated from user-verified sentences and / or pixels in the target image x.

5. An image degeneration device based on adversarial attack, characterized in that: include: A segmentation module is used to segment the target image and obtain a key mask of the target image; The adversarial module is used for adding adversarial perturbations to the key mask to obtain an anti-generative protection image of the target image.

6. The image degeneration device according to claim 5, characterized in that: The adversarial module includes: The perturbation adding module is used to add the adversarial perturbation δ to the key mask m to obtain the anti-generative protection image x' of the target image x, where x'=x+δ, and x' satisfies: is the training target of the latent diffusion model, which generates a synthetic image based on the output of x, y is the given condition for the generation of the latent diffusion model, and ∈ is the limiting threshold of δ.

7. The image degeneration device according to claim 6, characterized in that: The adversarial module includes: The perturbation generation module is used to obtain the minimum adversarial perturbation that satisfies the generation of x'; including: when satisfying and When , the minimum value of the adversarial perturbation δ is obtained; D(·) is the distance metric defined by the p-norm, α is the adversarial threshold, when Image classification attack methods can successfully induce attacks on this latent diffusion model.

8. The image degeneration device according to claim 5, characterized in that: The segmentation module uses the SAM image segmentation model to generate key masks m from user-verified sentences and / or pixels in the target image x.

9. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the computer executable instructions are executed, the image degeneration method based on adversarial attack as described in any one of claims 1 to 4 is implemented.

10. An electronic device, comprising the image de-generation device based on adversarial attack as described in any one of claims 5 to 8.

Images