Restorable image protection method, system, device and medium based on diffusion model

Abstract

This application discloses a method, system, device, and medium for restorable image protection based on a diffusion model. The method obtains an image to be protected and preset watermark information; constructs an adversarial sample generation module based on projected gradient descent, and constructs an adversarial sample purification module based on the diffusion model, including a watermark embedding submodule, wherein the watermark embedding submodule is constructed based on a decoder of a latent diffusion model; inputs the image to be protected into the adversarial sample generation module to generate an adversarial sample image; inputs the adversarial sample image into the adversarial sample purification module for denoising to obtain a restored image, which is the restored image to be protected; and adds the preset watermark information to the restored image through the watermark embedding submodule to obtain a restored image containing the watermark. This application can restore images while ensuring image quality and better protecting image privacy.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, system, device and medium for restoring image protection based on a diffusion model. Background Art

[0002] In the field of deep learning research, generative models, particularly diffusion models, are profoundly changing the paradigm of image creation and processing. Through their unique denoising processes, these models are capable of generating strikingly realistic and creative image content, and have been widely used in professional fields such as art creation, advertising design, and film and television special effects. However, the widespread sharing of image data also brings the risk of privacy leakage, especially for image information involving personal identity and privacy copyright. Unauthorized access and misuse can lead to serious privacy violations. Therefore, how to effectively protect sensitive information in images, prevent unauthorized access, and restore the original image when needed has become a critical research topic in the field of information security.

[0003] As research in deep learning and diffusion models advances, several methods have been proposed for image privacy protection. However, existing image privacy protection methods have various limitations. For example, some methods render encrypted images unusable, or their encrypted information is used only for copyright verification, failing to effectively prevent recognition models from snooping on the image content. While some methods can embed information in images and restore it when necessary, large embedding sizes can compromise the visual quality and detail of the image and reduce robustness against complex attacks (such as compression, cropping, and noise interference). Still other methods protect only facial information, lacking sufficient protection for other images. Furthermore, once generated, adversarial samples are nearly impossible to recover, rendering the images unusable.

[0004] In summary, existing image privacy protection methods are difficult to strike a balance between image quality, image recoverability, and image privacy protection, so the performance of existing image privacy protection methods is not perfect. Summary of the Invention

[0005] This application aims to propose a restorable image protection method, system, device and medium based on a diffusion model, which can restore images and better protect image privacy while ensuring image quality.

[0006] In a first aspect, an embodiment of the present application provides a method for restorable image protection based on a diffusion model, the method comprising:

[0007] Obtain the image to be protected and the preset watermark information;

[0008] Constructing an adversarial sample generation module based on projected gradient descent, and constructing an adversarial sample purification module based on the diffusion model, which includes a watermark embedding submodule, wherein the watermark embedding submodule is constructed based on the decoder of the latent diffusion model;

[0009] Inputting the image to be protected into the adversarial sample generation module to generate an adversarial sample image;

[0010] Inputting the adversarial sample image into the adversarial sample purification module for denoising to obtain a restored image, where the restored image is the restored image to be protected;

[0011] The preset watermark information is added to the restored image through the watermark embedding submodule to obtain a restored image containing the watermark.

[0012] Compared with the prior art, the first aspect of the present application has the following beneficial effects:

[0013] This method obtains an image to be protected and preset watermark information; constructs an adversarial sample generation module based on projected gradient descent, and constructs an adversarial sample purification module based on a diffusion model, including a watermark embedding submodule, wherein the watermark embedding submodule is constructed based on a decoder of a latent diffusion model; inputs the image to be protected into the adversarial sample generation module to generate an adversarial sample image; inputs the adversarial sample image into the adversarial sample purification module for denoising, thereby obtaining a restored image, which is the restored image to be protected; and adds the preset watermark information to the restored image via the watermark embedding submodule to obtain a restored image containing the watermark. Thus, by inputting the image to be protected into the adversarial sample generation module to generate an adversarial sample image, a highly offensive and high-quality adversarial sample image can be generated to protect user privacy and copyright information; by inputting the adversarial sample image into the adversarial sample purification module for denoising, the restored image can be restored to the image to be protected; and by adding the preset watermark information to the restored image via the watermark embedding submodule, a layer of personal copyright information can be further added to the image, so that only authorized recognition models can correctly identify the image information in the image, while unauthorized recognition models cannot. Therefore, this method can restore the image while ensuring the image quality and better protect the image privacy.

[0014] In some embodiments, constructing an adversarial sample generation module based on projected gradient descent includes:

[0015] Gradient-weighted class activation mapping is used to calculate high semantic information weight masks;

[0016] Inputting the high semantic information weight mask into stochastic differential editing to optimize the output result of the stochastic differential editing;

[0017] The projected gradient descent is iteratively updated according to the output result of the optimized random differential editing to construct an adversarial sample generation module.

[0018] In some embodiments, inputting the high semantic information weight mask into stochastic differential editing to optimize the output of the stochastic differential editing includes:

[0019] ;

[0020] ;

[0021] in, represents a high semantic information weight mask, represents the gradient-weighted class activation map, Indicates the The first round of projected gradient descent attack Step sample, Indicates the Round Projection Gradient Descent Attack The output of the random differential editing process, represents the inverse denoising process of random differential editing, Indicates the The first round of random differential editing Step sample, represents the forward noise addition process of random differential editing, Indicates the The first round of random differential editing Step sample, Indicates the initial input random differential editing Step sample.

[0022] In some embodiments, iteratively updating the projected gradient descent according to the output of the optimized stochastic differential editing includes:

[0023] ;

[0024] in, Indicates the Samples of round-projected gradient descent attack, express Projection operation on the sphere, Indicates the Samples of round-projected gradient descent attack, represents the iteration step size, represents the symbolic function, Represents the loss function after adding perturbation to the input The gradient with respect to the input, represents the target classifier, represents the true label.

[0025] In some embodiments, constructing an adversarial sample purification module including a watermark embedding submodule based on a diffusion model includes:

[0026] Build a constrained denoiser based on the Transformer architecture with multiple encoder and decoder stages;

[0027] Obtaining a roughly purified image after roughly purification by a diffusion model;

[0028] Using the result of the constrained denoiser as a guiding condition of a diffusion model, and combining the roughly purified image and the classifier, optimizing the diffusion inverse process of the diffusion model to obtain an optimized conditional diffusion model;

[0029] The optimized conditional diffusion model is connected to the watermark embedding submodule to construct an adversarial sample purification module.

[0030] In some embodiments, the step of using the result of the constrained denoiser as a guiding condition for a diffusion model and combining the roughly cleaned image and the classifier to optimize the diffusion inverse process of the diffusion model comprises:

[0031] ;

[0032] in, represents a classifier, Indicates the The denoised image of the step, Indicates the The denoised image of the step, represents the result of the constrained denoiser, represents a Gaussian distribution, represents the mean, represents a constant that controls the guiding strength, represents the variance, represents the gradient, Represents the distance function for calculating similarity, Represents a roughly cleaned image.

[0033] In some embodiments, after the watermark embedding submodule adds the preset watermark information to the restored image to obtain the restored image containing the watermark, the method further includes:

[0034] Pre-training the watermark extractor to obtain a trained watermark extractor;

[0035] extracting target watermark information from the restored image containing the watermark by using the trained watermark extractor;

[0036] The target watermark information and the preset watermark information are compared and traced.

[0037] In a second aspect, an embodiment of the present application further provides a restorable image protection system based on a diffusion model, the system comprising:

[0038] A data acquisition unit, used to acquire the image to be protected and preset watermark information;

[0039] A module construction unit is configured to construct an adversarial sample generation module based on projected gradient descent, and an adversarial sample purification module based on a diffusion model, the adversarial sample purification module including a watermark embedding submodule, wherein the watermark embedding submodule is constructed based on a decoder of a latent diffusion model;

[0040] An image generating unit, configured to input the image to be protected into the adversarial sample generating module to generate an adversarial sample image;

[0041] An image denoising unit, configured to input the adversarial sample image into the adversarial sample purification module for denoising to obtain a restored image, where the restored image is the restored image to be protected;

[0042] The watermark adding unit is used to add the preset watermark information to the restored image through the watermark embedding submodule to obtain a restored image containing the watermark.

[0043] In a third aspect, an embodiment of the present application also provides an electronic device comprising at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute a restorable image protection method based on a diffusion model as described above.

[0044] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned method for restorable image protection based on a diffusion model.

[0045] It can be understood that the beneficial effects of the above-mentioned second to fourth aspects compared with the relevant technologies are the same as the beneficial effects of the above-mentioned first aspect compared with the relevant technologies. Please refer to the relevant description in the above-mentioned first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0047] Figure 1 1 is a flow chart of an embodiment of a restorable image protection method based on a diffusion model provided by the present application;

[0048] Figure 2 This is a schematic diagram of the overall process of the best embodiment of the restorable image protection method based on the diffusion model provided by the present application;

[0049] Figure 3 This is a flow chart of the adversarial sample generation module in the best embodiment of the restorable image protection method based on the diffusion model provided by this application;

[0050] Figure 4 This is a flow chart of the adversarial sample purification module in the best embodiment of the restorable image protection method based on the diffusion model provided by this application;

[0051] Figure 5 This is a schematic diagram of the watermark embedding and extraction process in the best embodiment of the restorable image protection method based on the diffusion model provided by the present application;

[0052] Figure 6 1 is a schematic diagram of the architecture of a constrained denoiser in a preferred embodiment of the restorable image protection method based on a diffusion model provided by the present application;

[0053] Figure 7 1 is a schematic structural diagram of an embodiment of a restorable image protection system based on a diffusion model provided by the present application;

[0054] Figure 8 It is a structural diagram of an embodiment of the electronic device provided by this application. DETAILED DESCRIPTION

[0055] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0056] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0057] In the description of this application, it should be understood that descriptions involving orientation, such as the orientation or positional relationship indicated by up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0058] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technical personnel in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0059] As research in deep learning and diffusion models advances, several methods have been proposed for image privacy protection. However, existing image privacy protection methods have various limitations. For example, some methods render encrypted images unusable, or their encrypted information is used only for copyright verification, failing to effectively prevent recognition models from snooping on the image content. While some methods can embed information in images and restore it when necessary, large embedding sizes can compromise the visual quality and detail of the image and reduce robustness against complex attacks (such as compression, cropping, and noise interference). Still other methods protect only facial information, lacking sufficient protection for other images. Furthermore, once generated, adversarial samples are nearly impossible to recover, rendering the images unusable.

[0060] In summary, existing image privacy protection methods are difficult to strike a balance between image quality, image recoverability, and image privacy protection, so the performance of existing image privacy protection methods is not perfect.

[0061] To address the problem that the above-mentioned existing image privacy protection methods are difficult to strike a balance between image quality, image recoverability and image privacy protection, this application proposes a restorable image protection method, system, device and medium based on a diffusion model.

[0062] Reference Figure 1 , is a flow chart of a restorable image protection method based on a diffusion model provided by an embodiment of the present application. The restorable image protection method based on a diffusion model is applied to an electronic device, which may be a server or a mobile terminal. Figure 1 As shown, the restorable image protection method based on the diffusion model may include the following steps:

[0063] Step S100: obtaining an image to be protected and preset watermark information;

[0064] Step S200: constructing an adversarial sample generation module based on projected gradient descent, and constructing an adversarial sample purification module including a watermark embedding submodule based on the diffusion model, wherein the watermark embedding submodule is constructed based on the decoder of the latent diffusion model;

[0065] Step S300: Input the image to be protected into the adversarial sample generation module to generate an adversarial sample image;

[0066] Step S400: Input the adversarial sample image into the adversarial sample purification module for denoising to obtain a restored image, which is the restored image to be protected;

[0067] Step S500: Add the preset watermark information to the restored image through the watermark embedding submodule to obtain a restored image containing the watermark.

[0068] In this embodiment, an adversarial sample generation module and preset watermark information are obtained; an adversarial sample generation module is constructed based on projected gradient descent; and an adversarial sample purification module is constructed based on a diffusion model, including a watermark embedding submodule. The watermark embedding submodule is constructed based on a decoder of a latent diffusion model. The image to be protected is input into the adversarial sample generation module to generate an adversarial sample image. The adversarial sample image is input into the adversarial sample purification module for denoising to obtain a restored image, which is the restored image to be protected. The preset watermark information is added to the restored image via the watermark embedding submodule to obtain a restored image containing the watermark. Thus, by inputting the image to be protected into the adversarial sample generation module to generate an adversarial sample image, a highly offensive and high-quality adversarial sample image can be generated to protect user privacy and copyright information. By inputting the adversarial sample image into the adversarial sample purification module for denoising, the restored image can be restored to the image to be protected. By adding the preset watermark information to the restored image via the watermark embedding submodule, a layer of personal copyright information can be further added to the image. Only authorized recognition models can correctly recognize the image information in the image, while unauthorized recognition models cannot. Therefore, this embodiment can restore the image while ensuring the image quality and better protect the image privacy.

[0069] The preset watermark information may be a string of watermark information customized by the user.

[0070] The above-mentioned adversarial sample generation module based on projected gradient descent can be an adversarial sample generation module constructed by using projected gradient descent, or an adversarial sample generation module constructed based on optimized projected gradient descent, which is not specifically limited in this embodiment.

[0071] The above-mentioned adversarial sample purification module based on the diffusion model and including the watermark embedding submodule can be an adversarial sample purification module based on the diffusion model and the watermark embedding submodule, or an adversarial sample purification module based on the diffusion model and the watermark embedding submodule after introducing conditions. This embodiment does not make specific limitations.

[0072] In some embodiments, an adversarial example generation module is constructed based on projected gradient descent, including:

[0073] Gradient-weighted class activation mapping is used to calculate high semantic information weight masks;

[0074] Inputting high semantic information weight mask into stochastic differential editing to optimize the output of stochastic differential editing;

[0075] The projected gradient descent is iteratively updated according to the output results of the optimized stochastic differential editing to construct the adversarial sample generation module.

[0076] In this embodiment, a high-semantic-information weighted mask is calculated using gradient-weighted class activation mapping (GLM). This high-semantic-information weighted mask is then fed into stochastic differential editing to optimize the output of the stochastic differential editing. Projected gradient descent is then iteratively updated based on the optimized output of the stochastic differential editing to construct an adversarial example generation module. In this way, by optimizing GLM and stochastic differential editing together, the optimized projected gradient descent can generate highly aggressive and high-quality adversarial example images.

[0077] In some embodiments, inputting a high semantic information weight mask into stochastic differential editing to optimize the output of stochastic differential editing includes:

[0078] ;

[0079] ;

[0080] in, represents a high semantic information weight mask, represents the gradient-weighted class activation map, Indicates the The first round of projected gradient descent attack Step sample, Indicates the Round Projection Gradient Descent Attack The output of the random differential editing process, represents the inverse denoising process of random differential editing, Indicates the The first round of random differential editing Step sample, represents the forward noise addition process of random differential editing, Indicates the The first round of random differential editing Step sample, Indicates the initial input random differential editing Step sample.

[0081] In some embodiments, iteratively updating the projected gradient descent according to the output of the optimized stochastic differential editing includes:

[0082] ;

[0083] in, Indicates the Samples of round-projected gradient descent attack, express Projection operation on the sphere, Indicates the Samples of round-projected gradient descent attack, represents the iteration step size, represents the symbolic function, Represents the loss function after adding perturbation to the input The gradient with respect to the input, represents the target classifier, represents the true label.

[0084] In some embodiments, an adversarial sample purification module including a watermark embedding submodule is constructed based on a diffusion model, including:

[0085] Build a constrained denoiser based on the Transformer architecture with multiple encoder and decoder stages;

[0086] Obtaining a roughly purified image after roughly purification by a diffusion model;

[0087] The results of the constrained denoiser are used as the guiding conditions of the diffusion model, and combined with the roughly purified image and the classifier, the diffusion inverse process of the diffusion model is optimized to obtain the optimized conditional diffusion model.

[0088] The optimized conditional diffusion model is connected with the watermark embedding submodule to construct the adversarial sample purification module.

[0089] In this embodiment, a roughly purified image is obtained after being roughly purified using a diffusion model. A constrained denoiser is constructed based on a Transformer architecture with a multi-stage encoder and decoder. The results of the constrained denoiser are used as guiding conditions for the diffusion model. Combined with the roughly purified image and a classifier, the diffusion inverse process of the diffusion model is optimized to obtain an optimized conditional diffusion model. This optimized conditional diffusion model is then connected to the watermark embedding submodule to construct an adversarial sample purification module. By using the results of the constrained denoiser as guiding conditions for the diffusion model, the effectiveness and efficiency of adversarial purification can be improved, thereby enhancing the quality of the purified image.

[0090] In some embodiments, the results of the constrained denoiser are used as guiding conditions for the diffusion model, and combined with the roughly cleaned image and the classifier, the diffusion inverse process of the diffusion model is optimized, including:

[0091] ;

[0092] in, represents the classifier, Indicates the The denoised image of the step, Indicates the The denoised image of the step, represents the result of the constrained denoiser, represents a Gaussian distribution, represents the mean, represents a constant that controls the guiding strength, represents the variance, represents the gradient, Represents the distance function for calculating similarity, Represents a roughly cleaned image.

[0093] In this embodiment, by using the results of the constrained denoiser and the roughly purified image, and calculating parameters based on the SSIM distance function to guide the subsequent back diffusion process, a purified image (i.e., a restored image) with better denoising effect and better picture quality can be obtained.

[0094] In some embodiments, after the watermark embedding submodule adds the preset watermark information to the restored image to obtain the restored image containing the watermark, the method further includes:

[0095] Pre-training the watermark extractor to obtain a trained watermark extractor;

[0096] Extract the target watermark information from the restored image containing the watermark through the trained watermark extractor;

[0097] Compare the target watermark information with the preset watermark information to trace the source.

[0098] In this embodiment, a pre-trained watermark extractor is obtained. The trained watermark extractor is then used to extract the target watermark information from the restored watermarked image. The target watermark information is then compared with the pre-set watermark information for source tracing. This allows for identity identification and source tracing using watermarks, making it highly practical in scenarios such as image copyright protection and sensitive data sharing.

[0099] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0100] With the rapid development of digital technology and the internet, images have become a primary vehicle for information dissemination, widely used in fields such as social media, medical diagnosis, and public security. Within deep learning research, generative models, particularly diffusion models, are profoundly changing the paradigm of image creation and processing. Through their unique denoising processes, these models are capable of generating strikingly realistic and creative image content, and have been widely used in professional fields such as art, advertising design, and film and television special effects. However, the widespread sharing of image data also carries the risk of privacy leakage, particularly for images involving personal identity and copyright. Unauthorized access and misuse can lead to serious privacy violations. Therefore, how to effectively protect sensitive information in images, prevent unauthorized access, and restore the original image when needed, has become a critical research topic in the field of information security. Of particular concern is the inherent properties of diffusion models, which can be used as both a defensive tool and an offensive tool. On the one hand, their powerful image reconstruction capabilities can be used to develop novel defensive solutions; on the other hand, attackers can exploit diffusion models to generate more subtle adversarial examples. This double-edged sword nature of the technology makes the development of image protection methods that balance defensive effectiveness and practicality particularly urgent.

[0101] Existing image information protection technologies can be mainly divided into the following aspects:

[0102] 1. Image encryption technology: Image data is encrypted using an encryption algorithm. Only users holding the key can decrypt and view it. Users without the key cannot view the information.

[0103] 2. Digital watermark technology: embeds identification information invisible to the naked eye into the image. This identification information can be detected by specific detection methods and used for copyright protection or authentication, protecting the image information to a certain extent.

[0104] 3. Reversible Data Hiding (RDH): Embed private information into an image while maintaining a certain image quality, and restore the image when needed.

[0105] 4. Adversarial sample generation: Using adversarial attack technology, tiny perturbations are added to the image, making it impossible for the machine learning model to correctly recognize it and difficult for the human eye to detect it.

[0106] However, existing image information protection technologies have different limitations:

[0107] 1. It is difficult to balance privacy protection and image recoverability: Images encrypted by existing image encryption technologies and digital watermarking technologies cannot be used directly, or their encrypted information is only used for copyright identification but cannot effectively prevent recognition models from snooping on image content.

[0108] 2. There is a trade-off between embedding capacity and image quality: Although existing RDH technology can embed information in images and restore them when necessary, it often affects the visual quality and details of the image when the embedding capacity is large, and is less robust against complex attacks (such as compression, cropping, and noise interference).

[0109] 3. Shortcomings of diffusion model-based methods: Although diffusion models have great potential in image generation and editing, some existing implementations are prone to introducing noise or distortion during the restoration process, resulting in the generated images failing to fully match the original images in terms of detail and overall quality. In addition, some solutions have high requirements for model structure and training data during implementation, resulting in high costs and insufficient universality in practical applications.

[0110] 4. Limitations of adversarial sample generation technology: Existing adversarial sample generation technology and research are mostly used only to protect facial information, but are insufficient for protecting other images. Furthermore, adversarial samples in existing adversarial sample protection methods are almost impossible to recover once generated, making the images unusable.

[0111] To address the above shortcomings, this embodiment provides a restorable image protection method based on a diffusion model, the main purpose of which is to:

[0112] 1. Effective image protection: This embodiment generates adversarial samples based on a diffusion model, which can protect the privacy of all types of images. It can be applied in a wider range of scenarios. In addition, the adversarial nature of the generated adversarial samples is difficult to remove by other unauthorized recognition models through purification, thereby achieving more effective image information protection.

[0113] 2. High-quality restorability: While fully hiding the original image information from the recognition model, the diffusion model denoising and restoration mechanism designed by the method of this embodiment can ensure that the visual effect of the restored image is the same as the original. Figure 1 The feature information of the recognition model remains consistent with the original image, taking into account both privacy protection and reusability.

[0114] 3. Enhanced robustness and fault tolerance: The method of this embodiment can maintain a strong recovery effect for common image processing (such as compression, cropping, and noise interference), ensuring that the image protection mechanism has good robustness in various practical application scenarios.

[0115] 4. Reduce model training and usage costs: By improving the model architecture and training strategy, while maintaining protection and recovery performance, the system implementation is simplified, improving the usability and universality of practical applications. Compared with existing methods, this embodiment can complete image noise protection and denoising for reuse in a shorter time and cost.

[0116] 5. Balancing privacy protection and image practicality: This embodiment's method uses noise-adding technology to add adversarial properties to images, making them undetectable and unrecognizable by unauthorized recognition models. Compared to existing methods, this embodiment's method can ensure higher visual quality for generated images without affecting normal viewing and use. This ensures that protected images meet security requirements while maintaining a certain level of usability.

[0117] 6. Copyright protection of reused images: The method of this embodiment uses digital watermarking technology to add an implicit digital watermark to the reused image generated by the denoising part of the diffusion model. Through the recognition of the digital watermark, a layer of personal copyright information can be added to the generated reused image. Compared with other methods, the method of this embodiment can provide a more complete information protection process.

[0118] Specifically, refer to Figure 2 , the method of this embodiment consists of two modules: adversarial sample generation and watermarked adversarial sample purification. The image owner is the user of the system of this method. The user passes the original image through the adversarial sample generation module to obtain an adversarial image. The image is visually indistinguishable from the original image, but it will be misidentified using an unauthorized recognition model (such as identifying a panda image as a monkey). The user can customize a string of binary watermark information and use a pre-trained watermark extractor to add this watermark information to the adversarial sample purification module. For the recognition model authorized by the user, after the adversarial image passes through the adversarial sample purification module, a restored image with a watermark will be obtained. The image is visually indistinguishable from the original image and can be correctly recognized by the recognition model. After passing the image through the watermark extractor, the watermark information can be detected for comparison and tracing.

[0119] It should be noted that the recognition model of this embodiment can adopt an image recognition model that is well known to those skilled in the art and can recognize image information, which is not described in detail in this embodiment.

[0120] The user processes the image through the adversarial sample generation module, making it impossible for the image recognition system to correctly identify the generated adversarial sample image without affecting the visual quality of the image, thereby protecting the user's privacy and copyright information. For the authorized image recognition system, the watermark-containing adversarial sample purification module purifies the restored image so that it can be correctly recognized, ensuring that the restored image is still usable. At the same time, the restored image is added with implicit watermark information. The detection method can detect the exclusive watermark copyright information added by the user, providing a full-process protection means for the use of the image. The method of this embodiment specifically includes the following contents:

[0121] 1. Adversarial sample generation module.

[0122] Reference Figure 3 The original image (i.e., the image to be protected) is first evaluated using the GradCAM method to calculate its weight for the correct classification of the classifier. Based on the contribution to the correct classification, a heatmap mask is formed. Based on the mask information, less SDEdit diffusion cleanup is performed on the parts with greater contributions, while more SDEdit diffusion cleanup is performed on the parts with less contributions. The PGD weights are then updated based on the cleaned image to add aggressiveness to the image. After n iterations of this cycle, an adversarial sample image with the same visual effect as the original image is obtained.

[0123] This embodiment combines Gradient Weighted Class Activation Mapping (GradCAM) and Stochastic Differential Editing (SDEdit) to optimize the Projected Gradient Descent (PGD) attack method to generate highly aggressive and high-quality adversarial sample images. PGD is an adversarial attack method based on gradient iteration. Its core idea is to model the adversarial attack as a saddle point optimization problem (min-max optimization):

[0124] (1);

[0125] Slightly different from the Fast Gradient Sign Method (FGSM), this method model optimizes the perturbation by iterative , so that the model can be used in adversarial samples Maximize the loss on the , and optimize the model parameters through adversarial training , improve the model's robustness to perturbations. The key to PGD is to ensure that the perturbations always meet the preset constraints through projection operations and multiple iterations, each iteration taking a small step, that is, , It represents a preset constraint condition set. The preset constraint conditions can be changed according to actual conditions and are not specifically limited or described in this embodiment.

[0126] Let the original image be , the true label is , the iterative update formula of PGD is:

[0127] (2);

[0128] in, Indicates the The disturbance of the step, represents the step size of each iteration, Is a sign function, indicating the sign of the gradient direction, Represents the loss function after adding perturbation to the input The gradient with respect to the input, represents the projection operation, projecting into the perturbation limit, is the original input, is the true label.

[0129] SDEdit is an image generation and editing technology based on a diffusion model. It aims to generate high-quality images by adding noise and then removing the noise. Its core idea is to preserve the structural information of the input image by simulating the diffusion process while generating new content that matches the target text or style. The core process of SDEdit is divided into the following steps:

[0130] (1) Noise injection: Add random noise to the input image so that it gradually becomes a pure noise image. That is, diffuse the original distribution .

[0131] (2) Reverse denoising: gradually remove noise through the diffusion model, and generate a new image in combination with the target condition. Run the reverse denoising process in the parameterized diffusion model , to get the edited sample.

[0132] SDEdit updates hidden variables in the following ways :

[0133] (3);

[0134] in, Indicates the current diffusion state, represents the updated one step (closer to the original image), represents the noise term predicted by the diffusion model, represents the parameters in the diffusion process, Represents the total time steps.

[0135] The process of SDEdit can be described as a formula:

[0136] (4);

[0137] That is, given a raw input, SDEdit first executes Step forward process, from get , then execute Step 2 reverse process to establish the input distribution (i.e. deviation ) and the original distribution (Because of the That is the , to more specifically represent the initial input, we use ), which can be used for synthesis and editing from sketches to real pictures, and can also be used for tasks such as adversarial sample purification. Indicates the execution of the noise adding time step The reverse process.

[0138] GradCAM is a gradient-based class activation mapping technique used to visualize the image regions that convolutional neural networks focus on during prediction. It computes the gradient of the target category with respect to the last convolutional feature map (which contains both high-level features and preserves spatial information). After forward-propagating the image through the network, it calculates a score for the explained category and computes the gradient of this score with respect to the output of the selected convolutional layer. For each channel in the convolutional layer, it weights the channels using the global average of the gradients to generate a heatmap. This approach allows for intuitive visualization of the key regions where the model's decisions depend.

[0139] Assume that the feature map of the last convolutional layer is (No. channels), the gradient of the target category is , weight The calculation is as follows:

[0140] (5);

[0141] Heatmap for:

[0142] (6);

[0143] Based on the above principles, the adversarial sample generation module of the method in this embodiment is based on PGD and optimized using SDEdit and GradCAM to improve image quality and attack generalization, ensuring the naturalness and concealment of the generated images.

[0144] For images that need to be processed , get its true label and target classifier , need to generate adversarial sample images . In the original step size , In the PGD iteration, The update of the step can be expressed as follows:

[0145] (7);

[0146] in, express Projection operation on the ball. From the above formula, we can see that the gradient will guide the sample Far from the decision boundary of the recognition, deviating from the true distribution Due to the deviation from the distribution, the previous PGD method is easily purified by some distribution-based methods and has poor generalization, which greatly limits the attack strength.

[0147] In order to avoid the above problems, the method of this embodiment does not use the original , and use the image purified by SDEdit Calculate the loss function of PGD, that is, use SDEdit Replace the formula (7) . represents the image obtained by formula (4), because it is the input conduct The diffusion process is obtained, so it is recorded as .

[0148] However, previous methods using SDEdit failed to consider the centralization of image semantic information and instead used SDEdit to cleanse the entire image. However, overly strong cleansing effects can lead to excessively large gradients in the PGD calculations during each iteration, making it difficult to find the optimal saddle point solution for PGD, thus affecting attack accuracy and image quality. Therefore, the method in this embodiment uses GradCAM to calculate masks for interpolation, optimizing the SDEdit method and avoiding a strong cleansing effect.

[0149] (8);

[0150] After obtaining the high semantic information weight mask through formula (8), the SDEdit method is optimized using formula (9):

[0151] (9);

[0152] in, represents the inverse denoising process of the diffusion model, represents the forward noise addition process of the diffusion model, Indicates the The first round of SDEdit Step sample, Indicates the The first round of SDEdit Step sample, Indicates the Round PGD attack Samples from a round of SDEdit (i.e., the output of SDEdit). Specifically, SDEdit with the added mask uses a smaller cleanup margin in areas with higher weights and a larger cleanup margin in areas with lower weights. This method ensures the effectiveness of the attack on areas with high semantic weight while avoiding image quality degradation caused by over-cleaning.

[0153] Compared with the traditional PGD method, the method of this embodiment combines the diffusion model in the PGD attack, and uses the excellent image generation and purification capabilities of the diffusion model to prevent the traditional PGD attack from guiding the image to deviate too much from the classification decision boundary and the true distribution of the image, thereby causing the image to be easily purified and the attack generalization to be poor. In the diffusion model, the method of this embodiment innovatively combines the GradCAM method to prevent the existing method from excessively purifying the image in the diffusion step, resulting in excessively large gradients calculated in each PGD iteration, making it difficult to solve the optimal saddle point solution, thereby affecting the accuracy of the attack and the image quality. The method of this embodiment can ensure that the generated adversarial image has high aggressiveness and high generalization of the attack capability while ensuring the visual quality of the image.

[0154] 2. Watermarked adversarial sample purification module.

[0155] Reference Figure 4 In this embodiment, the method passes the adversarial sample image through a pretrained constrained denoiser to obtain a denoised prior image (i.e., the output of the constrained denoiser) that has been pre-trained to remove the adversarial effects. This denoised prior image serves as the input to the diffusion cleansing process. A forward diffusion process is used to obtain a noisy image. This noisy image is then passed through the reverse process of the diffusion model to remove the noise, resulting in a roughly denoised image (i.e., a roughly cleansed image). Using the denoised prior image from the constrained denoiser and the roughly denoised image, the SSIM distance function is used to calculate diffusion guidance parameters. The calculated parameters guide the reverse process of the diffusion model to obtain a cleansed result (i.e., a restored image).

[0156] This embodiment adopts the watermark embedded denoising constraint guided diffusion purification (D-GDP) method as the method of the adversarial sample purification module. By combining the denoising prior constraint and the guided diffusion model, the effect and efficiency of adversarial purification are improved. At the same time, after purifying the image, the watermark embedding submodule is connected to embed the invisible watermark into the restored image, and the watermark information can be extracted by the watermark extractor, so that the copyright owner of the image can trace the watermark of the purified non-adversarial image. Figure 5 The watermark embedding part is integrated into the adversarial sample purification module. Using a fine-tuned LDM watermark generator, an image with a specific binary-encoded watermark information can be generated. After passing the purified image through the watermark generator, an image embedded with the watermark can be obtained. The image has the same visual effect as the original image. After passing the watermarked image through the watermark extractor, the binary watermark information embedded in the image can be obtained, thereby tracing the watermark. Specifically:

[0157] Adversarial sample purification based on a diffusion model is a defense technique that uses generative models to remove adversarial perturbations and restore clean samples. Its core idea is to destroy the perturbation patterns in adversarial samples through the forward and reverse processes of the diffusion model while preserving the semantic information of the original data.

[0158] During the forward diffusion process, adversarial perturbations are gradually overwritten by added noise. Because perturbations are typically high-frequency, the addition of noise can disrupt their structure. The reverse denoising process leverages the diffusion model's ability to model data distributions, generating samples that conform to the natural data distribution, thereby eliminating any residual perturbations.

[0159] (10);

[0160] From formula (10), we can see that in the forward diffusion process, the clean data distribution p(x) and the adversarial perturbation data distribution q(x) are closer.

[0161] In formula (10) represents the KL divergence, Indicates the clean data p(x) in the forward diffusion process The distribution of steps, Indicates the adversarial perturbation data q(x) in the forward process step distribution, the entire inequality shows that as the diffusion time As increases, the KL divergence decreases or remains unchanged. The KL divergence measures the "difference" between two probability distributions, which is represented here. Relative to The smaller the information loss, the closer the two distributions are. Therefore, this formula shows that as the diffusion process proceeds, the distributions between the two will become closer and closer.

[0162] During the forward pass of diffusion, noise is added to the image time-step by time-step:

[0163] (11);

[0164] In formula (11), is the total time step, the original image is ,go through After the diffusion of the time step, a pure noise image can be obtained According to the research on diffusion model, using standard Gaussian as prior distribution, then It can be expressed by the following formula:

[0165] (12);

[0166] in, represents a Gaussian distribution, is a time step Related preset constants, Represents the identity matrix. By combining formula (11) and formula (12), we can get:

[0167] (13);

[0168] in By using formula (13), we can get the formula for adding closed-loop noise in the image domain:

[0169] (14);

[0170] In the reverse process of diffusion, the neural network predicts the noise Let's learn how to reverse the forward process:

[0171] (15);

[0172] During the entire diffusion process, the forward process adds enough Gaussian noise to the adversarial sample, making the original noise smaller than the added noise. During the reverse denoising process, the added noise can be removed together with the original noise, thus purifying the adversarial nature of the image.

[0173] The constrained denoiser is a preprocessing module within the D-GDP framework. Its primary function is to initially remove high-frequency noise from adversarial examples, providing input that more closely resembles a natural distribution for the subsequent diffusion purification process. Its design goals include: training a specialized model to filter high-frequency components of an image (such as edges and textures) where adversarial perturbations are concentrated; and employing a lightweight architecture to reduce computational overhead during the preprocessing phase, ensuring real-time performance of the overall defense process and improving the efficiency of diffusion purification.

[0174] Reference Figure 6 The constrained denoiser is based on a multi-stage encoder-decoder Transformer architecture and is trained by adding a variety of high-frequency noises (such as Gaussian noise, Poisson noise, etc.) to images to optimize its denoising capabilities. Specifically:

[0175] This example uses the Transformer fused with a U-net architecture as a pre-constrained image denoiser. In the encoder, the input noisy image (H×W×3) passes through a Dconv layer (depthwise convolutional layer), resulting in an H×W×C output. Features are then extracted through the Transformer L1 block, and the feature map is downsampled and fed into the next stage. As the feature map passes through the Transformer layers, it gradually becomes H / 8×W / 8×8C. Using a pyramid-like feature extraction structure, the resolution decreases while the number of channels increases. In the decoder, upsampling begins from the bottom up, with each layer employing concatenation and skip connections to fuse features from the corresponding encoder stage. The image is then dimensionalised through a Conv layer, fed into the corresponding Transformer block, and finally fed through a Dconv layer (depthwise convolutional layer) to restore the image to a H×W×3 denoised image.

[0176] This embodiment introduces a multi-layer Transformer block after downsampling, which can extract features from multiple scales, such as low-level perceptual details (edges, textures) and high-level semantic information (object shape, structure), extracting structural features of real images and helping to remove local random noise. The Transformer has a global attention mechanism that can model long-range dependencies. It is more perceptive of contextual information than traditional convolution and performs better in recovering large-area noise and texture. Leveraging this advantage, the constrained denoiser can remove noise based on information from the entire image, which is difficult due to the traditional method of using local filters.

[0177] The denoising prior obtained by constraining the denoiser can provide a reliable direction for the subsequent process to calculate reliable gradients in the reverse process, provide better initialization for the subsequent process, and improve the efficiency of the model.

[0178] Gradient guidance is introduced in the diffusion model's purification of adversarial examples. This guidance enhances control over the denoising direction, making the resulting purified samples closer to the original natural image distribution. Compared to unguided diffusion purification, this method effectively removes adversarial artifacts. Furthermore, it preserves more original information during the denoising process, avoiding information loss due to excessive denoising that can render the classifier incapable of recognizing the purified image.

[0179] Results using the Constrained Denoiser As a condition, and introduce a rough clean image , which can be expressed by formula (16):

[0180] (16);

[0181] According to the research, the conditions of use and classifier , combined with formula (15), the formula for the guided diffusion reverse process is:

[0182] (17);

[0183] in, is a constant that controls the strength of the guidance, Indicates that the conditional classifier is that the image belongs to the category The gradient of the log-likelihood of , For the time step The associated noise standard deviation, represents the mean, represents the variance, and Actually, there are two neural networks.

[0184] The result of the constrained denoiser As a condition, substitute into the above formula ,get:

[0185] (18);

[0186] In the above formula (18), It cannot be calculated directly, so the probability approximation is made using formula (19):

[0187] ;

[0188] in, is a normalization factor that can be eliminated in subsequent processes. Indicates measurement and In the method of this embodiment, the distance function is the structural similarity SSIM.

[0189] By using the proof of formula (19) and combining it with formula (18), we can obtain the guidance calculation formula of the method of this embodiment:

[0190] (20);

[0191] In the watermark embedding part, the method of this embodiment uses the decoder (i.e., the watermark embedding submodule) of the latent diffusion model (LDM) obtained by fine-tuning the Stable Signature method (source: arXiv:2303.15435) to embed an invisible traceable watermark into the image. This watermark still has good robustness when the image is rotated, cropped, and other common destruction methods. At the same time, the watermark extractor obtained can be used to extract the image watermark information, thereby realizing the traceability of the image watermark.

[0192] Specifically, we first use the HiDDeN method to jointly optimize the watermark encoder and extract the network Parameters of the pre-trained watermark extractor . For watermark encoder , input a picture without watermark With length A string of watermark information The watermark encoder generates a watermarked image. ,in is a factor, Is a Residual images of uniform size. It is a collection of common image editing processes, including cropping and compression, using extraction networks Extract watermark information after processing the watermarked image , using message loss To optimize the parameters:

[0193] (twenty one);

[0194] In LDM, the decoder Make fine adjustments, Represents the sigmoid activation function, which embeds a recognizable watermark into the image while generating it. The fine-tuning process is divided into two parts: one is the watermark information loss, and the other is the reconstructed image perception loss.

[0195] In the watermark information loss part, first preset a watermark information , then the training images Input to the LDM encoder In the diffusion process, we get an activation map latent space vector , then, the decoder reconstructs the image , and use the watermark extractor to extract the watermark information of the reconstructed image , watermark information loss Same as formula (21), and Binary Cross Entropy (BCE) between .

[0196] In the reconstructed image perception loss part, the original decoder is used Raw watermark-free images , then consider the above reconstructed image With no watermark Watson-VGG image perception loss between , allowing the decoder to learn brightness and contrast masking to improve the quality of the watermarked reconstructed image. Watson-VGG image perceptual loss is a prior art and will not be described in detail in this embodiment.

[0197] Optimize the watermark generation decoder using the AdamW optimizer in several backpropagation steps The weight of is used to minimize the weighted loss function of watermark information loss and reconstructed image perception loss:

[0198] (twenty two);

[0199] The entire adversarial sample purification module combines noise reduction priors, diffusion models, and gradient guidance to address the problems of traditional diffusion purification methods that lack reliable guidance, resulting in general purification effects and low efficiency. At the same time, it embeds detectable watermark information into the image after purification, providing an efficient and robust solution for adversarial defense.

[0200] The three-stage synergy of pre-processing denoising constraints, a full diffusion process, and semantic guidance based on the SSIM distance achieves efficient purification of adversarial examples. The constrained denoiser attenuates high-frequency perturbations and provides a better initial value for subsequent diffusion. The diffusion model, through a guidance strategy, preserves high-level semantics, ultimately generating a robust purified image in a small number of steps and a short time.

[0201] Compared to traditional diffusion purification methods, this embodiment incorporates a constrained denoiser and semantic guidance based on the SSIM distance. The denoised prior image generated by the constrained denoiser weakens the high-frequency perturbations of the adversarial image, providing a more optimal initial value to the diffusion model. This effectively reduces the diffusion step size and, therefore, the time required for image purification. Furthermore, this embodiment uses an SSIM distance function between the denoised prior image (i.e., the result of the constrained denoiser) and the roughly denoised image (i.e., the roughly purified image) to calculate parameters, guiding the subsequent back-diffusion process. This results in a purified image with improved denoising and higher quality.

[0202] The watermarked image obtained using the watermark embedding module of this embodiment has excellent image quality, with a visual effect that is indistinguishable from the original image. Furthermore, even after undergoing various image processing techniques such as rotation, compression, and cropping, the embedded watermark information can still be accurately detected. Compared to existing image information protection methods, this embodiment's method deeply combines adversarial attacks, adversarial defenses, and watermark protection to achieve full-process protection of privacy or copyright information, providing a highly effective, efficient, and robust solution for image information security.

[0203] Compared with existing image security protection technologies, this embodiment has the following advantages:

[0204] 1. Image attack protection: The diffusion-guided PGD attack module combined with Grad-CAM masks proposed in this embodiment not only uses the PGD method to obtain attack gradients during adversarial perturbation generation, but also utilizes the reverse sampling process of the diffusion model to make the perturbations more closely resemble natural image distributions, improving the naturalness and concealment of images, and increasing the attack success rate and deceptiveness of adversarial examples. The attack success rate of this embodiment reached 95.2% on the ImageNet benchmark, a 4.8% improvement over the existing Diff-PGD method. It also demonstrates strong generalization and transferability across multiple classification models (such as ResNet50, ResNet101, and VGG).

[0205] 2. In terms of image visual quality: The semantic weight heat map generated by Grad-CAM in this embodiment differentially regulates the perturbation intensity of different areas of the image. That is, weaker perturbations are applied to semantically important areas, and stronger perturbations are applied to non-important areas. This method effectively avoids the destruction of key semantic information of the image, thereby improving the perceptual quality of the image while maintaining the attack capability. By regulating the perturbation area through semantic masks and combining the generation capability of the diffusion model, the attack samples maintain extremely high visual fidelity. The PSNR and SSIM indicators reach 31.68 and 0.8793 respectively, and the image is almost imperceptible to the human eye. This embodiment uses a pre-trained lightweight Transformer network as a denoising prior to perform preliminary denoising on the adversarial sample; then the denoising result is used as the initial input of the diffusion model purification process to guide the inverse diffusion process to quickly approach the original image distribution; finally, structural similarity evaluation indicators such as SSIM are introduced as purification guidance functions to improve the quality and efficiency of the purified image.

[0206] 3. In terms of purification defense capabilities and efficiency: By combining noise reduction prior initialization with semantically guided sampling, the number of backward steps required by the diffusion model is significantly reduced, improving purification speed. Actual measurements show that the number of diffusion steps required is only one-quarter or even one-fifth of that of traditional diffusion purification. Compared with methods such as DiffPure and GDMP, the D-GDP purification module of this embodiment improves purification accuracy by an average of 2.5% to 3.5%. Furthermore, purification time on the ImageNet dataset is reduced to approximately 18.4 seconds (compared to over 70 seconds for the original model), demonstrating strong real-time application capabilities.

[0207] 4. In terms of system integrity: This embodiment consists of an "adversarial sample generation module" and a "watermarked adversarial sample purification module" connected in series. This structure can generate adversarial samples that are consistent with the visual effects of the original image. Unauthorized persons cannot accurately identify the image content of the adversarial sample through the model, thereby protecting the image content; authorized persons can restore the image structure through the purification module to achieve controllable reuse of the image; authorized persons can also generate a restored image with a watermark through the watermark module to achieve image identity recognition and traceability. Therefore, the "closed-loop" design of counterattack and purification recovery proposed in this embodiment solves the problem of functional fragmentation of previous solutions. The system can not only protect images from being recognized, but also restore them to an approximate original state. At the same time, watermarks can be used for identity recognition and traceability. It is extremely practical in scenarios such as image copyright protection and sensitive data sharing.

[0208] Reference Figure 7 The embodiment of the present application further provides a restorable image protection system based on a diffusion model, which includes a data acquisition unit 100, a module construction unit 200, an image generation unit 300, an image denoising unit 400, and a watermark adding unit 500, wherein:

[0209] The data acquisition unit 100 is used to acquire the image to be protected and the preset watermark information;

[0210] A module construction unit 200 is configured to construct an adversarial sample generation module based on projected gradient descent, and an adversarial sample purification module including a watermark embedding submodule based on a diffusion model, wherein the watermark embedding submodule is constructed based on a decoder of a latent diffusion model;

[0211] The image generation unit 300 is configured to input the image to be protected into the adversarial sample generation module to generate an adversarial sample image;

[0212] An image denoising unit 400 is configured to input the adversarial sample image into the adversarial sample purification module for denoising to obtain a restored image, where the restored image is the restored image to be protected;

[0213] The watermark adding unit 500 is used to add preset watermark information to the restored image through the watermark embedding submodule to obtain a restored image containing the watermark.

[0214] It should be noted that since the restorable image protection system based on a diffusion model in this embodiment and the above-mentioned restorable image protection method based on a diffusion model are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the system embodiment and will not be described in detail here.

[0215] Reference Figure 8 , an embodiment of the present application further provides an electronic device, the electronic device comprising:

[0216] at least one memory;

[0217] at least one processor;

[0218] at least one program;

[0219] The programs are stored in the memory, and the processor executes at least one program to implement the above-mentioned restorable image protection method based on the diffusion model in the present disclosure.

[0220] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a car computer, etc.

[0221] The electronic device according to the embodiment of the present application is described in detail below.

[0222] The processor 1600 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0223] Memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 1700 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program code is stored in memory 1700 and is called by processor 1600 to execute the restorable image protection method based on the diffusion model of the embodiments of this disclosure.

[0224] Input / output interface 1800, used for information input and output;

[0225] Communication interface 1900, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0226] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0227] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0228] An embodiment of the present disclosure further provides a storage medium, which is a computer-readable storage medium and stores computer-executable instructions. The computer-executable instructions are used to enable a computer to execute the above-mentioned restorable image protection method based on the diffusion model.

[0229] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0230] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0231] Those skilled in the art will understand that the technical solutions shown in the drawings do not constitute a limitation on the embodiments of the present disclosure, and may include more or fewer steps than shown in the drawings, or a combination of certain steps, or different steps.

[0232] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0233] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0234] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0235] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0236] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0237] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0238] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0239] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the purpose of the present application.

[0240] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A restorable image protection method based on a diffusion model, characterized in that: The method comprises: Obtain the image to be protected and the preset watermark information; An adversarial sample generation module is constructed based on projected gradient descent, and an adversarial sample purification module is constructed based on a diffusion model, including a watermark embedding submodule. The watermark embedding submodule is constructed based on a decoder of a latent diffusion model. A high-semantic-information weight mask is calculated using gradient-weighted class activation mapping. The high-semantic-information weight mask is input into a stochastic differential editing, and the output of the stochastic differential editing is optimized, specifically: ； ； in, represents a high semantic information weight mask, represents the gradient-weighted class activation map, Indicates the The first round of projected gradient descent attack Step sample, Indicates the Round Projection Gradient Descent Attack The output of the random differential editing process, represents the inverse denoising process of random differential editing, Indicates the The first round of random differential editing Step sample, represents the forward noise addition process of random differential editing, Indicates the The first round of random differential editing Step sample, Indicates the initial input random differential editing Step sample; Iteratively updating the projected gradient descent according to the output result of the optimized stochastic differential editing to construct an adversarial sample generation module; Inputting the image to be protected into the adversarial sample generation module to generate an adversarial sample image; Inputting the adversarial sample image into the adversarial sample purification module for denoising to obtain a restored image, where the restored image is the restored image to be protected; The preset watermark information is added to the restored image through the watermark embedding submodule to obtain a restored image containing the watermark.

2. The restorable image protection method based on the diffusion model according to claim 1, characterized in that: The iterative updating of the projected gradient descent according to the output result of the optimized stochastic differential editing comprises: ； in, Indicates the Samples of round-projected gradient descent attack, express Projection operation on the sphere, Indicates the Samples of round-projected gradient descent attack, represents the iteration step size, represents the symbolic function, Represents the loss function after adding perturbation to the input The gradient with respect to the input, represents the target classifier, represents the true label.

3. The restorable image protection method based on the diffusion model according to claim 1, characterized in that: The adversarial sample purification module including the watermark embedding submodule based on the diffusion model includes: Build a constrained denoiser based on the Transformer architecture with multiple encoder and decoder stages; Obtaining a roughly purified image after roughly purification by a diffusion model; Using the result of the constrained denoiser as a guiding condition of a diffusion model, and combining the roughly purified image and the classifier, optimizing the diffusion inverse process of the diffusion model to obtain an optimized conditional diffusion model; The optimized conditional diffusion model is connected to the watermark embedding submodule to construct an adversarial sample purification module.

4. The restorable image protection method based on the diffusion model according to claim 3 is characterized in that: The method of using the result of the constrained denoiser as a guiding condition of a diffusion model and combining the roughly purified image and the classifier to optimize the diffusion inverse process of the diffusion model comprises: ； in, represents a classifier, Indicates the The denoised image of the step, Indicates the The denoised image of the step, represents the result of the constrained denoiser, represents a Gaussian distribution, represents the mean, represents a constant that controls the guiding strength, represents the variance, represents the gradient, Represents the distance function for calculating similarity, Represents a roughly cleaned image.

5. The restorable image protection method based on the diffusion model according to claim 1 is characterized in that: After the watermark embedding submodule adds the preset watermark information to the restored image to obtain the restored image containing the watermark, the method further includes: Pre-training the watermark extractor to obtain a trained watermark extractor; extracting target watermark information from the restored image containing the watermark by using the trained watermark extractor; The target watermark information and the preset watermark information are compared and traced.

6. A restorable image protection system based on a diffusion model, characterized in that: The system comprises: A data acquisition unit, used to acquire the image to be protected and preset watermark information; A module construction unit is configured to construct an adversarial sample generation module based on projected gradient descent, and an adversarial sample purification module including a watermark embedding submodule based on a diffusion model, wherein the watermark embedding submodule is constructed based on a decoder of a latent diffusion model, wherein a high semantic information weight mask is calculated using a gradient-weighted class activation map; the high semantic information weight mask is input into a stochastic differential editing, and the output result of the stochastic differential editing is optimized, specifically: ； ； in, represents a high semantic information weight mask, represents the gradient-weighted class activation map, Indicates the The first round of projected gradient descent attack Step sample, Indicates the Round Projection Gradient Descent Attack The output of the random differential editing process, represents the inverse denoising process of random differential editing, Indicates the The first round of random differential editing Step sample, represents the forward noise addition process of random differential editing, Indicates the The first round of random differential editing Step sample, Indicates the initial input random differential editing Step sample; Iteratively updating the projected gradient descent according to the output result of the optimized stochastic differential editing to construct an adversarial sample generation module; An image generating unit, configured to input the image to be protected into the adversarial sample generating module to generate an adversarial sample image; An image denoising unit, configured to input the adversarial sample image into the adversarial sample purification module for denoising to obtain a restored image, where the restored image is the restored image to be protected; The watermark adding unit is used to add the preset watermark information to the restored image through the watermark embedding submodule to obtain a restored image containing the watermark.

7. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to execute the restorable image protection method based on the diffusion model as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the restorable image protection method based on a diffusion model according to any one of claims 1 to 5.

Images