Zero-order optimization-based diffusion model artistic copyright protection method and device

Abstract

The invention discloses a diffusion model artistic copyright protection method and device based on zero-order optimization. The method comprises the following steps: constructing a diffusion model; adding tiny disturbance to an original image to obtain an initial confrontation sample, and for the gradient of a loss function when the initial confrontation sample is input into a diffusion model for image generation, estimating the gradient by using a finite difference method through zero-order optimization to obtain a gradient approximate value; performing iterative updating from the initial adversarial sample based on the gradient approximate value to obtain an updated adversarial sample; and calculating the FID value between the original image and the updated generated image of the adversarial sample, and the FID value between the generated image of the original image and the updated generated image of the adversarial sample, and performing secondary judgment on the protection effect of the updated adversarial sample through the FID values. By designing a new algorithm, the generated adversarial sample can be suitable for other black box models, so that the art copyright protection of the digital art image works is better realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital art image work protection, and specifically relates to a diffusion model art copyright protection method and device based on zero-order optimization. Background Art

[0002] In recent years, diffusion models have developed rapidly in the field of computer vision. With their solid theoretical foundation and highly applicable technology, they have achieved remarkable results in a variety of tasks such as image synthesis, video editing, and image creation. In particular, latent diffusion models (DMs) have shown great potential in the field of artwork creation, triggering a wave of commercialization of art artificial intelligence.

[0003] However, with the successful application of diffusion models in the commercial field, the copyright infringement of artists caused by them has become increasingly prominent. One example is text reversal, which is a new function implemented in most AI-for-Art applications based on potential diffusion models. It can imitate the painting art style created by humans with only a small number of samples, but it also provides convenience for infringers. In some infringement cases, infringers can easily obtain works created by artists on the Internet and illegally use them to train models with the help of text reversal functions. Since the existing defense technology cannot effectively protect the copyright of content creators, criminals illegally collect artists' works to build data sets and use diffusion models to generate fake works to make huge profits. Although artists have the right to declare that their works are prohibited from being used to train AI art models, there is currently a lack of effective technology to prevent or track such illegal use, resulting in low infringement costs and difficulty in obtaining evidence. In addition, artists often lack sufficient resources to take legal action against infringers. Therefore, the art community urgently calls for the use of advanced technologies to protect artistic works from AI infringement.

[0004] Based on the above problems, a feasible method is to introduce perturbations into artworks to make them difficult to be recognized by diffusion models in AI-for-Art applications, so that the diffusion model cannot recognize the image as a normal image, thereby limiting the model's ability to extract image features or imitate artistic styles. These images with added perturbations are called adversarial samples of the diffusion model. By converting artworks into adversarial samples, the semantic integrity of the image can be maintained while solving the technical difficulties in copyright protection of artworks. However, how to generate adversarial samples with transferability in diffusion models is a major challenge currently faced.

[0005] First, there are differences in the definitions of diffusion models and deep learning models. Diffusion models are mathematical models based on partial differential equations, which are used to describe and simulate the diffusion process of matter in space, focusing on the propagation and diffusion laws of matter. Deep learning models are machine learning methods that use neural network structures for learning and inference, aiming to achieve various tasks by learning the characteristics and patterns of data, such as image classification, speech recognition, natural language processing, etc. Therefore, the adversarial sample generation methods that are effective in deep learning models may not be applicable or require special adjustments in diffusion models. Secondly, in protecting images, it is necessary to ensure that the diffusion model cannot identify and extract specific features in the image. In addition, the training objective of the diffusion model is indirectly optimized through mutation constraints, so it is not suitable for directly optimizing the generation of adversarial samples. Finally, how to achieve cross-model black box attacks is also a problem that needs to be solved urgently. It is for these reasons that the research on adversarial samples for diffusion models is still in the initial exploration stage. In order to achieve the goal of protecting the copyright of works of art, it is necessary to further study the generation method of adversarial samples of diffusion models to achieve artistic copyright protection. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a diffusion model art copyright protection method and device based on zero-order optimization, focusing on the construction of adversarial samples of the diffusion model, and by designing a new adversarial sample generation algorithm based on zero-order optimization, the generated adversarial samples can be applicable to other black box models, which are closer to actual application scenarios, thereby utilizing the generated adversarial samples to better realize the art copyright protection of digital art image works.

[0007] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0008] In a first aspect, an embodiment of the present invention provides a diffusion model art copyright protection method based on zero-order optimization, comprising the following steps:

[0009] Constructing a diffusion model for obtaining an original image generated image according to the original image, and obtaining an adversarial sample generated image according to the adversarial sample;

[0010] A small perturbation is added to the original image to obtain the initial adversarial sample. The gradient of the loss function when the initial adversarial sample is input into the diffusion model for image generation is estimated by using the finite difference method through zero-order optimization to obtain the gradient approximation.

[0011] Starting from the initial adversarial sample, a new adversarial sample is generated using the gradient approximation, and the gradient approximation corresponding to the current adversarial sample is recalculated at each iterative update to generate the next adversarial sample, and finally an updated adversarial sample is obtained;

[0012] The FID value between the original image and the generated image of the updated adversarial sample, as well as the FID value between the generated image of the original image and the generated image of the updated adversarial sample are calculated, and the protection effect of the updated adversarial sample is secondary judged by the FID value.

[0013] Preferably, the image generation process of the constructed diffusion model includes:

[0014] Initialize the input image and save it as the variables of the diffusion process and inverse diffusion process of the basic diffusion model;

[0015] The VAE encoder is used to encode the input image and map it to feature representation in the latent space;

[0016] Based on the feature representation in the latent space, noise is added in the diffusion process through sampling method to obtain the noisy feature representation;

[0017] The noisy feature representation is gradually restored to the original state through the inverse diffusion process to obtain the reconstructed feature representation;

[0018] The VAE decoder is used to decode the reconstructed feature representation to obtain the reconstructed generated image, realizing image-to-image.

[0019] Preferably, the gradient of the loss function for generating an image in the diffusion model for the initial adversarial sample input is estimated by using a finite difference method through zero-order optimization to obtain a gradient approximation, including:

[0020] For the gradient of the loss function of the initial adversarial sample input diffusion model for image generation, the finite difference method is used to estimate each feature dimension of the initial adversarial sample. For the gradient component of the i-th dimension, two sample points x are constructed by adding a small amount of perturbation in the positive and negative directions. adv′ and x adv″ , expressed as:

[0021] x adv′ =x adv +he i

[0022] x adv″ =x adv -he i

[0023] Among them, x adv represents the current initial adversarial sample, h represents a tiny amount, e i represents the unit vector of the i-th dimension;

[0024] Calculate the forward change f(x adv′ ) and negative changes f(xadv″ ), where f(·) represents the diffusion model function, which is regarded as a loss function or a corresponding probability estimation model;

[0025] The gradient is estimated using the finite difference method via zero-order optimization, expressed as:

[0026]

[0027] in, represents the gradient approximation of the i-th dimension, Represents the feature vector of the current initial adversarial sample in the i-th dimension;

[0028] By calculating the gradient approximation on each dimension, we can get the gradient approximation of the entire initial adversarial sample on all dimensions.

[0029] Preferably, at each iterative update, an optimization objective function is constructed to guide the estimation of the gradient approximation, and the optimization objective function is expressed as:

[0030]

[0031] Among them, p θ (x0′ :T ) represents the diffusion chain x0′ consisting of the gradually noisy images generated in the diffusion random process :T The joint probability of f(x′,θ) represents the loss function of the diffusion model, x′ represents the adversarial sample after adding a slight perturbation δ to the original image x, and θ represents the parameters of the diffusion model. It represents optimization equivalence. By maximizing the loss function, the diffusion model cannot effectively recognize and copy the input image during the generation process. The diffusion model parameters are fixed at each iterative update, and only the loss function and the corresponding gradient calculation are adjusted to update the adversarial sample.

[0032] Preferably, the I-FGSM method is used to generate adversarial samples using gradient approximation, and the adversarial sample of the i+1th step is generated by performing a gradient ascent with a step length of α on the signed gradient. It is expressed as:

[0033]

[0034] in, represents the adversarial sample at step i, represents the gradient approximation, sign(·) represents the sign function, ε represents the maximum value of the perturbation, and T represents the number of iterations;

[0035] Constrained clipping generated make Ensure that the newly generated adversarial samples do not cross the boundary. clip(·) represents the Clip function used to constrain clipping.

[0036] Preferably, the calculation formula of the FID value is:

[0037]

[0038] Among them, μ r represents the feature mean of the reference image, the reference image includes the original image or the image generated by the original image, μ g represents the generated image of the updated adversarial sample, Σ r represents the covariance matrix of the reference image, Σ g represents the covariance matrix of the generated image of the updated adversarial sample, and Tr(·) represents the trace.

[0039] Preferably, the second determination of the protection effect of the updated adversarial sample by the FID value includes:

[0040] The lower the FID, the better the protection effect. If the FID value of the generated image of the original image and the updated adversarial sample or the FID value of the generated image of the original image and the updated adversarial sample exceeds the set discrimination threshold, the adversarial sample generation will continue to increase the protection effect. If it is less than or equal to the set discrimination threshold, the current adversarial sample will be used as an image for copyright protection of digital art image works.

[0041] In a second aspect, in order to achieve the above-mentioned invention purpose, the embodiment of the present invention further provides a diffusion model art copyright protection device based on zero-order optimization, which is implemented by the above-mentioned diffusion model art copyright protection method based on zero-order optimization, including: a diffusion model construction module, a gradient approximation estimation module, an adversarial sample generation module and a protection effect determination module;

[0042] The diffusion model construction module is used to construct a diffusion model for obtaining an original image generated image according to the original image, and obtaining an adversarial sample generated image according to the adversarial sample;

[0043] The gradient approximation estimation module is used to add a small perturbation to the original image to obtain an initial adversarial sample, and to estimate the gradient of the loss function when the initial adversarial sample is input into the diffusion model for image generation, and to obtain a gradient approximation by using a finite difference method through zero-order optimization;

[0044] The adversarial sample generation module is used to generate new adversarial samples using gradient approximation starting from the initial adversarial sample, and recalculate the gradient approximation corresponding to the current adversarial sample at each iterative update to generate the next adversarial sample, and finally obtain the updated adversarial sample;

[0045] The protection effect determination module is used to calculate the FID value between the original image and the generated image of the updated adversarial sample, and the FID value between the generated image of the original image and the generated image of the updated adversarial sample, and perform a secondary determination on the protection effect of the updated adversarial sample through the FID value.

[0046] On the third aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned diffusion model art copyright protection method based on zero-order optimization when executing the computer program.

[0047] In a fourth aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned diffusion model art copyright protection method based on zero-order optimization is implemented.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] (1) The present invention uses a zero-order optimization method to solve the difficult problem of calculating the gradient of complex parameters in the diffusion model. At the same time, in a black box scenario, when the specific parameters of the model are invisible, the gradient approximation can be effectively calculated to generate and optimize adversarial samples.

[0050] (2) The present invention introduces a feature protection mechanism, which protects key information in the image by adding noise to the adversarial sample, prevents unauthorized diffusion models from restoring the image content, and improves the practicality of the image protection method.

[0051] (3) The diffusion model constructed by the present invention fully combines the latent space representation capability of VAE and the advantages of the diffusion model in image generation, thereby improving the quality of generated images and facilitating the further generation of high-quality adversarial samples in the iterative process.

[0052] (4) The present invention further optimizes the generation quality of adversarial samples through secondary judgment, thereby protecting digital artworks from copyright infringement by new AI models such as diffusion models while retaining the complete semantic information of digital artworks, so that the artworks do not lose their original value. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0054] Figure 1 It is a flow chart of a diffusion model art copyright protection method based on zero-order optimization provided by an embodiment of the present invention;

[0055] Figure 2 It is a schematic diagram of the framework of the diffusion model art copyright protection method based on zero-order optimization provided by an embodiment of the present invention;

[0056] Figure 3 It is a schematic diagram of the structure of a diffusion model art copyright protection device based on zero-order optimization provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0058] The inventive concept of the present invention is: in view of the limitations of the adversarial sample generation method for digital image works oriented to diffusion models in the prior art, the embodiment of the present invention provides a diffusion model art copyright protection method and device based on zero-order optimization. Firstly, the gradient approximation of the digital art image work in each coordinate direction is calculated respectively by zero-order optimization to solve the problem that the gradient of the diffusion model is difficult to obtain; secondly, the obtained gradient approximation is used to add disturbances to the digital art image work to generate adversarial samples of the digital art image work for the diffusion model, and the digital art image work is protected while ensuring that the disturbance cannot be recognized by the human eye; finally, the FID indicator in the image field is used in combination with the threshold judgment to perform a secondary judgment on the effect of the adversarial sample, so as to further enhance the art copyright protection effect of the digital art image work.

[0059] Figure 1 is a flow chart of a diffusion model art copyright protection method based on zero-order optimization provided by an embodiment of the present invention, Figure 2 FIG. 1 is a schematic diagram of a framework of a diffusion model art copyright protection method based on zero-order optimization provided by an embodiment of the present invention. Figure 1 and Figure 2 As shown, the embodiment provides a diffusion model art copyright protection method based on zero-order optimization, comprising the following steps:

[0060] S1, constructing a diffusion model for obtaining an original image generated image according to the original image, and obtaining an adversarial sample generated image according to the adversarial sample.

[0061] In the embodiment, the diffusion model effect is first initialized, and the diffusion process of the diffusion model combines the VAE (Variational Autoencoder) encoder and the VAE decoder, including the following steps.

[0062] S1.1, initialize the input image x, normalize it to [-1,1], and save it as the diffusion process q(x t |x t-1 ) and the reverse diffusion process p θ (x t-1 |x t ) variables.

[0063] S1.2, use the VAE encoder to encode the input image x and map it to the feature representation z in the latent space:

[0064] z=μ(x)+σ(x)·ε,ε~N(0,I)

[0065] Among them, μ(x) and σ(x) represent the mean and standard deviation of the neural network output, respectively, and ε represents a random variable.

[0066] S1.3, based on the feature representation z in the latent space, noise is added during the diffusion process through the DPM++SDE Karras sampling method to obtain the noisy feature representation, which is expressed as:

[0067]

[0068] Among them, z0 represents the initial feature representation, z t represents the feature representation after t times of noise addition; α t It represents the time-dependent parameter that controls the diffusion process. It is used to measure the retention ratio of the initial feature z0. It usually has a monotonically decreasing relationship with the time step t, indicating the degree to which the feature is gradually noisy. t It represents the noise term that conforms to the standard normal distribution N(0,I), and it represents the random noise in each diffusion step. Its function is to introduce randomness and thus generate different feature representations.

[0069] S1.4, the noisy feature representation is gradually restored to the original state through the inverse diffusion process to obtain the reconstructed feature representation. The inverse diffusion process is expressed as:

[0070]

[0071] S1.5, use the VAE decoder to decode the reconstructed feature representation to obtain the reconstructed generated image, realizing image-to-image.

[0072] S2, add a small perturbation to the original image to obtain the initial adversarial sample. For the gradient of the loss function when the initial adversarial sample is input into the diffusion model for image generation, the gradient is estimated using the finite difference method through zero-order optimization to obtain the gradient approximation.

[0073] In an embodiment, for adversarial samples of digital art images, it is necessary to set various hyperparameters involved in the iterative process, and to calculate the gradient of the loss function when the initial adversarial sample is input into the diffusion model for image generation. The approximate gradient is calculated based on these hyperparameters by using the finite difference method for each feature dimension of the initial adversarial sample, including the following steps.

[0074] S2.1, fixed input prompt is p, prompt is the text description generated by the digital art image, used to guide the diffusion model to form or recognize the image, the maximum infinite norm perturbation is ε, the iteration step is α, the iteration round is T, the diffusion model function is f, and the adversarial sample is x adv , the unit vector is e.

[0075] S2.2, based on the initial adversarial sample, during the optimization process, for the gradient component of the i-th dimension, two sample points x are constructed by adding a small amount of perturbation in the positive and negative directions adv′ and x adv″ , expressed as:

[0076] x adv′ =x adv +he i

[0077] x adv″ =x adv -he i

[0078] Among them, x adv represents the current initial adversarial sample, e i represents a unit vector with the i-th dimension being 1 and the other dimensions being 0, and h represents a tiny quantity, so x adv +he i and x adv -he i They represent the points after moving a distance h in the positive and negative directions in the i-th coordinate axis direction, respectively.

[0079] S2.3, calculate the forward change f(x adv′ ) and negative changes f(x adv″ ), where f(·) represents the diffusion model function, which is regarded as a loss function or a corresponding probability estimation model, and is used to measure the recognizability or reconstruction difficulty of the input image. The gradient is estimated using the finite difference method through zero-order optimization, which is expressed as:

[0080]

[0081] in, represents the gradient approximation of the i-th dimension, Represents the feature vector of the current initial adversarial sample in the i-th dimension;

[0082] S2.4, by calculating the gradient approximation on each dimension, we can get the gradient approximation of the entire initial adversarial sample on all dimensions.

[0083] S3, starting from the initial adversarial sample, uses the gradient approximation to generate a new adversarial sample, and recalculates the gradient approximation corresponding to the current adversarial sample at each iterative update to generate the next adversarial sample, and finally obtains the updated adversarial sample.

[0084] S3.1, define the optimization goal. The goal of adversarial sample generation is to achieve adversarial attacks on the diffusion model, so that the diffusion model cannot effectively identify and copy the original image during the generation process, and the generated adversarial sample image cannot be clearly distinguished from the human eye, that is, minimize the probability of the image being recognized by the diffusion model. In order to achieve this goal, add a small perturbation x′=x+δ to the original image, where x represents the original image and δ represents the small perturbation added to the original image, which satisfies:

[0085]

[0086] Among them, p θ (x+δ) represents the probability density of the diffusion model output, and ε represents the maximum value of the disturbance.

[0087] Because p θ (x) = ∫p θ (x 0:T )dx 1:T , the following function can be used as the target for the perturbation δ:

[0088]

[0089] Combining it with the diffusion model loss function, we get the final form of the optimization objective function:

[0090]

[0091] Among them, p θ (x0′ :T ) represents the diffusion chain x0′ consisting of the gradually noisy images generated in the diffusion random process :TThe joint probability of f(x′,θ) represents the loss function of the diffusion model, x′ represents the adversarial sample after adding a slight perturbation δ to the original image x, and θ represents the parameters of the diffusion model. It represents optimization equivalence. By maximizing the loss function, the diffusion model cannot effectively recognize and copy the input image during the generation process. The diffusion model parameters are fixed at each iterative update, and only the loss function and the corresponding gradient calculation are adjusted to update the adversarial sample.

[0092] S3.2, the I-FGSM method is used to generate adversarial samples using gradient approximation, and the sign function is used to limit the size of the estimated gradient. During the optimization process, the adversarial sample of the i+1th step is generated by performing a gradient ascent with a step size of α on the sign gradient. It is expressed as:

[0093]

[0094] in, represents the adversarial sample at step i, represents the gradient approximation, sign(·) represents the sign function, ε represents the maximum value of the perturbation, and T represents the number of iterations.

[0095] S3.3, Constrained Clipping Generation make Ensure that the newly generated adversarial samples do not cross the boundary. clip(·) represents the Clip function used to constrain clipping.

[0096] S3.4, repeat steps S3.2 to S3.3 until the iteration is completed and the updated adversarial sample is obtained.

[0097] S4, calculate the FID value between the original image and the generated image of the updated adversarial sample, and the FID value between the generated image of the original image and the generated image of the updated adversarial sample, and make a secondary judgment on the protection effect of the updated adversarial sample through the FID value.

[0098] In the embodiment, a second judgment is performed on the finally generated adversarial sample to ensure its effectiveness.

[0099] S4.1, referring to step S1, use the original image and the updated adversarial sample to input the diffusion model for image generation, and obtain the generated image of the original image and the generated image of the updated adversarial sample.

[0100] S4.2, use the generative model evaluation indicator FID to judge the protection effect of the adversarial sample, and calculate the FID value between the original image and the generated image of the updated adversarial sample, as well as the FID value between the generated image of the original image and the generated image of the updated adversarial sample.

[0101] The calculation of FID requires the use of a pre-trained image classification model, usually the Inception model. First, the original image and the generated image of the updated adversarial sample, or the generated image of the original image and the generated image of the updated adversarial sample are input into the classification model to extract their feature representations in the middle layer of the model. Then, the Fréchet distance (i.e., FID value) between the images is calculated by performing statistical analysis on these features.

[0102]

[0103] Among them, μ r represents the feature mean of the reference image, the reference image includes the original image or the image generated by the original image, μ g represents the generated image of the updated adversarial sample, ∑ r represents the covariance matrix of the reference image, ∑ g represents the covariance matrix of the generated image of the updated adversarial sample, and Tr(·) represents the trace.

[0104] S4.3, judging whether to continue generating adversarial samples according to the FID value. The lower the FID, the better the protection effect. If the FID value of the generated image of the original image and the updated adversarial sample or the FID value of the generated image of the original image and the updated adversarial sample exceeds the set discrimination threshold, the adversarial sample generation is continued to increase the protection effect. If it is less than or equal to the discrimination threshold, the current adversarial sample is used as an image for copyright protection of digital art image works.

[0105] In summary, the embodiment of the present invention provides an art copyright protection method based on a diffusion model of zero-order optimization. It uses zero-order optimization to effectively calculate the gradient approximation for adversarial sample generation and optimization, which is more suitable for black box scenarios. By adding noise to the adversarial sample, the key information in the image is protected, and the unauthorized diffusion model is prevented from restoring the image content, thereby improving the practicality of the image protection method. By constructing a diffusion model combined with VAE, the quality of the generated image is improved, which is conducive to further generating high-quality adversarial samples in the iterative process. The generation quality of the adversarial sample is further optimized through secondary judgment to ensure the art copyright protection effect of the digital art image works of the adversarial sample.

[0106] Based on the same inventive concept, Figure 3 As shown, an embodiment of the present invention also provides a diffusion model art copyright protection device 300 based on zero-order optimization, including: a diffusion model construction module 310, a gradient approximation estimation module 320, an adversarial sample generation module 330 and a protection effect determination module 340.

[0107] The diffusion model construction module 310 is used to construct a diffusion model for obtaining an original image generated image according to an original image, and obtaining an adversarial sample generated image according to an adversarial sample.

[0108] The gradient approximation estimation module 320 is used to add a small perturbation to the original image to obtain an initial adversarial sample. For the gradient of the loss function when the initial adversarial sample is input into the diffusion model for image generation, the gradient is estimated using the finite difference method through zero-order optimization to obtain a gradient approximation.

[0109] The adversarial sample generation module 330 is used to generate new adversarial samples using gradient approximations starting from the initial adversarial sample, and to recalculate the gradient approximation corresponding to the current adversarial sample at each iterative update to generate the next adversarial sample, and finally obtain an updated adversarial sample.

[0110] The protection effect determination module 340 is used to calculate the FID value between the original image and the generated image of the updated adversarial sample, and the FID value between the generated image of the original image and the generated image of the updated adversarial sample, and perform a secondary determination on the protection effect of the updated adversarial sample through the FID value.

[0111] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned diffusion model art copyright protection method based on zero-order optimization when executing the computer program.

[0112] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned diffusion model art copyright protection method based on zero-order optimization is implemented.

[0113] It should be noted that the diffusion model art copyright protection device based on zero-order optimization, the electronic device, and the computer-readable storage medium provided in the above-mentioned embodiments all belong to the same inventive concept as the diffusion model art copyright protection method based on zero-order optimization. The specific implementation process is detailed in the embodiment of the diffusion model art copyright protection method based on zero-order optimization, which will not be repeated here.

[0114] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A diffusion model art copyright protection method based on zero-order optimization, characterized in that: The following steps are involved: Constructing a diffusion model for obtaining an original image generated image according to the original image, and obtaining an adversarial sample generated image according to the adversarial sample; A small perturbation is added to the original image to obtain the initial adversarial sample. The gradient of the loss function when the initial adversarial sample is input into the diffusion model for image generation is estimated by using the finite difference method through zero-order optimization to obtain the gradient approximation. Starting from the initial adversarial sample, a new adversarial sample is generated using the gradient approximation, and the gradient approximation corresponding to the current adversarial sample is recalculated at each iterative update to generate the next adversarial sample, and finally an updated adversarial sample is obtained; The FID value between the original image and the generated image of the updated adversarial sample, as well as the FID value between the generated image of the original image and the generated image of the updated adversarial sample are calculated, and the protection effect of the updated adversarial sample is secondary judged by the FID value.

2. The method for protecting artistic copyright based on diffusion model of zero-order optimization according to claim 1, characterized in that: The image generation process of the constructed diffusion model includes: Initialize the input image and save it as the variables of the diffusion process and inverse diffusion process of the basic diffusion model; The VAE encoder is used to encode the input image and map it to feature representation in the latent space; Based on the feature representation in the latent space, noise is added in the diffusion process through sampling method to obtain the noisy feature representation; The noisy feature representation is gradually restored to the original state through the inverse diffusion process to obtain the reconstructed feature representation; The VAE decoder is used to decode the reconstructed feature representation to obtain the reconstructed generated image, realizing image-to-image.

3. The method for protecting artistic copyright based on diffusion model of zero-order optimization according to claim 1, characterized in that: The gradient of the loss function when generating an image in the diffusion model for the initial adversarial sample input is estimated by using a finite difference method through zero-order optimization to obtain a gradient approximation, including: For the gradient of the loss function of the initial adversarial sample input diffusion model for image generation, the finite difference method is used to estimate each feature dimension of the initial adversarial sample. For the gradient component of the i-th dimension, two sample points x are constructed by adding a small amount of perturbation in the positive and negative directions. adv′ and x adv″ , expressed as: x adv′ =x adv +he i x adv″ =x adv -he i Among them, x adv represents the current initial adversarial sample, h represents a tiny amount, e i represents the unit vector of the i-th dimension; Calculate the forward change f(x adv′ ) and negative changes f(x adv″ ), where f(·) represents the diffusion model function, which is regarded as a loss function or a corresponding probability estimation model; The gradient is estimated using the finite difference method via zero-order optimization, expressed as: in, represents the gradient approximation of the i-th dimension, Represents the feature vector of the current initial adversarial sample in the i-th dimension; By calculating the gradient approximation on each dimension, we can get the gradient approximation of the entire initial adversarial sample on all dimensions.

4. The method for protecting artistic copyright based on diffusion model of zero-order optimization according to claim 1, characterized in that: At each iterative update, an optimization objective function is constructed to guide the estimation of the gradient approximation. The optimization objective function is expressed as: Among them, p θ (x0′ :T ) represents the diffusion chain x0′ consisting of the gradually noisy images generated in the diffusion random process :T The joint probability of f(x′,θ) represents the loss function of the diffusion model, x′ represents the adversarial sample after adding a slight perturbation δ to the original image x, and θ represents the parameters of the diffusion model. It represents optimization equivalence. By maximizing the loss function, the diffusion model cannot effectively recognize and copy the input image during the generation process. The diffusion model parameters are fixed at each iterative update, and only the loss function and the corresponding gradient calculation are adjusted to update the adversarial sample.

5. The method for protecting artistic copyright based on diffusion model of zero-order optimization according to claim 1, characterized in that: The I-FGSM method is used to generate adversarial samples using gradient approximation. The adversarial sample of the i+1th step is generated by performing a gradient ascent with a step length of α on the signed gradient. It is expressed as: in, represents the adversarial sample at step i, represents the gradient approximation, sign(·) represents the sign function, ε represents the maximum value of the perturbation, and T represents the number of iterations; Constrained clipping generated make Ensure that the newly generated adversarial samples do not cross the boundary. clip(·) represents the Clip function used to constrain clipping.

6. The method for protecting artistic copyright based on diffusion model of zero-order optimization according to claim 1, characterized in that: The calculation formula of FID value is: Wherein, μr represents the feature mean of the reference image, the reference image includes the original image or the image generated by the original image, μ g represents the generated image of the updated adversarial sample, ∑ r represents the covariance matrix of the reference image, ∑ g represents the covariance matrix of the generated image of the updated adversarial sample, and Tr(·) represents the trace.

7. The method for protecting artistic copyright based on diffusion model of zero-order optimization according to claim 1, characterized in that: The second determination of the protection effect of the updated adversarial sample by the FID value includes: The lower the FID, the better the protection effect. If the FID value of the generated image of the original image and the updated adversarial sample or the FID value of the generated image of the original image and the updated adversarial sample exceeds the set discrimination threshold, the adversarial sample generation will continue to increase the protection effect. If it is less than or equal to the set discrimination threshold, the current adversarial sample will be used as an image for copyright protection of digital art image works.

8. A diffusion model art copyright protection device based on zero-order optimization, implemented by the diffusion model art copyright protection method based on zero-order optimization according to any one of claims 1 to 7, characterized in that: include: Diffusion model construction module, gradient approximation estimation module, adversarial sample generation module and protection effect determination module; The diffusion model construction module is used to construct a diffusion model for obtaining an original image generated image according to the original image, and obtaining an adversarial sample generated image according to the adversarial sample; The gradient approximation estimation module is used to add a small perturbation to the original image to obtain an initial adversarial sample, and to estimate the gradient of the loss function when the initial adversarial sample is input into the diffusion model for image generation, and to obtain a gradient approximation by using a finite difference method through zero-order optimization; The adversarial sample generation module is used to generate new adversarial samples using gradient approximation starting from the initial adversarial sample, and recalculate the gradient approximation corresponding to the current adversarial sample at each iterative update to generate the next adversarial sample, and finally obtain the updated adversarial sample; The protection effect determination module is used to calculate the FID value between the original image and the generated image of the updated adversarial sample, and the FID value between the generated image of the original image and the generated image of the updated adversarial sample, and perform a secondary determination on the protection effect of the updated adversarial sample through the FID value.

9. An electronic device comprising a memory and a processor, wherein the memory is used to store a computer program, wherein: The processor is used to implement the zero-order optimization-based diffusion model art copyright protection method described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer, the diffusion model art copyright protection method based on zero-order optimization described in any one of claims 1 to 7 is implemented.

Images