Method and device for measuring memory problem of text graph diffusion model based on reverse

Abstract

Compared with an existing measurement method, the method for measuring the memory problem based on the reverse text image diffusion model, provided by the invention, constructs a weighted objective function based on a given pre-trained text image diffusion model and an arbitrary image, reversely outputs a prompt text distribution and a noise distribution, and obtains the memory problem of the memory problem based on the reverse text image diffusion model. The random influence caused by evaluating the memory problem by detecting similar content of the image based on a randomly generated image in the past is solved; the measurement index function of the memory problem is constructed by defining the sensitivity of noise distribution, the memory problem of the image generation model is continuously and accurately measured, a quantitative result of the memory risk of any image is given, and the memory risk of the image generation model is audited. The invention further provides a device for measuring the memory problem based on the reverse text graph diffusion model, and the method for measuring the image memory problem is achieved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a measurement method and device for a memory problem of a reverse Wensheng graph diffusion model. Background Art

[0002] The diffusion model has shown outstanding capabilities in the fields of image, video and 3D model generation, and has spawned a large number of AIGC (Artificial Intelligence Generated Content) commercial systems and image generation tools, such as Stable Diffusion, Midjourney, DALLE and Imagen, which have attracted millions of users. The diffusion model is a data generation theory and method. Combined with deep learning technology, it can be used to fit real data and can generate various data types such as images, videos, and music. Diffusion model generation can also introduce conditions to achieve conditional generation, such as text generation of images, that is, using a piece of text as a condition to generate an image that meets the text description. Compared with other image synthesis technologies, the diffusion model has stable training, strong scalability, and higher quality of generated content.

[0003] However, under certain conditions, the diffusion model will frequently generate certain images, which can be figuratively referred to as the image generation model remembering these images, which is also called the memory problem of the image generation model. The memory problem goes against the expectations of the image generation model. Technical developers do not want the model to rigidly remember certain data, and expect it to generate diverse data. In addition, if the memory problem occurs on data involving copyright, privacy and other issues, it will bring potential security risks. Therefore, it is necessary to audit the diffusion model for memory problems in order to comprehensively and accurately analyze its memory risks. For example, the invention application with publication number CN117633899 A discloses a mask-based face image generation model privacy protection method, system and device, which obtains the degree of attention of facial features through the face image generation model, and trains a deep fake face generation model based on the degree of attention of facial features, which is used to protect the privacy of facial data sets and reduce the leakage of facial data.

[0004] The text-generated image diffusion model also has the risk of training data memorization. Specifically, when a certain prompt text is input, the model may generate an image that is very similar to a training image. For example, reference 1 (Carlini N, Hayes J, Nasr M, et al, Extracting training data from diffusion models, 32nd USENIX Security Symposium (USENIX Security 23), 2023, 5253-5270.) extracts training data based on the text-generated image diffusion model, inputs prompt text to detect data memory problems, and reduces privacy leakage during image generation. Specifically, a large amount of image description text is collected from the text-generated image diffusion model training set, and then each description text is input into the diffusion model, and different random noises are used for generation to obtain a large number of generated images. For each description text, several images (500 images) generated are detected to determine whether these generated images are the same. If there are more than 10 identical images, it is considered that this description text will cause memory risk. Finally, it is determined whether these identical images are the images corresponding to the description text in the training set; The technology proposed in Reference 2 (Somepalli G, Singla V, Goldblum M, et a1, Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, 6048-6058.) is similar to that in Reference 1, except that Reference 1 uses a pre-trained image retrieval model SSCD to detect whether there are images in the generated images that are very similar to the training images.

[0005] These works can solve the memory risk of image generation models to a certain extent and reduce privacy leakage in the image generation process. However, they are all based on analysis techniques based on prompt text. By randomly generating images and then detecting whether there is content similar to the training images, they will be affected by the randomness of the generation process. In addition, existing technologies cannot be used to continuously and accurately quantify the memory risk of any image. Therefore, a comprehensive and accurate audit of the memory risk of the diffusion model is an urgent problem to be solved to protect the security of image generation data. Summary of the invention

[0006] The purpose of the present invention is to provide a measurement method and device for memory problems based on a Wensheng graph diffusion model, and a measurement method and device for memory problems developed based on prompt text distribution and noise distribution, which are used to analyze any image and judge the memory problem of image generation; and provide measurement indicators of memory problems for accurately quantifying the memory risk of image generation, realizing a comprehensive and accurate audit of the memory risk of the diffusion model, and reducing the random influence of the image generation process.

[0007] To achieve the above-mentioned purpose of the invention, an embodiment provides a method and device for measuring memory problems based on an inverse Vincent graph diffusion model, including a method for measuring memory problems based on an inverse Vincent graph diffusion model and a device for measuring memory problems.

[0008] In one embodiment, the method for measuring the memory problem of the inverse Wensheng graph diffusion model comprises the following steps:

[0009] The noise distribution is constructed based on the noise adding process of the image to be evaluated in the Vignette Diffusion Model, and the prompt text distribution is constructed based on the text encoder of the Vignette Diffusion Model.

[0010] The sensitivity of the noise distribution is defined as the probability of generating the image to be evaluated by combining the random noise from the noise distribution and the prompt text sampled from the prompt text distribution, and a memory problem measurement indicator function is constructed based on the sensitivity;

[0011] A weighted objective function is constructed by using a denoising error constructed based on the prompt text distribution and the noise distribution and a normality regularization constructed based on the noise distribution;

[0012] The weighted objective function is optimized through an adaptive optimization algorithm to obtain the optimal prompt text distribution and noise distribution. The memory problem measurement index function is calculated based on the optimal noise distribution to obtain the memory problem measurement index.

[0013] In one embodiment, the text encoder based on the text graph diffusion model constructs the prompt text distribution, including:

[0014] (1) The word segmenter in the text encoder encodes the text in string format into a prompt text w of length L, wherein the prompt text w is a word sequence w = [w1, w2, ..., w L ]express;

[0015] (2) Based on the category distribution of each word, obtain the category distribution probability of each word [π i,1 , π i，2 , ..., π i，V ],∑ j＝1 π i，j =1,j∈[0,V],π i，jrepresents the j-th category probability of the i-th word, V represents the category type, and the category probability is parameterized as logπ i，j =φ i，j ,φ i，j represents the,th category hint text distribution parameter of the i-th word;

[0016] (3) Smoothly sample the category distribution probability of each word and construct the smoothed word based on the smoothed sample and category embedding vector The word vector The word vector sequence composed of all word vectors Input to the encoding network to obtain the text feature c = f(e(w)), and construct the prompt text distribution q with parameter φ using the prompt text w in the text feature and the image to be evaluated x0 φ (w|x0).

[0017] In one embodiment, the category distribution of each word is smoothly sampled, and a smoothed word is constructed based on the smoothed sample and the category embedding vector. The word vector include:

[0018] Use Gumbel-Softmax reparameterization to smoothly sample the category distribution:

[0019]

[0020] Among them, g i，j and g i，k Random sampling from Gumbel (0, 1) distribution, each g i，j Independent of each other, π i，k represents the k-th category probability of the i-th word, τ is a constant, called the temperature factor; V represents the category type, Represents the distribution from category π i A smoothed sample of Indicates the j-th category weight corresponding to the ith sample.

[0021] Construct smoothed word units based on smoothed samples and category embedding vectors e(j) The word vector

[0022]

[0023] In one embodiment, the memory problem measurement index function constructed based on sensitivity is:

[0024]

[0025] in, Represents the sensitivity of the noise distribution, specifically from the parameter The noise distribution The Gaussian noise ∈ randomly sampled from the prompt text distribution q with parameter φ φ The probability that the sampled prompt text w in (w|x0) generates the image x0 to be evaluated, Represents the image x0 to be evaluated and the noise distribution parameters The constructed normality regularization, memory problem metric function indicates that there is at least one prompt text w, and the sensitivity When x is 1, the minimum value of the normality regularization is used as the memory problem metric mem(x0).

[0026] In one embodiment, the normality regularization is performed by using the KL divergence measure of the noise distribution. The distance D from the standard normal distribution KL It means that when the noise distribution The fitting is with mean μ and variance σ 2 Normal distribution N(μ, σ 2 )hour, Normality The calculation method is:

[0027]

[0028] In one embodiment, the denoising error constructed based on the prompt text distribution and the noise distribution is expressed as l de :

[0029]

[0030] Where, ∈ represents the noise distribution Gaussian noise randomly sampled in, t represents the moment randomly sampled from the time uniform distribution U(1, T), T represents the maximum moment, x t represents the noisy image of the image to be evaluated x0 at time t in the forward diffusion noisy process in the Wensheng graph diffusion model, ∈ θ (x t , t, f(e(w))) represents the value based on x t , t, f(e(w)) is the prediction noise calculated by the neural network with parameter θ, f(e(w) is the text feature, α t Represents a constant related to time t, with a value between 0 and 1.

[0031] In one embodiment, the weighted objective function is expressed as:

[0032]

[0033] Among them, L represents the denoising error Normality and the weight λ to construct the weighted objective function.

[0034] In one embodiment, the step of optimizing the weighted objective function by an adaptive optimization algorithm to obtain the optimal prompt text distribution and noise distribution includes:

[0035] (1) Initialize internal variables, specifically, the prompt text distribution parameter φ is 0, the noise distribution parameter It obeys the normal distribution N(0, I), the weight λ is 1, and the previous denoising error tends to positive infinity, the patience value P is the tolerance period ρ, and the number of optimization steps i is 1;

[0036] (2) Based on the initialized internal variables, calculate the weighted objective function

[0037] (3) Calculate the weighted objective function L relative to the prompt text distribution parameter φ and the noise distribution parameter Gradient and

[0038] (4) Based on the gradient descent method, the prompt text distribution parameter φ and the noise distribution parameter Update to obtain the updated prompt text distribution parameter φ' and noise distribution parameter Specifically Where γ is the learning rate, which is used to control the step size of parameter update;

[0039] (5) Check whether the observation period C has been reached. Specifically, observe every C steps. If the optimization step number i>1 and the optimization step number i modulo the observation period C is 0, indicating that C steps have just passed, then proceed to step (6). If the optimization step number i modulo the observation period C is not 0, then proceed to step (10);

[0040] (6) Checking the denoising error Whether it decreases, specifically, if If the optimization is not successful, proceed to step (7); This indicates that the optimization is successful, and step (8) is performed;

[0041] (7) If Then the weight λ decays by a multiple of λ / 2, the patience value P decays by P-1, and then jumps to step (9);

[0042] (8) If Then the weight λ increases by the weight increment δ, and the patience value returns to the patience period ρ;

[0043] (9) Previous denoising error To update, Perform step (11);

[0044] (10) Increase the weight δ appropriately;

[0045] (11) If the patience value P = 0, jump to step (13), otherwise proceed to step (12) to update the optimization step number i;

[0046] (12) updating the optimization step number i. Specifically, the optimization step number i is optimized with an increment of 1. If the optimization step number i is greater than the total number of iteration steps S, proceed to step (13); otherwise, return to step (2) for iteration;

[0047] (13) The optimized generation result is tested. Specifically, from the prompt text distribution q φ (w|x0) and noise distribution Sample prompt text and random noise to generate images, observe these generated images manually or use a third-party image matching model to compare; if the generated image is very similar to the image to be evaluated, output the prompt text distribution q φ (w|x0), noise distribution and normality regularization If there is an image that is different from the image to be evaluated, the prompt text distribution q is output φ (w|x0), noise distribution and +∞.

[0048] In one embodiment, the optimal prompt text distribution and noise distribution are obtained to achieve weighted objective function minimization, which is expressed as:

[0049]

[0050] where φ * Expressed as the optimal prompt text distribution parameter, Denotes the optimal noise distribution parameter, λ * is represented as the optimal weight.

[0051] In order to clearly demonstrate the measurement method of memory problems, a measurement device for memory problems based on the reverse Vincent graph diffusion model is provided, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to implement the measurement method for memory problems based on the reverse Vincent graph diffusion model when executing the computer program.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The present invention provides a method for measuring memory problems based on an inverse Vincent graph diffusion model. Compared with existing measurement methods, the method is based on a given pre-trained Vincent graph diffusion model and an arbitrary image, constructs a weighted objective function, and inversely outputs a prompt text distribution and a noise distribution, thereby solving the random influence brought by the previous method of evaluating memory problems based on randomly generated images to detect image similarity content; by defining the sensitivity of the noise distribution to construct a measurement index function for the memory problem, the memory problem of the image generation model is continuously and accurately measured, the quantitative result of the memory risk of any image is given, and the memory risk of the image generation model is audited.

[0054] The present invention also provides a measurement device for memory problems based on the reverse Vincent graph diffusion model, and implements a measurement method for image memory problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0056] Figure 1 Flowchart of the measurement method for memory problems;

[0057] Figure 2 A schematic diagram of the structure of the measurement method for memory problems;

[0058] Figure 3 Flowchart of the adaptive optimization algorithm. DETAILED DESCRIPTION

[0059] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0060] In order to continuously and accurately quantify the memory risk of any image and realize a comprehensive and accurate audit of the memory risk of the diffusion model, an embodiment provides a memory problem measurement method based on an inverse Vincent graph diffusion model. The prediction method constructs a weighted objective function by giving a pre-trained Vincent graph diffusion model and an arbitrary image, and inversely outputs a prompt text distribution and a noise distribution. The distribution parameters are optimized based on the weighted objective function to create a measurement index of the image memory problem, which is used to continuously and accurately measure the image memory problem.

[0061] In the embodiment, the diffusion model training is completed by the following denoising and denoising methods. Randomly sample a noise from a standard Gaussian distribution N(0, I) of the same dimension as x0 Randomly sample a time t from the uniform distribution U(1, T) and superimpose noise on the training image in the following way:

[0062]

[0063] where α t is a constant related to time t, and α at all times t , t∈[1,T] are all preset manually. The larger t is, the larger α is. t The closer to 0.

[0064] The noisy training image x t , time t, text feature c is input into the pre-trained denoising network to obtain the predicted noise ∈ θ (x t ,t,c), prediction noise∈ θ The error with the true value noise ∈ 0 is calculated by the bi-norm between the two Represents. With the goal of minimizing this error, the parameters θ of the pre-trained denoising network are continuously optimized to complete the training of the pre-trained denoising network.

[0065] The Wensheng graph diffusion model is essentially a denoiser. Using a trained denoising network, a random noise image from a standard Gaussian distribution can be gradually denoised to generate a real image. It is usually implemented using a deep neural network. The function implemented by the denoising network is expressed as:

[0066]

[0067] Among them, θ is its pre-training parameter; is a noisy image, H×W×C1 are the height, width and number of channels of the image respectively; t∈[1,T] represents a moment, T represents the maximum moment, which is a positive integer; Indicates the condition for guiding text to generate an image, corresponding to the input prompt text. The input prompt text consists of L tokens, each of which is a C2-dimensional real vector.

[0068] The actual text graph diffusion model uses another pre-trained text encoder to encode the input string text into continuous features, namely c mentioned above. The text encoder consists of a Tokenizer, an EmbeddingLayer, and an Encoder Network. The tokenizer encodes the input string text into a word sequence w = [w1, w2, ..., w L ],w i ∈[1, V], where V represents the category, that is, the number of all possible word units, and w irepresents the i-th word. Then, the embedding layer e transforms each word w i Mapped to the C2-dimensional word vector e(w i ). Finally, all word vectors e(w i ) are sequentially input into the encoding network f to obtain the semantically rich prompt text feature c = f(e(w)), where e(w) = [e(w1), e(w2), ..., e(W L )].

[0069] like Figure 1 As shown, the embodiment provides a method for measuring memory problems based on the reverse Vincent graph diffusion model, comprising the following steps:

[0070] S1. A noise distribution is constructed based on the noise adding process of the image to be evaluated in the text-based graph diffusion model, and a prompt text distribution is constructed based on the text encoder of the text-based graph diffusion model.

[0071] The text distribution is modeled in the text encoder embedding layer e: the tokenizer encodes the string-formatted text into a prompt text w = [w1, w2, ..., w L ]. Assume that each word w i It obeys a V-dimensional categorical distribution, and the categorical distribution probability of each word is [π i，1 , π i，2 , ..., π i，V ],∑ j＝1 π i，j =1,j∈[0,V],π i，j represents the j-th category probability of the ith word, V represents the category type, L words obey L independent category distributions, and the probability of each category distribution can be parameterized as logπ i，j =φ i，j ,φ i，j represents the distribution parameter of the j-th category hint text of the i-th word;

[0072] Since the prompt text distribution is not differentiable in sampling, the category distribution probability of each word is reparameterized, and the category distribution is smoothly sampled using the Gumbel-Softmax reparameterization method:

[0073]

[0074] Among them, g i，j and g i,k Random sampling from Gumbel (0, 1) distribution, each g i，j Independent of each other, π i，krepresents the k-th category probability of the i-th word, τ is a constant, called the temperature factor; V represents the category type, Represents the distribution from category π i A smoothed sample of Indicates the j-th category weight corresponding to the ith sample.

[0075] Construct smoothed word units based on smoothed samples and category embedding vectors e(j) The word vector

[0076]

[0077] The word vector sequence composed of all word vectors Input to the encoding network to obtain the text feature c = f(e(w)), and construct the prompt text distribution q with parameter φ using the prompt text w in the text feature and the image to be evaluated x0 φ (w|x0).

[0078] S2. Define the sensitivity of the noise distribution as the probability of generating the image to be evaluated by combining the random noise from the noise distribution and the prompt text sampled from the prompt text distribution, and construct a memory problem measurement indicator function based on the sensitivity.

[0079] In the embodiment, the memory problem measurement index function is constructed based on sensitivity:

[0080]

[0081] in, Represents the sensitivity of the noise distribution, specifically from the parameter The noise distribution A randomly sampled Gaussian noise ∈ is combined with the hint text distribution q from parameter φ φ The probability that the sampled prompt text w in (w|x0) generates the image x0 to be evaluated, Represents the image x0 to be evaluated and the noise distribution parameters The constructed normality regularization, memory problem metric function indicates that there is at least one prompt text w, and the sensitivity Under the condition that is 1, the minimum value of the normality regularization is used as the memory problem measurement indicator mem(x0); the indicator mem(x0) is a non-negative real number, which indicates the degree to which the image is remembered by the given model. The closer the indicator mem(x0) is to 0, the greater the degree of memory, and the closer it is to positive infinity, the smaller the degree of memory.

[0082] S3. A weighted objective function is constructed by using a denoising error constructed based on the prompt text distribution and the noise distribution and a normality regularization constructed based on the noise distribution.

[0083] In the embodiment, Figure 2 As shown in the figure, based on an arbitrary image, a pre-trained text diffusion model is combined to reversely output a prompt text distribution and a noise distribution, and the degree of memory is measured based on the noise distribution.

[0084] In the embodiment, the noise distribution is measured by using KL divergence The distance D from the standard normal distribution KL It means that when the noise distribution The fitting is with mean μ and variance σ 2 Normal distribution N(μ, σ 2 )hour, Normality The calculation method is:

[0085]

[0086] in, Represents the image x0 to be evaluated and the noise distribution parameters The constructed normality regularization.

[0087] The denoising error constructed based on the prompt text distribution and noise distribution is expressed as l de :

[0088]

[0089] Where, ∈ represents the noise distribution Gaussian noise randomly sampled in, t represents the moment randomly sampled from the time uniform distribution U(1, T), T represents the maximum moment, x t represents the noisy image of the image to be evaluated x0 at time t in the forward diffusion noisy process in the Wensheng graph diffusion model, ∈ θ (x t , t, f(e(w))) represents the value based on x t , t, f(e(w)) is the prediction noise calculated by the neural network with parameter θ, f(e(w) is the text feature, α t Represents a constant related to time t, with a value between 0 and 1.

[0090] Based on the denoising error l de and normality regularization nr The constructed weighted objective function is:

[0091]

[0092] Where λ is the introduced weight, L represents the denoising error Normality and the weight λ to construct the weighted objective function.

[0093] S4. Optimizing the weighted objective function through an adaptive optimization algorithm to obtain the optimal prompt text distribution and noise distribution, and calculating the memory problem measurement index function based on the optimal noise distribution to obtain the memory problem measurement index.

[0094] In order to optimize the weighted objective function to obtain the optimal prompt text distribution and noise distribution parameters, the present invention proposes an adaptive optimization algorithm to optimize and minimize the weighted objective function. During the optimization process, the parameters θ of the pre-trained denoising network remain unchanged. The algorithm input includes: the pre-trained denoising network ∈ θ , text encoder, image to be evaluated x0, total number of iterations S, observation period C, weight increment δ, denoising error threshold ξ, learning rate γ and tolerance period ρ;

[0095] like Figure 3 As shown, the steps of the adaptive optimization algorithm are as follows:

[0096] (1) Initialize internal variables, specifically, the prompt text distribution parameter φ is 0, the noise distribution parameter It obeys the normal distribution N(0, I), the weight λ is 1, and the previous denoising error tends to positive infinity, the patience value P is the tolerance period ρ, and the number of optimization steps i is 1;

[0097] (2) Based on the initialized internal variables, calculate the weighted objective function

[0098] (3) Calculate the weighted objective function L relative to the prompt text distribution parameter φ and the noise distribution parameter Gradient and

[0099] (4) Based on the gradient descent method, the prompt text distribution parameter φ and the noise distribution parameter Update to obtain the updated prompt text distribution parameter φ' and noise distribution parameter Specifically Where γ is the learning rate, which is used to control the step size of parameter update;

[0100] (5) Check whether the observation period C has been reached. Specifically, observe every C steps. If the optimization step number i>1 and the optimization step number i modulo the observation period C is 0, indicating that C steps have just passed, then proceed to step (6). If the optimization step number i modulo the observation period C is not 0, then proceed to step (10);

[0101] (6) Checking the denoising error Whether it decreases, specifically, if If the optimization is not successful, proceed to step (7); This indicates that the optimization is successful, and step (8) is performed;

[0102] (7) If Then the weight λ decays by a multiple of λ / 2, the patience value P decays by P-1, and then jumps to step (9);

[0103] (8) If Then the weight λ increases by the weight increment δ, and the patience value returns to the patience period ρ;

[0104] (9) Previous denoising error To update, Perform step (11);

[0105] (10) Increase the weight δ appropriately;

[0106] (11) If the patience value P = 0, jump to step (13), otherwise proceed to step (12) to update the optimization step number i;

[0107] (12) updating the optimization step number i. Specifically, the optimization step number i is optimized with an increment of 1. If the optimization step number i is greater than the total number of iteration steps S, proceed to step (13); otherwise, return to step (2) for iteration;

[0108] (13) The optimized generation result is tested. Specifically, from the prompt text distribution q φ (w|x0) and noise distribution Sample prompt text and random noise to generate images, observe these generated images manually or use a third-party image matching model to compare; if the generated image is very similar to the image to be evaluated, output the prompt text distribution qφ(w|x0) and the noise distribution and normality regularization If there is an image that is different from the image to be evaluated, the prompt text distribution q is output φ (w|x0), noise distribution and +∞.

[0109] Based on the adaptive optimization algorithm, the optimal prompt text distribution and noise distribution are obtained to minimize the weighted objective function, which is expressed as:

[0110]

[0111] where φ * Expressed as the optimal prompt text distribution parameter, Denotes the optimal noise distribution parameter, λ * is represented as the optimal weight.

[0112] The memory problem measurement index function is solved based on the optimal prompt text distribution and noise distribution, the memory problem measurement index is obtained, and the measurement of image memory problems is realized.

[0113] In order to solve these problems, the present invention also provides a device for measuring memory problems based on the inverse Vincent graph diffusion model, including a memory and a processor, the memory being used to store a computer program, and the processor being used to implement a method for measuring memory problems based on the inverse Vincent graph diffusion model when executing the computer program.

[0114] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A measurement method for the memory problem of the inverse Wensheng graph diffusion model, characterized in that: The following steps are involved: The noise distribution is constructed based on the noise adding process of the image to be evaluated in the Vignette Diffusion Model, and the prompt text distribution is constructed based on the text encoder of the Vignette Diffusion Model. The sensitivity of the noise distribution is defined as the probability of generating the image to be evaluated by combining the random noise from the noise distribution and the prompt text sampled from the prompt text distribution, and a memory problem measurement indicator function is constructed based on the sensitivity; A weighted objective function is constructed by using a denoising error constructed based on the prompt text distribution and the noise distribution and a normality regularization constructed based on the noise distribution; The weighted objective function is optimized through an adaptive optimization algorithm to obtain the optimal prompt text distribution and noise distribution. The memory problem measurement index function is calculated based on the optimal noise distribution to obtain the memory problem measurement index.

2. The method for measuring memory problems according to claim 1, characterized in that: The text encoder based on the text graph diffusion model constructs the prompt text distribution, including: (1) The word segmenter in the text encoder encodes the text in string format into a prompt text w of length L, wherein the prompt text w is a word sequence w = [w1, w2, ..., w L ]express; (2) Based on the category distribution of each word, obtain the category distribution probability of each word [π i，1 , π i，2 , …, π i，V ],∑ j＝1 π i，j =1,j∈[0,V],π i，j represents the j-th category probability of the i-th word, V represents the category type, and the category probability is parameterized as logπ i，j =φ i，j ,φ i，j represents the,th category hint text distribution parameter of the i-th word; (3) Smoothly sample the category distribution probability of each word and construct the smoothed word based on the smoothed sample and category embedding vector The word vector The word vector sequence composed of all word vectors Input to the encoding network to obtain the text feature c = f(e(w)), and construct the prompt text distribution q with parameter φ using the prompt text w in the text feature and the image to be evaluated x0 φ (w|x0).

3. The method for measuring memory problems according to claim 2, characterized in that: The above method performs smooth sampling on the category distribution of each word unit and constructs the smoothed word unit based on the smoothed sample and the category embedding vector. The word vector include: Use Gumbel-Softmax reparameterization to smoothly sample the category distribution: Among them, g i，j and g i，k Random sampling from Gumbel (0, 1) distribution, each g i，j Independent of each other, π i，k represents the k-th category probability of the i-th word, τ is a constant, called the temperature factor; V represents the category type, Represents the distribution from category π i A smoothed sample of Indicates the j-th category weight corresponding to the ith sample. Construct smoothed word units based on smoothed samples and category embedding vectors e(j) The word vector 4. The method for measuring memory problems according to claim 1, characterized in that: The memory problem measurement index function constructed based on sensitivity is: in, Represents the sensitivity of the noise distribution, specifically from the parameter The noise distribution The probability of generating the image x0 to be evaluated is generated by the randomly sampled Gaussian noise ∈ in and the hint text w sampled from the hint text distribution qφ(w|x0) with parameter φ, Represents the image x0 to be evaluated and the noise distribution parameters The constructed normality regularization, memory problem metric function indicates that there is at least one prompt text w, and the sensitivity When x is 1, the minimum value of the normality regularization is used as the memory problem metric mem(x0).

5. The method for measuring memory problems according to claim 4, characterized in that: The normality regularization is measured by using the KL divergence to measure the noise distribution The distance D from the standard normal distribution KL It means that when the noise distribution The fitting is with mean μ and variance σ 2 Normal distribution N(μ, σ 2 )hour, Normality The calculation method is:

6. The method for measuring memory problems according to claim 1, characterized in that: The denoising error constructed based on the prompt text distribution and the noise distribution is expressed as l de : Where, ∈ represents the noise distribution Gaussian noise randomly sampled in, t represents the moment randomly sampled from the time uniform distribution U(1, T), T represents the maximum moment, x t represents the noisy image of the image to be evaluated x0 at time t in the forward diffusion noisy process in the Wensheng graph diffusion model, ∈ θ (x t , t, f(e(w))) represents the value based on x t , t, f(e(w)) is the prediction noise calculated by the neural network with parameter θ, f(e(w) is the text feature, α t Represents a constant related to time t, with a value between 0 and 1.

7. The method for measuring memory problems according to claim 1, characterized in that: The weighted objective function is expressed as: Among them, L represents the denoising error Normality and the weight λ to construct the weighted objective function.

8. The method for measuring memory problems according to claim 1, characterized in that: The method of optimizing the weighted objective function by an adaptive optimization algorithm to obtain the optimal prompt text distribution and noise distribution includes: (1) Initialize internal variables, specifically, the prompt text distribution parameter φ is 0, the noise distribution parameter It obeys the normal distribution N(0, I), the weight λ is 1, and the previous denoising error tends to positive infinity, the patience value P is the tolerance period ρ, and the number of optimization steps i is 1; (2) Based on the initialized internal variables, calculate the weighted objective function (3) Calculate the weighted objective function L relative to the prompt text distribution parameter φ and the noise distribution parameter Gradient and (4) Based on the gradient descent method, the prompt text distribution parameter φ and the noise distribution parameter Update to obtain the updated prompt text distribution parameter φ' and noise distribution parameter Specifically Where γ is the learning rate, which is used to control the step size of parameter update; (5) Check whether the observation period C has been reached. Specifically, observe every C steps. If the optimization step number i>1 and the optimization step number i modulo the observation period C is 0, indicating that C steps have just passed, then proceed to step (6). If the optimization step number i modulo the observation period C is not 0, then proceed to step (10); (6) Checking the denoising error Whether it decreases, specifically, if If the optimization is not successful, proceed to step (7); This indicates that the optimization is successful, and proceed to step (8); (7) If Then the weight λ decays by a multiple of λ / 2, the patience value P decays by P-1, and then jumps to step (9); (8) If Then the weight λ increases by the weight increment δ, and the patience value returns to the patience period ρ; (9) Previous denoising error To update, Perform step (11); (10) Increase the weight δ appropriately; (11) If the patience value P = 0, jump to step (13), otherwise proceed to step (12) to update the optimization step number i; (12) updating the optimization step number i. Specifically, the optimization step number i is optimized with an increment of 1. If the optimization step number i is greater than the total number of iteration steps S, proceed to step (13); otherwise, return to step (2) for iteration; (13) The optimized generation result is tested. Specifically, from the prompt text distribution q φ (w|x0) and noise distribution Sample prompt text and random noise to generate images, observe these generated images manually or use a third-party image matching model to compare; if the generated image is very similar to the image to be evaluated, output the prompt text distribution qφ(w|x0) and the noise distribution and normality regularization If there is an image different from the image to be evaluated, the prompt text distribution qφ(w|x0) and the noise distribution are output and +∞.

9. The method for measuring memory problems according to claim 8, characterized in that: The optimal prompt text distribution and noise distribution are obtained to achieve weighted objective function minimization, which is expressed as: where φ * Expressed as the optimal prompt text distribution parameter, Denotes the optimal noise distribution parameter, λ * is represented as the optimal weight.

10. A device for measuring memory problems based on an inverse Wensheng graph diffusion model, comprising a memory and a processor, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the method for measuring memory problems based on the inverse Vincent graph diffusion model as described in any one of claims 1 to 9 when executing the computer program.

Images