A method, device and medium for training and sampling of a controllable diffusion model

Abstract

The application discloses a training and sampling method, device and medium of a regulatable diffusion model, relates to the field of data processing, and comprises the following steps: obtaining an initial regulatable diffusion model, adding a control layer in the initial regulatable diffusion model, and obtaining a diffusion model; training the diffusion model by using a training set until the trained diffusion model meets a preset condition, and obtaining a trained diffusion model; deploying the trained diffusion model on a user end, optimizing the trained diffusion model by using the user end, obtaining a regulatable diffusion model, and sampling by using the regulatable diffusion model to obtain a sampling result. The application can prevent the diffusion model from generating harmful samples in an intermediate process.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method, device, and medium for training and sampling a regulated diffusion model. Background Art

[0002] In recent years, diffusion models have become a mainstream image generation technique, enabling them to generate a wide range of rich, vivid, and diverse images. However, this has led to challenges in preventing diffusion models from being trained to generate harmful samples and from being influenced by harmful training samples.

[0003] The primary solution currently used to address this issue is post-processing, which involves performing post-image detection and preventing harmful samples from being displayed to end users. The main drawback of this approach is that if users decompile the model after distribution and obtain the intermediate results of the diffusion model, they can directly use these intermediate results for harmful purposes. Therefore, preventing the generation of harmful samples during the diffusion model's intermediate stages has become a pressing technical challenge in this field. Summary of the Invention

[0004] The purpose of this application is to provide a supervised diffusion model training and sampling method, equipment and medium, which can prevent the diffusion model from generating harmful samples in the intermediate process.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a method for training and sampling a regulated diffusion model. The method is implemented based on the RSS framework. The method comprises:

[0007] Obtaining an initial diffusion model that can be supervised, and adding a control layer to the initial diffusion model to obtain a diffusion model;

[0008] The diffusion model is trained using a training set until the trained diffusion model meets a preset condition, thereby obtaining a trained diffusion model;

[0009] The trained diffusion model is deployed on a user terminal, the trained diffusion model is tuned by using the user terminal to obtain a regulated diffusion model, and sampling is performed using the regulated diffusion model to obtain a sampling result.

[0010] Optionally, the control layer is added between the convolutional layer and the pooling layer of the neural network architecture of the initial diffusion model.

[0011] Optionally, the expression of the control layer is:

[0012] O (l) :=γ (l) ⊙I (l) +β (l) ;

[0013] Where ⊙ is the dot product symbol, O (l) and I (l) are the output and input of the RR layer, γ (l) and β (l) are two coefficients related to the diffusion model parameters, γ (l) =U (γ) (l,:,:)Ω y (x t , pc τ )V (γ) (l,:,:),β (l) =U (β) (l,:,:)Ω y (x t , pc τ )V (β) (l,:,:),U (γ) 、V (γ) 、U (β) 、V (β) are all mapping functions, Ω y (x t , pc τ ) is the intermediate generation result of the t-th step of the diffusion model, l is the l-th layer of the neural network, x t is a matrix, pc τ A one-time password generated for the current system time τ.

[0014] Optionally, an autoregressive that only retains the encoder part is used to determine the intermediate generation result Ω of the diffusion model in the tth step y (x t , pc τ ), where Ω y (x t , pc τ )=EC(x t , pc τ ,y);

[0015] Where, EC(x t , pc τ ,y) represents the function of the autoregressive that only retains the encoder part, y is x t The label of the matrix.

[0016] Optionally, the diffusion model is trained using a training set until the trained diffusion model meets a preset condition, thereby obtaining a trained diffusion model, including:

[0017] Initialize the parameters of the diffusion model;

[0018] Taking samples from the training set, obtaining a sampling step from a uniform distribution, obtaining a sampling distribution value from a Gaussian distribution, and determining an intermediate result of a current sampling step in a diffusion model;

[0019] Get the current UNIX timestamp;

[0020] determining a mapping function based on the current UNIX timestamp and the intermediate result;

[0021] Construct the objective function;

[0022] The objective function is used to derive the parameters and mapping function of the diffusion model, and the parameters and mapping function of the diffusion model are iteratively updated through the gradient descent method to obtain a trained diffusion model, until the value change of each dimension of the parameters of the trained diffusion model is less than the set value relative to the previous cycle, thereby obtaining a trained diffusion model.

[0023] Optionally, the objective function is expressed as:

[0024]

[0025] Where L is the optimization target, is the mathematical expectation, and are switching coefficients, ∈ is the sampling distribution value, is the diffusion model after training, t is the sampling step, L is the KL distance, x t is the matrix when the sampling step is t, pc τ The one-time password generated by the current system time τ, is a Gaussian distribution, I is the unit matrix, Ω - (x t , pc τ ) and Ω + (x t , pc τ ) are state matrices related to the intermediate results and the one-time password, α t are pre-set hyperparameters, is the intermediate amount, α s is the hyperparameter when s, x t-i is the matrix when the sampling step is ti.

[0026] Optionally, a regulated diffusion model is used for sampling to obtain sampling results, including:

[0027] In the user terminal, determining an intermediate result of the diffusion model that can be supervised;

[0028] A classifier is used on the supervisor side to generate labels based on the intermediate results of the diffusion model that can be supervised;

[0029] determining whether the intermediate result of the regulated diffusion model contains harmful information based on the label;

[0030] When it is determined that harmful information exists, the training process or sampling process is blocked;

[0031] When it is determined that no harmful information exists, the intermediate result of the regulated diffusion model is iteratively modified until an initial value is obtained, and the initial value is used as the sampling result.

[0032] Optionally, when it is determined that no harmful information exists, the formula Iteratively modifying an intermediate result of the regulated diffusion model until an initial value is obtained, and using the initial value as the sampling result;

[0033] Where, is the matrix when the sampling step is t-1 in the supervised diffusion model, is an intermediate result of the diffusion model that can be regulated, is the matrix when the sampling step is t in the supervised diffusion model, ∈ θ is a regulated diffusion model, β t are the coefficients of the diffusion model that can be supervised when the sampling step is t.

[0034] In a second aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the training and sampling method of the regulated diffusion model provided above.

[0035] In a third aspect, the present application provides a non-transitory computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the training and sampling method of the regulated diffusion model provided above.

[0036] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0037] This application provides a method, device, and medium for training and sampling a regulated diffusion model. By training a diffusion model obtained by adding a control layer to an initial diffusion model, a trained diffusion model is obtained. By tuning the trained diffusion model, a regulated diffusion model can be obtained. During sampling using the regulated diffusion model, the diffusion model can be prevented from generating harmful samples mid-process, thereby preventing the diffusion model from being trained to generate harmful samples. Furthermore, by using a user end to perform model training and tune the trained diffusion model, the diffusion model can be protected from being affected by harmful training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0039] Figure 1 This is a diagram of an application environment for a training and sampling method for a regulated diffusion model in one embodiment of the present application;

[0040] Figure 2 A flowchart of a method for training and sampling a regulated diffusion model provided in one embodiment of the present application;

[0041] Figure 3 A schematic diagram of the structure of a diffusion model provided in one embodiment of the present application;

[0042] Figure 4 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0044] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0045] The training and sampling method of the supervised diffusion model provided in the embodiment of the present application can be applied to Figure 1The RSS is defined by three parties: the model owner, the user, and the regulator. The purpose of the adversarial example resistance training is to mitigate the diffusion model (see the literature “Jonathan Ho, Ajay Jain, Pieter Abbeel, Denoising Diffusion Probabilistic Models, In Proc. of NeurIPS 2020.”) from generating harmful information, or to prevent the diffusion model from being poisoned by tuning on harmful data.

[0046] From the hardware perspective, the model owner, the user, and the regulator can all be regarded as a computer. The control layer can be regarded as a device installed on the user side. This device can control the user side to perform specific training, tuning, sampling methods, and the like.

[0047] The model owner trains the diffusion model on the training data . The specific background knowledge of the diffusion model can be found in the literature “Jonathan Ho, Ajay Jain, Pieter Abbeel, Denoising Diffusion Probabilistic Models, In Proc. of NeurIPS 2020.”.

[0048] The user downloads the regulatable diffusion model ∈ θ and directly uses or tunes the regulatable diffusion model ∈ θ on private data.

[0049] The regulator is an independent third party responsible for regulating the tuning and sampling stages of the regulatable diffusion model ∈ θ to prevent the generation of harmful information. The regulator has a classifier f: x→{+,-}, the input is the intermediate result of the tuning and sampling stage of the regulatable diffusion model ∈ θ , and the output is a + / - label. The purpose is to monitor whether there is harmful information in the intermediate result of the regulatable diffusion model ∈ θ .

[0050] In an exemplary embodiment, as Figure 2 shown, a training and sampling method of a regulatable diffusion model is provided, which is executed by a computer device, specifically by a terminal or a server, or by a terminal and a server together. In the embodiments of the present application, the method is applied to the RSS framework of Figure 1 , including the following steps 200 to 202.

[0051] in:

[0052] Step 200: Obtain an initial diffusion model that can be supervised, and add a control layer to the initial diffusion model to obtain a diffusion model.

[0053] Step 201: Using a training set to train a diffusion model, until the trained diffusion model meets a preset condition, a trained diffusion model is obtained.

[0054] Step 202: deploy the trained diffusion model on the user end, use the user end to tune the trained diffusion model to obtain a regulated diffusion model, and use the regulated diffusion model to perform sampling to obtain sampling results.

[0055] Implementing steps 200 to 202 above can prevent the diffusion model from generating harmful samples during the intermediate process, thereby preventing the diffusion model from being trained to generate harmful samples. In addition, by using the user end to perform model training and fine-tuning the trained diffusion model, the present application can prevent the diffusion model from being affected by harmful training samples.

[0056] In another exemplary embodiment of the present application, a control layer is added to the U-Net neural network architecture of the diffusion model, such as Figure 2 As shown, the control layer is located between the convolutional layer and the subsequent pooling layer.

[0057] The definition of the control layer is as follows:

[0058] O (l) :=γ (l) ⊙I (l) +β (l) (1)

[0059] γ (l) =U (γ) (l,:,:)Ω y (x t , pc τ )V (γ) (l,:,:) (2)

[0060] β (l) =U (β) (l,:,:)Ω y (x t , pc τ )V (β) (l,:,:) (3)

[0061] Where ⊙ is the dot product symbol, O (l) and I (l) are the output and input of the RR layer, γ (l) and β (l)are two coefficients related to the diffusion model parameters. U (γ) , V (γ) , U (β) , V (β) are mapping functions that expand the dimensionality of Ω to the size of the input / output. Mathematically, they are changing the size of a matrix, incidentally becoming model parameters. Ω y (x t , pc τ ) is the t-th step intermediate generation result of the diffusion model in the current training / testing process, l is the l-th layer of the neural network, x t is a matrix, pc τ is a one-time password generated at the current system time τ, y is the output of the classifier f: x→{+, -}, which is the label of x t . Ω, γ (l) , β (l) The sizes of the three matrices are Ω∈R M×N , γ (l) ∈R H×W , β (l) ∈R H×W . U (γ) , U (β) ∈R L×H×M and V (γ) , V (β) ∈R L×N×W .

[0062] In another exemplary embodiment of the present application, an auto-encoder that only retains the encoder part can be used to determine the t-th step intermediate generation result Ω y (x t , pc τ ) of the diffusion model. Based on this, Ω y (x t , pc τ ) is calculated as follows:

[0063] (1) Train an auto-encoder, denoted as AE, whose input and output are both [x t , pc τ , y]. Where [.,.,.] represents the concatenation of feature vectors. The value of y is + or -, which can be replaced by +1 / -1 in actual operation. The architecture of the auto-encoder can refer to the literature “G. E. Hinton, R. R. Salakhutdinov, Reducing the Dimensionality of Data with Neural Networks. Science 313, 504-507 (2006). DOI: 10.1126 / science.1127647.”.

[0064] (2) Remove the decoder part of the auto-regressive model, keep the encoder part, denoted as function EC(.).

[0065] (3) Calculate Ω y (x t , pc τ ) using the following formula

[0066] Ω y (x t , pc τ ) = EC(x t , pc τ , y) to calculate Ω y (x t , pc τ ). (4)

[0067] In the formula, EC(x t , pc τ , y) represents the function of the auto-regressive model that only keeps the encoder part, and y is the label of the x t matrix.

[0068] In another exemplary embodiment of the present application, the training input of the diffusion model is the training set and the hyperparameter a t , and the output is the trained diffusion model Based on this, the training process of the diffusion model under the RSS framework can be described as:

[0069] (1) Initialize the parameters of the diffusion model.

[0070] (2) Take a sample x from the training set , and get a sampling step t from the uniform distribution {1, 2, 3,..., T}, a sampling distribution value from the Gaussian distribution, and determine the intermediate result of the current sampling step in the diffusion model. Wherein, the intermediate result of the t-th step in the diffusion model is: Calculate Ω - (x t , pc τ ) according to formula (4).

[0071] (3) Get the current UNIX timestamp τ.

[0072] (4) Determine the mapping function based on the current UNIX timestamp and the intermediate result.

[0073] (5) Construct the objective function. Wherein, the constructed objective function is represented as:

[0074]

[0075] In the formula, L is an optimization target, is a mathematical expectation, and are switching coefficients, and ∈ is a sampling distribution value, is a trained diffusion model, t is a sampling step, Ω - (x t , pc τ ) and Ω + (x t , pc τ ) are state matrices related to intermediate results and one-time passwords, x t is a matrix at a sampling step t, pc τ is a one-time password generated by a current system time τ, is a Gaussian distribution, and I is a unit matrix, α t is a pre-set hyperparameter, is an intermediate quantity, α s is a hyperparameter at s. x t-i is a matrix at a sampling step t-i. f is a classifier of a regulatory end defined at the beginning, and its physical meaning is that when the current sample x has f(x) = +, the second term of the optimization target is adopted to determine the optimization target, otherwise, the first term is adopted to determine the optimization target. Here, KL is a mathematical KL distance, which is used to describe the distance between two probability distributions.

[0076] (6) Derivation of the parameters and mapping functions of the diffusion model is performed by using the objective function, and the parameters and mapping functions of the diffusion model are iteratively updated by using the gradient descent method to obtain a trained diffusion model, until the value change of each dimension of the parameters of the trained diffusion model is less than a set value (for example, 10 -8 ) relative to the last cycle, and the trained diffusion model is obtained.

[0077] wherein Ω - (x t , pc τ ) is brought into the diffusion model (in the diffusion model , a control layer as described in section A has been added between the convolution layer and the pooling layer). Here, the bringing in refers to bringing Ω - (x t , pc τ ) into the calculation formulas of γ (l) and β (l)

i.e., formula (2) and formula (3)

[0078] The objective function is applied to the diffusion model parameters θ and {U (γ) , V (γ) , U (β) , V (β) By gradient descent method, update the parameters θ and {U (γ) , V (γ) , U (β) , V (β)}, and update as follows:

[0079]

[0080]

[0081]

[0082]

[0083]

[0084] Where, {U′ (γ) , V′ (γ) , U′ (β) , V′ (β)} represents the updated {U (γ) , V (γ) , U (β) , V (β)}.

[0085] After the above steps, steps (2) to (6) are to iteratively update the parameters until convergence. Step (2) is the normal operation of the diffusion model, which is used to calculate the intermediate result of the t-th step in the diffusion process defined by the diffusion model. It should be noted here that during the tuning process, it cannot be guaranteed that there are no harmful samples in the data set used for tuning, so it must pass through the classifier on the supervisor side. At this time, the algorithm blocks the training process to wait for the results. After the discrimination result is returned, the latest Ω can be calculated y (x t , pc τ ), we can calculate the parameters θ and {U (γ) , V (γ) , U (β) , V (β)} is derived, and then the model parameters are updated using the gradient descent method. Finally, the trained diffusion model is returned.

[0086] In another exemplary embodiment of the present application, the tuning algorithm is mainly deployed on the user side, and its input is a regulated diffusion model and a hyperparameter α t, the output is the sample result. Based on this, in the above step 202 of the present application, the sampling is performed using a regulated diffusion model, and the implementation process of obtaining the sampling result may include:

[0087] 1) On the user side, determine the intermediate results of the diffusion model that can be supervised.

[0088] 2) A classifier is used on the supervisor side to generate labels based on the intermediate results of the regulated diffusion model.

[0089] 3) Determine whether there is harmful information in the intermediate results of the regulated diffusion model based on the label.

[0090] 4) When it is determined that harmful information exists, the training process or the sampling process is blocked.

[0091] 5) When it is determined that no harmful information exists, the intermediate results of the regulated diffusion model are iteratively modified until the initial value is obtained, and the initial value is used as the sampling result. For example, using the formula The intermediate results of the regulated diffusion model are iteratively modified until an initial value is obtained, and the initial value is used as the sampling result.

[0092] Where, is the matrix when the sampling step is t-1 in the supervised diffusion model, is an intermediate result of the diffusion model that can be regulated, is the matrix when the sampling step is t in the supervised diffusion model, ∈ θ is a regulated diffusion model, β t are the coefficients of the diffusion model that can be supervised when the sampling step is t.

[0093] Based on the above description, in the actual citation process, the above sampling process can be described as:

[0094] 1. Sample the initial value of the sampling process from a Gaussian distribution

[0095] 2. Get the current UNIX timestamp τ.

[0096] 3. Sent to the supervisor.

[0097] 4. Block the current training process until the supervisor sends back

[0098] 5. Loop t=T, T-1, ..., 2, 1.

[0099] 6. Sampling from a Gaussian distribution

[0100] 7. Bring in a regulated diffusion model ∈ θ .

[0101] 8. Calculate the intermediate results of step t-1 in the regulated diffusion model

[0102] 9. Get the current UNIX timestamp τ.

[0103] 10. Sent to the supervisor.

[0104] 11. Block the current training process until the supervisor sends back

[0105] 12. Assignment

[0106] 13. Return x, which is the final sample generated by the model. This sample is used for subsequent experimental evaluation.

[0107] In another exemplary embodiment of this application, an experiment was conducted on the I2P benchmark dataset. I2P collects 8 types of potentially harmful (image, prompt word) pairs. Diffusion models such as stable diffusion can be induced to produce corresponding harmful images. In this embodiment, the I2P dataset is used to construct training, validation, and test sets in a 90:5:5 ratio. The experiment is divided into two parts:

[0108] Part 1: In order to verify the effectiveness of this application in preventing the diffusion model from generating harmful images, stablediffusion 1.4 was selected as the corresponding diffusion model, its architecture was modified (i.e., a control layer was added), and the proposed tuning method was used to tune the original stable diffusion training data. Then, the prompt words in the test set were used as input, and the proportion of harmful content in the sample results generated by the proposed RSS method (i.e., the training and sampling method of the regulated diffusion model provided by this application) was counted. The harmful content here is detected using the Q16 / NudeNet classifier. The experimental results are shown in Table 1 below.

[0109] Table 1 First Experimental Results

[0110] Dataset SD-v1.4 RSS-DS (items) hatred 0.40 0.04 harassment 0.34 0.04 Violence 0.43 0.10 Self-harm 0.40 0.04 sex 0.35 0.04 intimidation 0.52 0.10 criminal behavior 0.34 0.03 overall 0.39 0.07

[0111] SD-v1.4 and RSS-DS are the proportions of harmful content generated by stable diffusion based on the prompt words in the I2P test set before and after using the method provided by this application. It can be seen that the method provided by this application can effectively reduce the proportion of harmful information generated by the diffusion model.

[0112] Part 2: In order to verify the effectiveness of the method proposed in this application in preventing the model from being tuned on harmful data. This embodiment compares the ratio of the loss function values ​​(Loss-IvR) of the model on two types of data containing harmful (picture, prompt word) pairs and harmless (picture, prompt word) pairs when using and not using the RSS method after tuning on I2P. The larger this ratio is, the better it proves that the trained model can fit harmless data rather than harmful data. The experimental results are shown in Table 2 below, where the harmful data comes from the I2P dataset and the harmless samples come from the original training set of stable diffusion.

[0113] Table 2 Second Experimental Results

[0114]

[0115]

[0116] It can be seen that the method provided by this application can effectively reduce the impact of harmful data on model training, because the model is more likely to fit harmless samples rather than harmful samples.

[0117] In summary, this application presents a method that, for the first time, enables supervision of the tuning and sampling process of open-source diffusion models. This effectively mitigates the risk of diffusion models being used to generate harmful information or being poisoned by tuning on harmful data. This framework is groundbreaking and has no existing alternatives, effectively preventing the generation of harmful samples.

[0118] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store sampling results and intermediate results of a regulated diffusion model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a training and sampling method for a regulated diffusion model is implemented.

[0119] Those skilled in the art will understand that Figure 4 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution of the present application and does not constitute a limitation on the computer device to which the solution of the present application is applied. A specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the above-mentioned method embodiments when executing the computer program.

[0120] In an exemplary embodiment, a non-transitory computer-readable medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0121] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0122] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0123] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0124] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0125] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for training and sampling a supervised diffusion model, characterized in that: The training and sampling method of the regulated diffusion model is implemented based on the RSS framework; the training and sampling method of the regulated diffusion model includes: Obtaining an initial diffusion model that can be supervised, and adding a control layer to the initial diffusion model to obtain a diffusion model; The diffusion model is trained using a training set until the trained diffusion model meets a preset condition, thereby obtaining a trained diffusion model; The trained diffusion model is deployed on a user end, the trained diffusion model is tuned by the user end to obtain a regulated diffusion model, and the regulated diffusion model is used for sampling to obtain a sampling result, including: determining an intermediate result of the regulated diffusion model in the user end; using a classifier on the regulator end to generate a label based on the intermediate result of the regulated diffusion model; determining whether the intermediate result of the regulated diffusion model contains harmful information based on the label; when it is determined that harmful information exists, blocking the training process or the sampling process; when it is determined that no harmful information exists, iteratively modifying the intermediate result of the regulated diffusion model until an initial value is obtained, and using the initial value as the sampling result.

2. The method for training and sampling a supervised diffusion model according to claim 1, wherein: The control layer is added between the convolutional layer and the pooling layer of the neural network architecture of the initial diffusion model.

3. The method for training and sampling a supervised diffusion model according to claim 2, wherein: The expression of the control layer is: The (l) :=c (l) ⊙I (l) +b (l) ; Where ⊙ is the dot product symbol, O (l) and I (l) are the output and input of the RR layer, γ (l) and β (l) are two coefficients related to the diffusion model parameters, γ (l) =U (γ) (l,:,:)Ω y (x t ,pc τ )V (γ) (l,:,:), β (l) =U (β) (l,:,:)Ω y (x t ,pc τ )V (β) (l,:,:), U (γ) 、V (γ) 、U (β) 、V (β) are all mapping functions, Ω y (x t ,pc τ ) is the intermediate generation result of the t-th step of the diffusion model, l is the l-th layer of the neural network, x t is a matrix, pc τ A one-time password generated for the current system time τ.

4. The method for training and sampling a supervised diffusion model according to claim 3, wherein: The autoregressive function that only retains the encoder part is used to determine the intermediate generation result Ω of the diffusion model in the tth step y (x t ,pc τ ), where Ω y (x t ,pc τ )=EC(x t ,pc τ ,y); Where, EC(x t ,pc τ ,y) represents the function of the autoregressive that only retains the encoder part, y is x t The label of the matrix.

5. The method for training and sampling a supervised diffusion model according to claim 1, wherein: The diffusion model is trained using the training set until the trained diffusion model meets a preset condition, thereby obtaining a trained diffusion model, including: Initialize the parameters of the diffusion model; Taking samples from the training set, obtaining a sampling step from a uniform distribution, obtaining a sampling distribution value from a Gaussian distribution, and determining an intermediate result of a current sampling step in a diffusion model; Get the current UNIX timestamp; determining a mapping function based on the current UNIX timestamp and the intermediate result; Construct the objective function; The objective function is used to derive the parameters and mapping function of the diffusion model, and the parameters and mapping function of the diffusion model are iteratively updated through the gradient descent method to obtain a trained diffusion model, until the value change of each dimension of the parameters of the trained diffusion model is less than the set value relative to the previous cycle, thereby obtaining a trained diffusion model.

6. The method for training and sampling a supervised diffusion model according to claim 5, wherein: The objective function is expressed as: Where L is the optimization target, is the mathematical expectation, and are switching coefficients, ∈ is the sampling distribution value, is the diffusion model after training, t is the sampling step, KL is the KL distance, x t is the matrix when the sampling step is t, pc τ The one-time password generated by the current system time τ, is a Gaussian distribution, I is the unit matrix, Ω - (x t ,pc τ ) and Ω + (x t ,pc τ ) are state matrices related to the intermediate results and the one-time password, α t are pre-set hyperparameters, is the intermediate amount, α s is the hyperparameter when s, x t-i is the matrix when the sampling step is ti.

7. The method for training and sampling a supervised diffusion model according to claim 1, wherein: When it is determined that no harmful information exists, the formula Iteratively modifying an intermediate result of the regulated diffusion model until an initial value is obtained, and using the initial value as the sampling result; Where, is the matrix when the sampling step is t-1 in the supervised diffusion model, is an intermediate result of the diffusion model that can be regulated, is the matrix when the sampling step is t in the supervised diffusion model, ∈ θ is a regulated diffusion model, β t are the coefficients of the diffusion model that can be supervised when the sampling step is t.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the training and sampling method of the supervised diffusion model according to any one of claims 1 to 7.

9. A non-transitory computer-readable medium having a computer program stored thereon, characterized in that When the computer program is executed by a processor, the training and sampling method of the supervised diffusion model according to any one of claims 1 to 7 is implemented.

Images