Method for guiding sampling from diffusion model by energy function based on contrastive learning
Patent Information
- Application Number
- CN202310363996.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-06
- Publication Date
- 2026-10-27
- Estimated Expiration
- 2043-04-06
AI Technical Summary
[0005]本发明提供一种基于对比学习通过能量函数引导从扩散模型采样的方法，用以解决现有扩散模型采样不精准导致数据样本误差大的问题
[0040]本发明提供的一种于对比学习通过能量函数引导从扩散模型采样的方法，通过精确推导出了正确的基于能量函数引导扩散模型采样的正确数学形式，基于对比学习方法，提出了不同的优化轨迹能量引导梯度函数的损失函数。实现损失函数能够保证训练损失稳定收敛，在采样效果上明显好于现有的算法。采样得到的样本点分布更加符合数据分布，会更少地出现采样到离群点的情况，并且实际应用中训练收敛速度更快，训练损失函数更加稳定。
Smart Images
Figure CN116502699B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer performance improvement technology, and in particular to a method for sampling from a diffusion model based on contrastive learning and guided by an energy function. Background Technology
[0002] Diffusion models show promising applications in realistic image synthesis, speech synthesis, molecular generation, and 3D shape generation. The key to applying diffusion models to practical tasks lies in embedding human controllability into the sampling process. In particular, a common approach to diffusion models that incorporate human preferences is "guided sampling," which includes "classifier-guided" and "classifier-free" methods.
[0003] In existing technologies, sampling variance is either "classifier-guided" or "classifier-free." Both methods require training a classifier or describing human preferences based on a conditional distribution of a conditional variable, and then guiding the sampling process according to that classifier. These techniques have been successfully applied in fields such as realistic image synthesis, speech synthesis, molecular generation, and 3D shape generation. However, in some cases, it is difficult to describe human preferences using certain conditional variables (such as image categories in image classification problems), and preferences must be embedded using a scalar function. Examples of such functions include reward functions or pre-trained Q-functions in reinforcement learning, human feedback in dialogue systems, cosine similarity functions between sample features and specified features in image synthesis, or distance functions between sample frames and the previous frame in video synthesis. In such cases, we must have a technical means to guide the sampling of the diffusion model through an "energy function," i.e., the "energy function-guided" method. Existing technologies have attempted to guide the sampling process of the diffusion model using energy functions, such as EGSDE and DPS techniques. These techniques often aim to sample data from a distribution "edited" by an energy function. To achieve a similar sampling effect, EGSDE technology needs to first learn a new energy function on a noisy data distribution and use the derivative of that energy function to guide data sampling. DPS technology, on the other hand, first predicts noisy data from noisy data and then estimates the energy function from the noisy data, thereby achieving guided sampling.
[0004] Existing techniques for sampling from diffusion models using "energy function guidance" are all manually designed and lack theoretical guarantees. In fact, it can be proven that all these existing techniques are "imprecise" and cannot guarantee that the final sampled model data distribution will converge to a predefined energy function distribution. This imprecision may cause the diffusion model to sample some data points that do not belong to the original data distribution. For example, in image generation tasks, some blurry or oddly shaped images may appear. The fundamental reason for this "imprecision" is that the energy function guidance process of existing schemes such as DPS and EGSDE is designed manually by humans based on human experience, rather than derived from rigorous theory. Therefore, the training loss function is biased, and thus can only produce data samples that are possibly correct, but cannot produce completely accurate data samples. Summary of the Invention
[0005] This invention provides a method for sampling from a diffusion model based on contrastive learning and guided by an energy function, in order to solve the problem of large data sample errors caused by inaccurate sampling in existing diffusion models.
[0006] This invention provides a method for sampling from a diffusion model based on contrastive learning and guided by an energy function, comprising:
[0007] Obtain the pre-trained diffusion model and energy function;
[0008] Predict the energy function at intermediate sampling moments under different states using a comparative learning algorithm;
[0009] Set the current time step T, and calculate the sampling gradient function using the pre-trained diffusion model and the predicted energy function;
[0010] The gradient function is written into a preset diffusion model sampling solver to generate data point positions with a time step of T-1;
[0011] The data point positions are obtained by iteratively until the current time step is 0. The obtained data point positions are the final required sampling data points.
[0012] According to the present invention, a method for sampling from a diffusion model based on contrastive learning and guided by an energy function is provided. The method for predicting the energy function at intermediate sampling times under different states using a contrastive learning algorithm specifically includes:
[0013] Simultaneously sample K data points and K time steps, as well as the energy function value corresponding to each data point at time 0, and add Gaussian noise to each data point according to the corresponding time step;
[0014] Construct an energy gradient prediction neural network, inputting data points with added Gaussian noise and corresponding time steps, and outputting a scalar energy value;
[0015] The loss function of the energy gradient prediction neural network is calculated according to the set formula.
[0016] The current energy gradient prediction neural network is optimized using the gradient backpropagation algorithm, and multiple iterations are performed to predict the energy function.
[0017] According to the present invention, a method for sampling from a diffusion model based on contrastive learning and guided by an energy function is provided, wherein the energy function at intermediate sampling times under different states is predicted by a contrastive learning algorithm, and further includes:
[0018] Simultaneously sample K data points and the additional condition variables corresponding to each data point, randomly and uniformly sample K time steps, and the energy function value corresponding to each data point at time 0. Add Gaussian noise to each data point according to the corresponding time step to obtain noisy data.
[0019] Construct an energy gradient prediction neural network, inputting data points with added Gaussian noise, the corresponding time step, and the corresponding additional condition variables, and outputting a scalar energy value;
[0020] The loss function of the energy gradient prediction neural network is calculated according to the set formula.
[0021] The current energy gradient prediction neural network is optimized using the gradient backpropagation algorithm, and multiple iterations are performed to predict the energy function.
[0022] According to the present invention, a method for sampling from a diffusion model based on contrastive learning and guided by an energy function is provided. The method involves setting the current time step T and calculating the sampling gradient function using the pre-trained diffusion model and the predicted energy function. Specifically, the method includes:
[0023] The initial sampling gradient function is estimated using the pre-trained diffusion model;
[0024] The energy-guided gradient function is defined by the energy function;
[0025] The sampling gradient function required for actual diffusion model sampling is obtained by subtracting the initial sampling gradient function from the energy-guided gradient function.
[0026] According to the present invention, a method for sampling from a diffusion model based on contrastive learning and guided by an energy function is provided, wherein the gradient function is written into a preset diffusion model sampling solver to generate data point locations with a time step of T-1, specifically including:
[0027] The sampling gradient function required for actual diffusion model sampling is entered into the preset diffusion model sampling solver;
[0028] The sampling calculation is performed using the diffusion model sampling solver to generate the location of data points with a time step of T-1.
[0029] According to the present invention, a method for sampling from a diffusion model based on contrastive learning and guided by an energy function is provided. The method involves iteratively calculating the data point positions until the current time step is 0, and the obtained data point positions are the final required sampling data points. Specifically, the method includes:
[0030] The sampling gradient function is calculated iteratively at the current time step T, and the sampling gradient function is entered into the diffusion model sampling solver to generate the data point position at time step T-1;
[0031] The data points obtained until the current time step is 0 are the final required sampling data points.
[0032] The present invention also provides a system for sampling from a diffusion model based on contrastive learning using an energy function, the system comprising:
[0033] The model acquisition module is used to acquire the pre-trained diffusion model and energy function;
[0034] The energy function calculation module is used to predict the energy function at intermediate sampling moments under different states through a contrastive learning algorithm;
[0035] The sampling gradient function calculation module is used to set the current time step T and calculate the sampling gradient function using the pre-trained diffusion model and the predicted energy function.
[0036] The data point acquisition module is used to write the gradient function into a preset diffusion model sampling solver to generate data point positions with a time step of T-1.
[0037] A sampling point determination module is used to iteratively calculate the data point positions until the current time step is 0, and the obtained data point positions are the final required sampling data points. This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method described above for sampling from a diffusion model guided by an energy function through contrastive learning.
[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for sampling from a diffusion model guided by an energy function in contrastive learning as described above.
[0039] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method for sampling from a diffusion model guided by an energy function through contrastive learning as described above.
[0040] This invention provides a method for sampling from a diffusion model using energy function-guided contrastive learning. It accurately derives the correct mathematical form of energy function-guided sampling from the diffusion model and proposes different loss functions based on the energy-guided gradient function of the optimized trajectory. The implemented loss function ensures stable convergence of the training loss and significantly outperforms existing algorithms in sampling performance. The sampled point distribution more closely matches the data distribution, resulting in fewer outliers. Furthermore, in practical applications, training convergence is faster, and the training loss function is more stable. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0042] Figure 1 This is one of the flowcharts of a method for sampling from a diffusion model guided by an energy function in contrastive learning provided by the present invention;
[0043] Figure 2 This is the second flowchart of a method for sampling from a diffusion model guided by an energy function in contrastive learning provided by the present invention;
[0044] Figure 3 This is the third flowchart of a method for sampling from a diffusion model guided by an energy function in contrastive learning provided by the present invention;
[0045] Figure 4 This is the fourth flowchart of a method for sampling from a diffusion model guided by an energy function in contrastive learning provided by the present invention;
[0046] Figure 5 This is the fifth flowchart of a method for sampling from a diffusion model guided by an energy function in contrastive learning provided by the present invention;
[0047] Figure 6 This is the sixth flowchart of a method for sampling from a diffusion model guided by an energy function in contrastive learning provided by the present invention;
[0048] Figure 7 This is a schematic diagram of the system module connection for a system of sampling from a diffusion model guided by an energy function in contrastive learning, as provided by the present invention.
[0049] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention.
[0050] Figure label:
[0051] 110: Model acquisition module; 120: Energy function calculation module; 130: Sampling gradient function calculation module; 140: Data point acquisition module; 150: Sampling point determination module;
[0052] 810: Processor; 820: Communication interface; 830: Memory; 840: Communication bus. Detailed Implementation
[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0054] The following is combined with Figures 1-6 The present invention describes a method for sampling from a diffusion model based on contrastive learning guided by an energy function, comprising:
[0055] S100, Obtain the pre-trained diffusion model and energy function;
[0056] S200, Predict the energy function at intermediate sampling time under different states using a contrastive learning algorithm;
[0057] S300. Set the current time step T, and calculate the sampling gradient function using the pre-trained diffusion model and the predicted energy function;
[0058] S400. Write the gradient function into the preset diffusion model sampling solver to generate data point positions with a time step of T-1;
[0059] S500: Loop through the data point positions until the current time step is 0. The obtained data point positions are the final required sampling data points.
[0060] This invention overcomes the inaccuracies of existing energy-guided sampling techniques from diffusion models. Based on rigorous mathematical theory, it proposes a precise and theoretically guaranteed method for energy function-guided sampling from diffusion models. It avoids manually designing sampling procedures based on human experience, instead deriving the correct sampling procedure from the theory itself.
[0061] The energy function at intermediate sampling times under different states is predicted using a comparative learning algorithm, specifically including:
[0062] S101. Simultaneously sample K data points and K time steps, as well as the energy function value corresponding to each data point at time 0, and add Gaussian noise to each data point according to the corresponding time step;
[0063] S102. Construct an energy gradient prediction neural network, inputting data points with added Gaussian noise and corresponding time steps, and outputting a scalar energy value.
[0064] S103. Calculate the loss function of the energy gradient prediction neural network according to the set formula;
[0065] The calculation formula is:
[0066]
[0067] S104. Optimize the current energy gradient prediction neural network through the gradient backpropagation algorithm, perform multiple iterations, and predict the energy function.
[0068] By using comparative learning, the energy function at intermediate sampling moments can be predicted, ensuring that the correct mathematical form of the energy function-guided diffusion model sampling is accurately derived, thus avoiding inaccuracies caused by manual design based on human experience.
[0069] The method for predicting the energy function at intermediate sampling times under different states using a comparative learning algorithm also includes:
[0070] S201. Simultaneously sample K data points and the additional condition variable c corresponding to each data point, randomly and uniformly sample K time steps, and the energy function value corresponding to each data point at time 0. Add Gaussian noise to each data point according to the corresponding time step to obtain noisy data.
[0071] S202. Construct an energy gradient prediction neural network. Input the data points with added Gaussian noise, the corresponding time step, and the corresponding additional condition variables. The output is a scalar energy value.
[0072] S203. Calculate the loss function of the energy gradient prediction neural network according to the set formula;
[0073] The calculation formula is:
[0074]
[0075] S204. Optimize the current energy gradient prediction neural network through the gradient backpropagation algorithm, perform multiple iterations, and predict the energy function.
[0076] By adding conditional variables, the energy function at intermediate sampling moments can be predicted according to actual needs, ensuring that the correct mathematical form of the energy function-guided diffusion model sampling is accurately derived, thus improving accuracy.
[0077] Set the current time step T, and calculate the sampling gradient function using the pre-trained diffusion model and the predicted energy function, specifically including:
[0078] S301. Estimate the initial sampling gradient function using the pre-trained diffusion model;
[0079] S302. Define the energy-guided gradient function using the energy function;
[0080] S303. Obtain the sampling gradient function required for actual diffusion model sampling by subtracting the initial sampling gradient function from the energy-guided gradient function.
[0081] The formula for calculating the sampling gradient function required for actual diffusion model sampling is as follows:
[0082]
[0083] The left side of the formula is the "sampling gradient function" required when actually sampling the diffusion model. The first term on the right side of the formula is the initial "sampling gradient function" of the data estimated by the pre-trained diffusion model. The second term on the right side of the formula is the "energy-guided gradient function" defined by the energy function.
[0084] The gradient function is written into a preset diffusion model sampling solver to generate data point locations with a time step of T-1, specifically including:
[0085] S401. Input the sampling gradient function required for actual diffusion model sampling into the preset diffusion model sampling solver;
[0086] S402. The sampling calculation is performed by the diffusion model sampling solver to generate the data point positions with a time step of T-1.
[0087] The data point positions are iterated until the current time step is 0. The obtained data point positions are the final required sampling data points, specifically including:
[0088] S501. Calculate the sampling gradient function at the current time step T in a loop, and input the sampling gradient function into the diffusion model sampling solver to generate the data point position at time step T-1.
[0089] S502. Continue until the current time step is 0, and the obtained data points are the final required sampling data points.
[0090] This invention provides a method for sampling from a diffusion model using energy function-guided contrastive learning. It accurately derives the correct mathematical form of energy function-guided sampling from the diffusion model and proposes different loss functions based on the energy-guided gradient function of the optimized trajectory. The implemented loss function ensures stable convergence of the training loss and significantly outperforms existing algorithms in sampling performance. The sampled point distribution more closely matches the data distribution, resulting in fewer outliers. Furthermore, in practical applications, training convergence is faster, and the training loss function is more stable.
[0091] refer to Figure 7 The present invention also discloses a system for sampling from a diffusion model guided by an energy function based on contrastive learning, the system comprising:
[0092] The model acquisition module 110 is used to acquire the pre-trained diffusion model and energy function;
[0093] The energy function calculation module 120 is used to predict the energy function at intermediate sampling moments under different states through a comparative learning algorithm.
[0094] The sampling gradient function calculation module 130 is used to set the current time step T and calculate the sampling gradient function through the pre-trained diffusion model and the predicted energy function.
[0095] The data point acquisition module 140 is used to write the gradient function into a preset diffusion model sampling solver to generate data point positions with a time step of T-1.
[0096] The sampling point determination module 150 is used to cyclically calculate the data point position until the current time step is 0, and the obtained data point position is the final required sampling data point.
[0097] Among them, the energy function calculation module simultaneously samples K data points and K time steps, as well as the energy function value corresponding to each data point at time 0, and adds Gaussian noise to each data point according to the corresponding time step;
[0098] Construct an energy gradient prediction neural network, inputting data points with added Gaussian noise and corresponding time steps, and outputting a scalar energy value;
[0099] The loss function of the energy gradient prediction neural network is calculated according to the set formula.
[0100] The current energy gradient prediction neural network is optimized using the gradient backpropagation algorithm, and multiple iterations are performed to predict the energy function.
[0101] Alternatively, simultaneously sample K data points and the additional condition variables corresponding to each data point, randomly and uniformly sample K time steps, and the energy function value corresponding to each data point at time 0, and add Gaussian noise to each data point according to the corresponding time step to obtain noisy data;
[0102] Construct an energy gradient prediction neural network, inputting data points with added Gaussian noise, the corresponding time step, and the corresponding additional condition variables, and outputting a scalar energy value;
[0103] The loss function of the energy gradient prediction neural network is calculated according to the set formula.
[0104] The current energy gradient prediction neural network is optimized using the gradient backpropagation algorithm, and multiple iterations are performed to predict the energy function.
[0105] The sampling gradient function calculation module estimates the initial sampling gradient function using the pre-trained diffusion model;
[0106] The energy-guided gradient function is defined by the energy function;
[0107] The sampling gradient function required for actual diffusion model sampling is obtained by subtracting the initial sampling gradient function from the energy-guided gradient function.
[0108] The data point acquisition module inputs the sampling gradient function required for the actual diffusion model sampling into the preset diffusion model sampling solver;
[0109] The sampling calculation is performed using the diffusion model sampling solver to generate the location of data points with a time step of T-1.
[0110] The sampling point determination module iteratively calculates the sampling gradient function at the current time step T, and inputs the sampling gradient function into the diffusion model sampling solver to generate the data point position at time step T-1;
[0111] The data points obtained until the current time step is 0 are the final required sampling data points.
[0112] This invention provides a system for sampling from a diffusion model using energy function-guided contrastive learning. The correct mathematical form of energy function-guided sampling for the diffusion model is precisely derived. Based on the contrastive learning method, different loss functions for optimizing trajectory energy-guided gradient functions are proposed. The implemented loss function ensures stable convergence of the training loss and significantly outperforms existing algorithms in sampling performance. The sampled point distribution more closely matches the data distribution, resulting in fewer outliers. Furthermore, in practical applications, training convergence is faster, and the training loss function is more stable.
[0113] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a method for sampling from a diffusion model guided by an energy function based on contrastive learning. This method includes: acquiring a pre-trained diffusion model and an energy function;
[0114] Predict the energy function at intermediate sampling moments under different states using a comparative learning algorithm;
[0115] Set the current time step T, and calculate the sampling gradient function using the pre-trained diffusion model and the predicted energy function;
[0116] The gradient function is written into a preset diffusion model sampling solver to generate data point positions with a time step of T-1;
[0117] The data point positions are obtained by iteratively until the current time step is 0. The obtained data point positions are the final required sampling data points.
[0118] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0119] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute a method provided by the above methods that is based on contrastive learning and guided by an energy function to sample from a diffusion model, the method including: obtaining a pre-trained diffusion model and an energy function;
[0120] Predict the energy function at intermediate sampling moments under different states using a comparative learning algorithm;
[0121] Set the current time step T, and calculate the sampling gradient function using the pre-trained diffusion model and the predicted energy function;
[0122] The gradient function is written into a preset diffusion model sampling solver to generate data point positions with a time step of T-1;
[0123] The data point positions are obtained by iteratively until the current time step is 0. The obtained data point positions are the final required sampling data points.
[0124] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform a method for sampling from a diffusion model based on contrastive learning guided by an energy function, as provided by the methods described above, the method comprising: acquiring a pre-trained diffusion model and an energy function;
[0125] Predict the energy function at intermediate sampling moments under different states using a comparative learning algorithm;
[0126] Set the current time step T, and calculate the sampling gradient function using the pre-trained diffusion model and the predicted energy function;
[0127] The gradient function is written into a preset diffusion model sampling solver to generate data point positions with a time step of T-1;
[0128] The data point positions are obtained by iteratively until the current time step is 0. The obtained data point positions are the final required sampling data points.
[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for sampling from a diffusion model based on contrastive learning using an energy function, characterized in that, The diffusion model is applied to realistic image synthesis, including: Obtain the pre-trained diffusion model and energy function; Predict the energy function at intermediate sampling moments under different states using a comparative learning algorithm; Set the current time step T, and calculate the sampling gradient function using the pre-trained diffusion model and the predicted energy function; The gradient function is written into a preset diffusion model sampling solver to generate data point positions with a time step of T-1; The data point positions are obtained by iteratively until the current time step is 0. The obtained data point positions are the final required sampling data points, so that the distribution of the sampled data points is more consistent with the data distribution and fewer outlier data points are sampled. Specifically, the prediction of the energy function at intermediate sampling times under different states using a contrastive learning algorithm includes: Simultaneously sample K data points and K time steps, as well as the energy function value corresponding to each data point at time 0, and add Gaussian noise to each data point according to the corresponding time step; Construct an energy gradient prediction neural network, inputting data points with added Gaussian noise and corresponding time steps, and outputting a scalar energy value; The loss function of the energy gradient prediction neural network is calculated according to a predetermined formula, which is: ； Alternatively, the setting formula is: ； The current energy gradient prediction neural network is optimized using the gradient backpropagation algorithm, and multiple iterations are performed to predict the energy function. The method of predicting the energy function at intermediate sampling times under different states using a contrastive learning algorithm further includes: Simultaneously sample K data points and the additional condition variable corresponding to each data point, randomly and uniformly sample K time steps, and the energy function value corresponding to each data point at time 0. Add Gaussian noise to each data point according to the corresponding time step to obtain noisy data. The additional condition variable is the image category. Construct an energy gradient prediction neural network, inputting data points with added Gaussian noise, the corresponding time step, and the corresponding additional condition variables, and outputting a scalar energy value; The loss function of the energy gradient prediction neural network is calculated according to the set formula. The current energy gradient prediction neural network is optimized using the gradient backpropagation algorithm, and multiple iterations are performed to predict the energy function. The step of setting the current time step T and calculating the sampling gradient function using the pre-trained diffusion model and the predicted energy function specifically includes: The initial sampling gradient function is estimated using the pre-trained diffusion model, and the formula for calculating the sampling gradient function is as follows: ； The left side of the formula is the sampling gradient function required when actually sampling the diffusion model. The first term on the right side of the formula is the initial sampling gradient function of the data estimated by the pre-trained diffusion model. The second term on the right side of the formula is the energy-guided gradient function defined by the energy function. The energy-guided gradient function is defined by the energy function; The sampling gradient function required for actual diffusion model sampling is obtained by subtracting the initial sampling gradient function from the energy-guided gradient function.
2. The method for sampling from a diffusion model based on contrastive learning using an energy function as described in claim 1, characterized in that, The gradient function is written into a preset diffusion model sampling solver to generate data point locations with a time step of T-1, specifically including: The sampling gradient function required for actual diffusion model sampling is entered into the preset diffusion model sampling solver; The sampling calculation is performed using the diffusion model sampling solver to generate the location of data points with a time step of T-1.
3. The method for sampling from a diffusion model based on contrastive learning using an energy function as described in claim 1, characterized in that, The process of iteratively calculating the data point positions until the current time step is 0, and the resulting data point positions are the final required sampled data points, specifically includes: The sampling gradient function is calculated iteratively at the current time step T, and the sampling gradient function is entered into the diffusion model sampling solver to generate the data point position at time step T-1; The data points obtained until the current time step is 0 are the final required sampling data points.
4. A system for sampling from a diffusion model based on contrastive learning using an energy function, characterized in that, The diffusion model is applied to realistic image synthesis, and the system includes: The model acquisition module is used to acquire the pre-trained diffusion model and energy function; The energy function calculation module is used to predict the energy function at intermediate sampling moments under different states through a contrastive learning algorithm; The sampling gradient function calculation module is used to set the current time step T and calculate the sampling gradient function using the pre-trained diffusion model and the predicted energy function. The data point acquisition module is used to write the gradient function into a preset diffusion model sampling solver to generate data point positions with a time step of T-1. The sampling point determination module is used to iteratively calculate the data point positions until the current time step is 0. The obtained data point positions are the final required sampling data points, so that the distribution of the sampled data points is more consistent with the data distribution and fewer outlier data points are sampled. Specifically, the prediction of the energy function at intermediate sampling times under different states using a contrastive learning algorithm includes: Simultaneously sample K data points and K time steps, as well as the energy function value corresponding to each data point at time 0, and add Gaussian noise to each data point according to the corresponding time step; Construct an energy gradient prediction neural network, inputting data points with added Gaussian noise and corresponding time steps, and outputting a scalar energy value; The loss function of the energy gradient prediction neural network is calculated according to a predetermined formula, which is: ； Alternatively, the setting formula is: ； The current energy gradient prediction neural network is optimized using the gradient backpropagation algorithm, and multiple iterations are performed to predict the energy function. The method of predicting the energy function at intermediate sampling times under different states using a contrastive learning algorithm further includes: Simultaneously sample K data points and the additional condition variable corresponding to each data point, randomly and uniformly sample K time steps, and the energy function value corresponding to each data point at time 0. Add Gaussian noise to each data point according to the corresponding time step to obtain noisy data. The additional condition variable is the image category. Construct an energy gradient prediction neural network, inputting data points with added Gaussian noise, the corresponding time step, and the corresponding additional condition variables, and outputting a scalar energy value; The loss function of the energy gradient prediction neural network is calculated according to the set formula. The current energy gradient prediction neural network is optimized using the gradient backpropagation algorithm, and multiple iterations are performed to predict the energy function. The step of setting the current time step T and calculating the sampling gradient function using the pre-trained diffusion model and the predicted energy function specifically includes: The initial sampling gradient function is estimated using the pre-trained diffusion model, and the formula for calculating the sampling gradient function is as follows: ； The left side of the formula is the sampling gradient function required when actually sampling the diffusion model. The first term on the right side of the formula is the initial sampling gradient function of the data estimated by the pre-trained diffusion model. The second term on the right side of the formula is the energy-guided gradient function defined by the energy function. The energy-guided gradient function is defined by the energy function; The sampling gradient function required for actual diffusion model sampling is obtained by subtracting the initial sampling gradient function from the energy-guided gradient function.
5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method of sampling from a diffusion model based on contrastive learning guided by an energy function as described in any one of claims 1 to 3.
6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method for sampling from a diffusion model based on contrastive learning guided by an energy function as described in any one of claims 1 to 3.
7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for sampling from a diffusion model based on contrastive learning guided by an energy function as described in any one of claims 1 to 3.