Optimization method and device of initial noise distribution, equipment, medium and program

Abstract

The invention relates to the field of generative artificial intelligence image processing, and provides an initial noise distribution optimization method, device, equipment, medium and program, and the method comprises the steps: taking a denoising process as a fixed mapping relation, creating a trainable distribution parameter, and constructing a distribution parameter updating formula based on the fixed mapping relation; constructing a dynamic reward calibration module, calculating a difference value between a reward value of a current initial noise distribution generated image and a reward value of an original standard normal distribution generated image after the diffusion model outputs the generated image every time, and taking the difference value as a relative reward value; constraining the updating process of the distribution parameters by adopting a proportional clipping algorithm; and calculating a parameter updating step length based on a gradient estimation result, and carrying out proportional clipping on the updating step length. The method is used for improving the consistency of content and prompt semantics in a text-to-image generation task by optimizing the initial distribution parameters of the diffusion model, and meanwhile, the generation quality and the calculation efficiency are kept.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of generative artificial intelligence image processing, and in particular to an initial noise distribution optimization method, device, equipment, medium and program. BACKGROUND

[0002] In related technologies, large-scale pre-training diffusion models have made breakthrough progress in the field of image generation due to their excellent generation ability. These models can generate high-fidelity and widely-conceptual visual content by utilizing massive network image-text paired data. However, it is found in practice that although diffusion models exhibit strong generation potential, their output results tend to be biased towards common objects and scenes that frequently appear in the data set, rather than strictly following the semantics of the text prompt provided by the user. This bias leads to a significant difference between the generated image content and the user's intention when dealing with unconventional prompts such as "a dog on the moon", which seriously affects the practicality and user experience of the model.

[0003] To address this challenge, existing research mainly follows two technical routes: the first type of method introduces additional control signals such as depth maps, edge maps, or segmentation masks to guide the generation process, which improves content controllability in specific scenarios, but significantly increases computational burden and user operation complexity, requiring additional auxiliary models for preprocessing and being difficult to adapt to diverse prompts in open domains; the second type of method uses reinforcement learning techniques to directly optimize diffusion model parameters, enhancing text-image alignment by iteratively fine-tuning network weights, which can improve consistency without additional training data, but due to the need to adjust the large parameter system of the diffusion model, a single prompt optimization often takes several hours, requiring high computational resources, and easily leading to degradation of original generation ability, causing training instability problems. In addition, these methods usually lack effective utilization mechanisms for historical optimization experience, have low sample efficiency, and lack effective constraint strategies in the parameter updating process, making it difficult to balance generation quality and prompt consistency.

[0004] It is worth noting that the iterative denoising process of the diffusion model essentially constitutes a mapping chain from the initial noise to the final image, and the selection of the initial noise distribution has a decisive influence on the generation result. In theory, customizing the optimal initial noise distribution for different text prompts may improve generation consistency without modifying the pre-training model parameters. However, due to the multi-step non-linear transformation involved in the diffusion process, traditional gradient backpropagation cannot effectively act on the initial noise, and there is a lack of stable control mechanism for the optimization process, making this direction face serious challenges in practice. Existing technologies have not yet solved how to efficiently and stably optimize the initial noise distribution while maintaining the original generation ability of the diffusion model to meet the diverse text prompt requirements, which has seriously restricted the reliability and universality of diffusion models in practical application scenarios.

[0005] Therefore, there is an urgent need for an initial noise distribution optimization method, device, equipment, medium and program to improve the above problems. SUMMARY

[0006] The present application provides an initial noise distribution optimization method, device, equipment, medium and program, which is used to improve the consistency of content and prompt semantics in the text-to-image generation task without modifying the pre-training model structure by optimizing the initial distribution parameters of the diffusion model, while maintaining the generation quality and computational efficiency.

[0007] According to a first aspect of an embodiment of the present application, an initial noise distribution optimization method is provided, comprising: defining a distribution parameter of a trainable initial noise using the deterministic sampling characteristics of a diffusion model; constructing a distribution parameter update formula based on the mapping relationship from the initial noise to the generated image implied by the deterministic sampling characteristics; constructing a dynamic reward calibration module, after the diffusion model outputs a generated image each time, calculating the difference between the reward value of the generated image of the current initial noise distribution and the reward value of the generated image of the original standard normal distribution through a pre-trained text-image matching model and a multi-layer perception network, taking the difference as a relative reward value, and determining the adjustment direction of the initial noise parameter according to the sign of the relative reward value; using a proportional clipping algorithm to constrain the update process of the distribution parameter, integrating historical training data through an importance sampling mechanism to generate a gradient estimation result; calculating a parameter update step based on the gradient estimation result, and proportionally clipping the update step to control the amplitude of each update of the distribution parameter, so that the KL divergence between the updated distribution parameter and the standard normal distribution parameter is kept within a pre-set threshold range; performing the above steps multiple times according to a pre-set iteration number or a pre-set convergence condition, and applying the updated distribution parameter to the diffusion model to generate a final image after each iteration is completed.

[0008] In one embodiment, based on the mapping relationship from the initial noise to the generated image implied by the deterministic sampling characteristics, the distribution parameter update formula is constructed, comprising: modeling the distribution optimization step of the initial noise as a Markov decision process, which is defined by a five-tuple (S, A, , P, R), wherein: S represents a state space, corresponding to a set of text prompts input by a user; A represents an action space, corresponding to a parameter space of the initial noise distribution; represents an initial state distribution, corresponding to the statistical distribution of the text prompts of the training data; P represents a state transition function, corresponding to the deterministic denoising mapping relationship of the diffusion model; R represents a reward function, corresponding to the semantic matching degree between the generated image and the text prompt.

[0009] In one embodiment, the distribution optimization step of the initial noise is modeled as a Markov decision process, satisfying:

[0010]

[0011] where s is a specific state in the state space S, c is the text input by the user; a is a specific action in the action space A, is the initial noise of the diffusion model; parameter is the distribution parameter, including one of the mean and variance, is the current policy, is the conditional probability distribution adjusted based on the parameter , that is, the action taken at the current step under the previous state and action; r is the reward value obtained after the reward function R is calculated; is the final image generated at the end of the diffusion process; is the semantic similarity score calculated based on the pre-trained text-image matching model.

[0012] In an embodiment, the updating process of the distribution parameter is constrained by using a proportional clipping algorithm, and the historical training data is integrated to generate a gradient estimation result by using an importance sampling mechanism, including: calculating a policy gradient, wherein the expectation of the policy gradient is represented as:

[0013] wherein, is the value function under the policy π; t is the time step; T is the upper limit of time; is the state variable of the time step t; is the action variable of the time step t; R is the reward function; is the expectation value of all possible trajectories , trajectory is sampled by the theoretical policy π, and the theoretical policy π is instantiated as the current policy ; the initial noise distribution is set as the parameter of the policy gradient by using an initial noise distribution fine-tuning algorithm, satisfying:

[0014] The parameter of the policy gradient is optimized to satisfy:

[0015] wherein, parameter is the distribution parameter, including one of the mean and variance, is the gradient operator with respect to the parameter θ; is the expectation value under the policy , is the initial action; is the reward function; To optimize the operator, parameters are found that maximize the reward function .

[0016] In an embodiment, the direction of adjustment of the initial noise parameter is determined according to the sign of the relative reward value, including: setting a multi-layer perceptron network g, the optimization reward value of the current sample is defined as:

[0017] Wherein r is the reward value, the expected reward of the multi-layer perceptron network g is defined as , wherein the loss function of the multi-layer perceptron network g is defined as:

[0018] Wherein m is the number of samples of the current optimization iteration, k is a positive integer in [1, m].

[0019] In an embodiment, the amplitude of each update of the distribution parameter is controlled, including the current policy Loss function is defined as:

[0020] Wherein, is the historical strategy adopted by the current policy ; is the expected value of the historical strategy ; The gradient form of the loss function satisfies:

[0021] Wherein the parameter is the distribution parameter, including one of the mean and variance, is the gradient operator with respect to the parameter θ.

[0022] In an embodiment, the amplitude of each update of the distribution parameter is controlled, so that the KL divergence between the updated distribution parameter and the standard normal distribution parameter is kept within a preset threshold range, including: The loss function of the current policy after updating the amplitude limit satisfies:

[0023]

[0024] Wherein, is the amplitude of each update.

[0025] ​According to a second aspect of the embodiments of the present application, an initial noise distribution optimization device is provided for the method of any one of the first aspect, and the device comprises: a noise distribution mapping unit configured to define distribution parameters of a trainable initial noise by using a deterministic sampling property of a diffusion model; construct a distribution parameter update formula based on a mapping relationship from the initial noise to a generated image implied by the deterministic sampling property; a dynamic reward unit configured to construct a dynamic reward calibration module, calculate a difference value between a reward value of a generated image of the initial noise distribution and a reward value of a generated image of an original standard normal distribution after each output of the generated image by the diffusion model through a pre-trained text-image matching model and a multi-layer perception network, take the difference value as a relative reward value, and determine an adjustment direction of the initial noise parameters according to a sign of the relative reward value; an update constraint unit configured to constrain an update process of the distribution parameters by using a proportion clipping algorithm, integrate historical training data by using an importance sampling mechanism to generate a gradient estimation result, calculate a parameter update step based on the gradient estimation result, and proportionally clip the update step to control an amplitude of each update of the distribution parameters, so that a KL divergence between the updated distribution parameters and the standard normal distribution parameters is kept within a preset threshold range; and an iteration control unit connected to the noise distribution mapping unit, the dynamic reward unit and the update constraint unit, configured to control the noise distribution mapping unit, the dynamic reward unit and the update constraint unit to perform multiple iteration calculations according to a preset iteration number or a preset convergence condition, and apply the updated distribution parameters to the diffusion model to generate a final image after each iteration is completed.

[0026] According to a third aspect of the embodiments of the present application, an electronic device is provided, comprising a memory and a processor, the memory is configured to store a computer program executable by the processor; and the processor is configured to execute the computer program in the memory to implement the above-mentioned method.

[0027] According to a fourth aspect of the embodiments of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, when the executable computer program in the storage medium is executed by a processor, the above-mentioned method can be implemented.

[0028] According to a fifth aspect of the embodiments of the present application, a computer program product is provided, comprising a computer program, and the computer program is executed by a processor to implement the above-mentioned method.

[0029] Compared with the prior art, the beneficial effects of the present application are that by fully utilizing the deterministic sampling characteristics of the diffusion model, the denoising process is regarded as a fixed mapping relationship, so that the initial distribution parameters can be directly optimized, and complex adjustment of the entire diffusion model is avoided. On this basis, the dynamic reward calibration module is constructed, which compares the quality difference of the current noise distribution and the standard normal distribution generated image, and calculates the relative reward value. This mechanism can accurately determine the parameter adjustment direction and effectively improve the semantic consistency of the generated image and the text prompt. At the same time, the proportional clipping algorithm combined with the importance sampling mechanism can not only reuse historical training data to improve sample efficiency, but also constrain the parameter update amplitude to ensure that the optimized noise distribution and the original distribution maintain a reasonable distance, thereby maintaining the stability of the generated quality. Through multiple optimizations of the preset iteration number or convergence condition, this method can significantly improve the content controllability while maintaining the original generation ability of the diffusion model, solving the problems of inconsistency between the generated results and the prompts, unstable training and large consumption of computing resources in the prior art. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a flow chart of an initial noise distribution optimization method according to an exemplary embodiment.

[0031] Figure 2 is a structural schematic diagram of an initial noise distribution optimization image generation model according to an exemplary embodiment.

[0032] Figure 3 is a relative reward-based initial noise distribution optimization schematic diagram according to an exemplary embodiment.

[0033] Figure 4 is a structural block diagram of an initial noise distribution optimization device according to an exemplary embodiment.

[0034] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment.

[0035] FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. 1, initial noise distribution optimization device; 101, noise distribution mapping unit; 102, dynamic reward unit; 103, update constraint unit; 104, iteration control unit; 900, electronic device; 922, processing component; 926, power supply component; 932, memory; 950, network interface; 958, input and output interface. DETAILED DESCRIPTION

[0036] Unless otherwise defined, the technical or scientific terms used in this specification should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. Specific embodiments of the invention will be described below with reference to the accompanying drawings. It should be noted that, in order to provide a concise description, this specification cannot provide a detailed description of all features of the actual embodiments. Without departing from the spirit and scope of the invention, those skilled in the art can make modifications and substitutions to the embodiments of the invention, and the resulting embodiments are also within the protection scope of the invention.

[0037] like Figure 1 As shown, the first embodiment of the present invention provides a method for optimizing the initial noise distribution, including the following steps S1-S4: S1. Utilizing the deterministic sampling characteristics of the diffusion model, the distribution parameters of the trainable initial noise are defined; based on the mapping relationship from the initial noise to the generated image implied by the deterministic sampling characteristics, the distribution parameter update formula is constructed.

[0038] S2. Construct a dynamic reward calibration module. After each output image generated by the diffusion model, calculate the difference between the reward value of the image generated by the current initial noise distribution and the reward value of the image generated by the original standard normal distribution through a pre-trained text-image matching model and a multilayer perceptron network. Use this difference as the relative reward value and determine the adjustment direction of the initial noise parameters based on the sign of the relative reward value.

[0039] S3 employs a proportional pruning algorithm to constrain the update process of the distribution parameters. It integrates historical training data through an importance sampling mechanism to generate gradient estimation results. Based on the gradient estimation results, it calculates the parameter update step size and performs proportional pruning on the update step size to control the magnitude of each update of the distribution parameters, so that the KL divergence between the updated distribution parameters and the standard normal distribution parameters remains within a preset threshold range.

[0040] S4. Execute steps S1-S3 multiple times according to the preset number of iterations or preset convergence conditions. After each iteration, apply the updated distribution parameters to the diffusion model to generate the final image.

[0041] In some specific embodiments, after the user inputs a prompt to enter the diffusion model to generate an image and a video related thereto, a denoising diffusion implicit model (DDIM) is used to obtain a noise-disturbed image. Due to the characteristics of DDIM itself, the quality of the generated image after denoising is determined by the distribution parameters of the initial noise. To ensure that the optimization of the noise distribution is indeed effective and theoretically sound, the denoising process can also be considered as an environmental freeze. Then, the initial noise distribution is set as the parameter of the policy gradient by using a Fine-tuning Initial Noise Distribution (FIND) formula, so that the final image obtained by DDIM under different noise can be optimized for rewards, and a more consistent image or video that conforms to the user input prompt is finally obtained. The FIND formula is set to optimize the distribution of the initial noise as a parameter, so that the noise distribution of the diffusion model can be optimized. In the FIND formula, DDIM is used as the diffusion model to effectively affect the generated image and video by optimizing the distribution of the noise parameter. It is explained that the distribution of the initial noise provides a priori for the consistency of the generated image or video.

[0042] It is worth noting that the dynamic reward calibration module (DRCM) is integrated after the DDIM generates the image model in the present embodiment. The DRCM is composed of a pre-trained text-image matching model and a multi-layer perceptron (MLP) network. It solves the inherent limitations of the pre-trained reward model. Since the absolute reward value generated by the pre-trained model lacks a clear quality dividing standard, i.e., the zero value cannot accurately distinguish the quality of the generated image, the DRCM dynamically predicts the expected reward of the current initial noise distribution through the MLP network, and converts the absolute reward into a relative reward value, i.e., the difference between the current sampling reward and the expected reward. This relative reward mechanism provides an accurate gradient direction for the optimization of the initial noise distribution, so that the noise parameter can be effectively adjusted at each iteration, gradually guiding the generated image to more accurately meet the semantic requirements of the user input prompt, while avoiding the optimization direction deviation problem caused by inconsistent reward scales in traditional methods. In some examples, the MLP is set to three layers.

[0043] In some embodiments, based on the mapping relationship from the initial noise to the generated image implied by the deterministic sampling characteristics, a distribution parameter update formula is constructed, including: The distribution optimization step of the initial noise is modeled as a Markov decision process, which is composed of a five-tuple (S, A, , P, R) defined as follows: S represents a state space, corresponding to a set of text prompts input by a user; A represents an action space, corresponding to a parameter space of an initial noise distribution; represents an initial state distribution, corresponding to a statistical distribution of text prompts in training data; P represents a state transition function, corresponding to a deterministic denoising mapping relationship of the diffusion model; and R represents a reward function, corresponding to a semantic matching degree between a generated image and a text prompt.

[0044] In some embodiments, the step of optimizing the distribution of the initial noise is modeled as a Markov decision process, and in a reinforcement learning Markov decision process (MDP) formula, the FIND formula satisfies:

[0045]

[0046] where s is a specific state in the state space S, c is a text input by a user, a is a specific action in the action space A, is an initial noise of the diffusion model; and parameters is a distribution parameter, including one of a mean and a variance, is a current policy, is a conditional probability distribution adjusted based on the parameters , that is, an action taken at the current step under the previous state and action; and r is a reward value obtained after the reward function R is calculated. is a final image generated at the end of the diffusion process; is a semantic similarity score calculated based on a pre-trained text-image matching model.

[0047] In some specific embodiments, the DDIM diffusion model is used as the basic framework, which fully utilizes the key feature of deterministic sampling. Under the condition of the same initial noise and text prompt, the model will always generate completely consistent output images. Based on this feature, the optimization process of the initial noise distribution is constructed as a Markov decision process framework of reinforcement learning, where the entire denoising process is regarded as an environmental interaction process. Specifically, in each optimization iteration, the system first samples an initial noise action from the current policy, then generates the final image by performing the complete denoising process through the DDIM model, then evaluates the matching degree of the generated image and the text prompt by the dynamic reward calibration module and assigns the corresponding reward value, and finally updates the noise distribution policy by using the reward signal through the policy gradient method. This optimization cycle continues, while the proportional clipping mechanism is used to ensure that the magnitude of each parameter update is reasonably constrained to prevent the policy from changing drastically and damaging the generation quality. The whole process will continue to iterate until the preset convergence criteria are met, such as the reward value tends to be stable or reaches the maximum number of iterations, and finally obtains the optimized distribution parameters that can generate highly consistent content with the user prompt.

[0048] It is worth noting that in the traditional reward mechanism, zero is usually taken as the dividing standard of reward quality, i.e. the generated result that meets the text prompt obtains a positive reward, otherwise a negative reward. However, since the pre-trained text-image matching model is used as the reward basis, the absolute reward value distribution output by the model is not centered on zero, resulting in that zero cannot accurately reflect the quality of the generated results. In this embodiment, the expected reward value under the current policy is dynamically estimated by a multi-layer perceptron network, and the absolute reward is converted into a relative reward (i.e. the difference between the current sampled reward and the expected reward), which is used as the basis for judging the optimization direction. This relative reward mechanism effectively eliminates the distribution deviation problem of the pre-trained reward model, enabling the policy optimization process to accurately identify high-quality generated samples, thereby guiding the initial distribution parameters to converge in the direction of improving the consistency of text-image.

[0049] In the optimization process of the dynamic reward calibration module (DRCM), the present embodiment introduces a ratio clipping algorithm (RCA) to adaptively constrain the strategy update amplitude. The algorithm calculates the probability ratio of the current strategy and the historical strategy, and clips the ratio to a preset threshold interval, effectively limiting the step size of parameter update. This mechanism ensures that the optimization process of the noise distribution maintains convergence efficiency while not deviating too far from the historical experience distribution due to single large step updates, thereby maintaining the KL divergence with the original standard normal distribution within a safe range. RCA not only significantly improves training stability and avoids performance degradation of the diffusion model due to overfitting, but also efficiently reuses historical training data through an importance sampling mechanism, reducing sample requirements, so that the optimization process more accurately aligns with the semantic requirements of user prompts while maintaining generation quality.

[0050] In some embodiments, the update process of the distribution parameters is constrained using the ratio clipping algorithm, and historical training data is integrated to generate gradient estimation results through an importance sampling mechanism, including: calculating a policy gradient, where the expectation of the policy gradient is represented as:

[0051] wherein, is the value function under the policy π; t is the time step; T is the upper limit of time; is the state variable at time step t; is the action variable at time step t; R is the reward function; is the expected value for all possible trajectories , and the trajectory is sampled from the theoretical policy π, which is instantiated as the current policy ; The initial noise distribution fine-tuning algorithm is used to set the initial noise distribution as the parameter of the policy gradient, satisfying:

[0052] The optimization objective of the parameter of the policy gradient satisfies:

[0053] wherein, the parameter is the distribution parameter, including one of the mean and the variance, is the gradient operator with respect to the parameter θ; is the expected value under the policy , is the initial action; is the reward function; is the optimization operator for finding the parameter that maximizes the reward function .

[0054] In some embodiments, determining the adjustment direction of the initial noise parameter according to the sign of the relative reward value comprises: setting a multi-layer perceptron network g, the optimization reward value of the current sample is defined as:

[0055] wherein r is the reward value, the expected reward of the multi-layer perceptron network g is defined as , wherein the loss function of the multi-layer perceptron network g is defined as:

[0056] wherein m is the number of samples of the current optimization iteration, and k is a positive integer in [1, m].

[0057] It is worth noting that the prior art has significant limitations in optimizing the initial noise distribution. The traditional method only updates the parameters based on the current sampling data, lacking effective use of historical optimization experience, resulting in low optimization efficiency. At the same time, the multi-step denoising feature of the diffusion model makes the calculation cost of each sampling process high, further reducing the optimization speed. More importantly, the unconstrained optimization process is easy to make the initial noise distribution deviate too far from the original standard normal distribution, causing the training and inference distribution to mismatch, ultimately leading to the decline of the generation quality. This is because the diffusion model always assumes that the initial noise follows the standard normal distribution during the training phase, and when the optimized distribution deviates significantly from this assumption, the denoising ability of the model will be severely impaired.

[0058] The proportional clipping algorithm (RCA) proposed in this embodiment solves the above problems through a double mechanism: on the one hand, the importance sampling mechanism is introduced to integrate historical training data, so that the gradient estimation can fully utilize the past optimization experience and improve the sample efficiency; on the other hand, adaptive constraints are imposed on the update amplitude of the strategy, by calculating the probability ratio of the current strategy and the historical strategy and clipping it to a pre-set threshold range, ensuring that the KL divergence between the optimized noise distribution and the original standard normal distribution remains within a safe threshold. This constraint mechanism provides a stable direction guide and controllable update step for the optimization process, preventing performance degradation caused by distribution deviation, and avoiding training shock caused by single large step update, thereby significantly improving the optimization efficiency and convergence stability while ensuring the generation quality.

[0059] In some embodiments, controlling the amplitude of each update of the distribution parameter comprises calculating the probability ratio of the current strategy loss function is defined as:

[0060] wherein, is the current policy adopted in the previous step; is the historical policy adopted in the previous step; the gradient form of the loss function satisfies:

[0061] wherein, the parameter is a distribution parameter, including one of mean and variance, is a gradient operator with respect to the parameter

[0062] In some specific embodiments, the RCA of the present embodiment is essentially different from the Direct Preference Optimization with KL divergence constraint (DPOK) method: DPOK uses the KL divergence of the initial model to adjust the degree of parameter update, so as to avoid too large difference from the original model. In contrast, RCA adopts an innovative strategy ratio clipping mechanism, which directly constrains the amplitude of parameter update by limiting the probability ratio between the current policy and the historical policy within a preset threshold range. This mechanism does not depend on the distribution difference measurement from the initial model, but is based on the dynamic experience in the optimization process for adaptive adjustment, which can effectively prevent the noise distribution from deviating too much from the original standard normal distribution, and also maintain the convergence efficiency of the optimization process, so as to ensure the generation quality while realizing more accurate semantic alignment.

[0063] In some embodiments, the amplitude of each update of the distribution parameter is controlled so that the KL divergence between the updated distribution parameter and the standard normal distribution parameter is maintained within a preset threshold range, including: the loss function of the current policy after the update amplitude is limited satisfies:

[0064]

[0065] wherein, is the amplitude of each update.

[0066] As shown in Figure 2 , a second embodiment of the present application provides an image generation model based on optimization of an initial noise distribution. The model first defines a trainable initial noise distribution, i.e. the current policy in the initial state , which is an initial noise distribution with mean and varianceParameterization is used to generate the initial noise that serves as the starting point for the diffusion process. Subsequently, this initial noise The final image is generated step by step through T-step denoising using a U-Net with frozen parameters. In this process, each denoising operation is considered an interaction with the environment, collectively forming a complete Markov Decision Process (MDP). The resulting final image... The pre-trained text-image matching model g is used for quality evaluation, and the dynamic reward calibration module calculates the optimized reward value based on the semantic matching degree between the generated image and the text prompt c. The reward value will be optimized. Feedback is sent to the strategy optimization module to dynamically adjust the initial distribution parameters. This guides the generation process to more accurately meet user expectations. Through this closed-loop optimization mechanism, the FIND method improves the semantic consistency between the generated image content and the text prompts while maintaining the original high-efficiency generation capability of the diffusion model.

[0067] In some examples, the FIND framework demonstrates how the Dynamic Reward Calibration Module (DRCM) optimizes the initial noise distribution to improve consistency between the generated image and the text prompt. First, it examines different trainable initial noise distributions. Multiple candidate samples were obtained through mid-sampling. , , ..., Each sample is based on a specific strategy. The generated images. Then, these generated images... They will be fed into a pre-trained model for evaluation to obtain a quality score, i.e., a reward value, relative to the input text prompts. To ensure the accuracy of the optimization process, a simple multilayer perceptron network g is introduced to predict the expected reward under the current policy. Next, the relative reward for each sample is calculated. This is achieved by providing the actual reward for each sample. Subtract expected reward The derived. Such a processing mode solves the problem of optimization direction deviation caused by directly using absolute rewards, because the reward values of the pre-trained model are not centered on zero. Therefore, by comparing the actual reward and the expected reward, the initial noise distribution can be effectively adjusted to more accurately optimize in the direction of improving text-image consistency. This mechanism not only enhances the stability of the optimization process, but also avoids optimization errors caused by inconsistent reward scales. Ultimately, the method can help the model generate more content that meets the user's intent.

[0068] As Figure 3 shown, the horizontal axis of the figure represents the trainable initial noise distribution , the vertical axis represents the reward value R of the generated image and the text prompt, and the three-dimensional surface represents the corresponding expected reward distribution under different distribution parameters. The blue plane in the figure represents the expected reward under the current strategy, which serves as a benchmark reference line to measure the difference between the actual reward r of each sampling point and the overall average level. When a certain sampling point corresponds to a reward r higher than the expected value , the relative reward is positive, indicating that the noise distribution is better and should be encouraged; otherwise, if , the relative reward is negative, indicating that the distribution performs poorly and needs to be adjusted. In this way, DRCM converts absolute rewards into relative rewards, effectively eliminating the problem of inconsistent reward scales of pre-trained models, enabling the optimization process to accurately identify high-quality samples and guide the initial noise distribution to converge in the direction of improving text-image consistency, thereby achieving a stable and efficient optimization goal.

[0069] As Figure 4As shown, according to the third embodiment of the present application, an initial noise distribution optimization device 1 is provided for the method of any one of the above embodiments, which comprises: a noise distribution mapping unit 101, configured to define the distribution parameters of the trainable initial noise by using the deterministic sampling characteristics of the diffusion model; construct a distribution parameter update formula based on the mapping relationship from the initial noise to the generated image implied by the deterministic sampling characteristics; a dynamic reward unit 102, configured to construct a dynamic reward calibration module, calculate the difference between the reward value of the generated image of the current initial noise distribution and the reward value of the generated image of the original standard normal distribution after each output of the generated image of the diffusion model through the pre-trained text-image matching model and the multi-layer perception network, take the difference as the relative reward value, and determine the adjustment direction of the initial noise parameter according to the sign of the relative reward value; an update constraint unit 103, configured to constrain the update process of the distribution parameter by using the proportion clipping algorithm, and integrate the historical training data by using the importance sampling mechanism to generate a gradient estimation result; calculate the parameter update step based on the gradient estimation result, and proportionally clip the update step to control the amplitude of each update of the distribution parameter, so that the KL divergence between the updated distribution parameter and the standard normal distribution parameter is kept within a preset threshold range; an iteration control unit 104 connected to the noise distribution mapping unit 101, the dynamic reward unit 102 and the update constraint unit 103, configured to control the noise distribution mapping unit 101, the dynamic reward unit 102 and the update constraint unit 103 to perform multiple iteration calculations according to a preset iteration number or a preset convergence condition, and apply the updated distribution parameter to the diffusion model to generate a final image after each iteration is completed.

[0070] The fourth embodiment of the present application provides an electronic device comprising a memory and a processor, the memory being configured to store a computer program executable by the processor; and the processor being configured to execute the computer program in the memory to implement the method of any one of the above embodiments.

[0071] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 900 can be provided as a server. The electronic device 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932, for storing instructions executable by the processing component 922, such as an application program. The application program stored in the memory 932 can include one or more modules each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute the instructions to perform the above method.

[0072] The electronic device 900 can further include a power supply component 926 configured to perform power management of the electronic device 900, a wired or wireless network interface 950 configured to connect the electronic device 900 to a network, and an input / output (I / O) interface 958. The electronic device 900 can operate based on an operating system stored in the memory 932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0073] In exemplary embodiments, a non-transitory computer readable storage medium, such as the memory 932 including instructions, is also provided, which instructions are executable by the processing component 922 of the electronic device 900 to perform the above-described methods. For example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device.

[0074] In exemplary embodiments, a non-transitory computer readable storage medium, such as the memory 932 including instructions, is also provided, which instructions are executable by the processing component 922 of the electronic device 900 to perform the above-described methods. For example, the non-transitory computer readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device.

[0075] A fifth embodiment of the present application provides a readable storage medium, the readable storage medium storing a program, the program being executed to implement the method of any one of the above-described embodiments.

[0076] A sixth embodiment of the present application provides a computer program product, including a computer program, the computer program being executed to implement the method of any one of the above-described embodiments.

[0077] In the present application, the terms "first", "second" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance. The term "a plurality of" means two or more, unless otherwise explicitly limited.

[0078] The above description of the embodiments is for the purpose of enabling one of ordinary skill in the art to understand and apply the present application. Those skilled in the art can easily make various modifications to the embodiments and apply the general principles described herein to other embodiments without having to resort to inventive labor. Therefore, the present application is not limited to the embodiments described herein, and improvements and modifications made by those skilled in the art based on the disclosure of the present application without departing from the scope and spirit of the present application are within the scope of the present application.

Claims

1. A method for optimizing initial noise distribution, characterized in that, The method includes: By utilizing the deterministic sampling characteristics of the diffusion model, the distribution parameters of the trainable initial noise are defined; based on the mapping relationship from the initial noise to the generated image implied by the deterministic sampling characteristics, the distribution parameter update formula is constructed. A dynamic reward calibration module is constructed. After each output of the diffusion model, the difference between the reward value of the image generated by the current initial noise distribution and the reward value of the image generated by the original standard normal distribution is calculated through a pre-trained text-image matching model and a multilayer perceptron network. This difference is used as the relative reward value, and the adjustment direction of the initial noise parameters is determined according to the sign of the relative reward value. The update process of the distribution parameters is constrained by a proportional pruning algorithm. Historical training data is integrated through an importance sampling mechanism to generate gradient estimation results. The parameter update step size is calculated based on the gradient estimation results, and the update step size is proportionally pruned to control the magnitude of each update of the distribution parameters, so that the KL divergence between the updated distribution parameters and the standard normal distribution parameters remains within a preset threshold range. The above steps are executed multiple times according to a preset number of iterations or a preset convergence condition. After each iteration, the updated distribution parameters are applied to the diffusion model to generate the final image.

2. The method according to claim 1, characterized in that, Based on the mapping relationship from initial noise to generated image implied by the deterministic sampling characteristics, a distribution parameter update formula is constructed, including: The initial noise distribution optimization step is modeled as a Markov decision process, which consists of a quintuple (S, A, ...). The definition of S(P,R) is as follows: S represents the state space, corresponding to the set of text prompts input by the user; A represents the action space, corresponding to the parameter space of the initial noise distribution. denoted by , representing the initial state distribution, corresponding to the statistical distribution of text prompts in the training data; P represents the state transition function, corresponding to the deterministic denoising mapping relationship of the diffusion model; R represents the reward function, corresponding to the semantic matching degree between the generated image and the text prompts.

3. The method according to claim 2, characterized in that, The step of optimizing the distribution of the initial noise is modeled as a Markov decision process, satisfying: Where s represents a specific state in state space S, c represents the text input by the user, and a represents a specific action in action space A. The initial noise for the diffusion model; parameters The distribution parameters include one of the mean and variance. As the current strategy, For parameter-based The adjusted conditional probability distribution is the action to be taken in the current step given the previous state and action; r is the reward value obtained after calculating the reward function R. The final image generated at the end of the diffusion process; The semantic similarity score is calculated based on a pre-trained text-image matching model.

4. The method according to claim 1, characterized in that, The update process of the distribution parameters is constrained by a proportional pruning algorithm, and historical training data is integrated through an importance sampling mechanism to generate gradient estimation results, including: Calculate the policy gradient, where the expectation of the policy gradient is expressed as: in, Let be the value function under strategy π; t be the time step; T be the upper time limit; For time step t, the state variables are... Here, R is the action variable at time step t; R is the reward function. For all possible trajectories The expected value of the trajectory The policy is obtained by sampling from the theoretical policy π, which is then instantiated as the current policy. ; The initial noise distribution is set as a parameter of the policy gradient using an initial noise distribution fine-tuning algorithm, satisfying the following: The objective of optimizing the parameters of the policy gradient satisfies: Among them, parameters The distribution parameters include one of the mean and variance. Let be the gradient operator with respect to the parameter θ; In the current strategy The expected value under, This is the initial action; For the reward function; To optimize the operator, parameters are used to find the parameters that maximize the reward function. .

5. The method according to claim 4, characterized in that, The adjustment direction of the initial noise parameters is determined based on the sign of the relative reward value, including: setting the multilayer perceptron network g, and the optimized reward value of the current sample. Defined as: Where r is the reward value, the expected reward of the multilayer perceptron network g. Defined as The loss function of the multilayer perceptron network g is... Defined as: Where m is the number of samples in the current optimization iteration, and k is a positive integer in [1, m].

6. The method according to claim 5, characterized in that, Controlling the magnitude of each update of the distribution parameters, including the current strategy. loss function Defined as: in, To adopt the current strategy The historical strategies previously employed; It is a historical strategy Expected value; The gradient form of the loss function satisfies: Among them, parameters The distribution parameters include one of the mean and variance. Let be the gradient operator with respect to the parameter θ.

7. The method according to claim 6, characterized in that, Controlling the magnitude of each update of the distribution parameters to keep the KL divergence between the updated distribution parameters and the standard normal distribution parameters within a preset threshold range includes: Current strategy after updating the amplitude limit loss function satisfy: in, This refers to the magnitude of each update.

8. An apparatus for optimizing the initial noise distribution, used in the method according to any one of claims 1 to 7, characterized in that, The device includes: The noise distribution mapping unit is used to define the distribution parameters of trainable initial noise by utilizing the deterministic sampling characteristics of the diffusion model; and to construct the distribution parameter update formula based on the mapping relationship from initial noise to generated image implied by the deterministic sampling characteristics. The dynamic reward unit is used to construct the dynamic reward calibration module. After each output of the diffusion model, the difference between the reward value of the image generated by the current initial noise distribution and the reward value of the image generated by the original standard normal distribution is calculated through the pre-trained text-image matching model and multilayer perceptron network. This difference is used as the relative reward value, and the adjustment direction of the initial noise parameters is determined according to the sign of the relative reward value. The update constraint unit is used to constrain the update process of the distribution parameters using a proportional pruning algorithm, integrate historical training data through an importance sampling mechanism to generate gradient estimation results, calculate the parameter update step size based on the gradient estimation results, and perform proportional pruning on the update step size to control the magnitude of each update of the distribution parameters, so that the KL divergence between the updated distribution parameters and the standard normal distribution parameters remains within a preset threshold range. An iterative control unit, connected to the noise distribution mapping unit, the dynamic reward unit, and the update constraint unit, is used to control the noise distribution mapping unit, the dynamic reward unit, and the update constraint unit to perform multiple iterative calculations according to a preset number of iterations or a preset convergence condition. After each iteration, the updated distribution parameters are applied to the diffusion model to generate the final image.

9. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program executable by the processor; and the processor executes the computer program in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the executable computer program in the storage medium is executed by a processor, it can implement the method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images