Model adjusting method and device, video generating method and related equipment

Abstract

The present disclosure provides a model adjustment method and device, a video generation method and related equipment. The method comprises: determining initial video noise to be denoised by a video generation model; performing multi-round noise prediction and denoising processing on the initial video noise and a noise intensity adjustment sequence by the video generation model to obtain video noise after the last round of denoising; wherein the noise intensity adjustment sequence is generated based on a random parameter and a prediction progress parameter corresponding to multi-round noise prediction; decoding the video noise after the last round of denoising to obtain a model adjustment video; adjusting model parameters of the video generation model according to the model adjustment video to obtain an adjusted video generation model. According to the embodiments of the present disclosure, a video generation model with better performance can be obtained, and the video quality generated by the video generation model is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a model adjustment method and apparatus, a video generation method, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the continuous development of artificial intelligence technology, neural network models are widely used in various business scenarios to solve users' actual business needs; for example, in video generation scenarios, video generation models can be used to generate the videos required by users.

[0003] Before application, video generation models of related technologies may be trained using a relatively coarse and fixed training method, resulting in poor performance and low quality of the generated videos. Therefore, obtaining a high-performance video generation model has become an urgent technical problem to be solved. Summary of the Invention

[0004] This disclosure provides a model adjustment method and apparatus, a video generation method, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] In a first aspect, this disclosure provides a model adjustment method, which includes: determining initial video noise to be denoised using a video generation model; performing multiple rounds of noise prediction and denoising processing using the video generation model based on the initial video noise and a noise intensity adjustment sequence to obtain the denoised video noise after the last round, wherein the noise intensity adjustment sequence is generated based on random parameters and prediction progress parameters corresponding to the multiple rounds of noise prediction; decoding the denoised video noise after the last round to obtain a model-adjusted video; and adjusting the model parameters of the video generation model based on the model-adjusted video to obtain an adjusted video generation model.

[0006] Secondly, this disclosure provides a video generation method, which includes: using an adjusted video generation model to perform multiple rounds of noise prediction and denoising processing on the video noise to be denoised, to obtain the video noise after the last round of denoising, and decoding the video noise after the last round of denoising to obtain the video corresponding to the video noise to be denoised, wherein the adjusted video generation model is determined according to the above-mentioned model adjustment method.

[0007] Thirdly, this disclosure provides a model adjustment device, which includes:

[0008] The noise determination module is used to determine the initial video noise to be denoised using the video generation model.

[0009] The noise processing module is used to perform multiple rounds of noise prediction and denoising processing through the video generation model based on the initial video noise and the noise intensity adjustment sequence, to obtain the video noise after the last round of denoising. The noise intensity adjustment sequence is generated based on random parameters and prediction progress parameters corresponding to the multiple rounds of noise prediction.

[0010] The decoding module is used to decode the video noise after the last round of denoising to obtain the model-adjusted video.

[0011] The model adjustment module is used to adjust the video according to the model, and to adjust the model parameters of the video generation model to obtain the adjusted video generation model.

[0012] Fourthly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the model adjustment method or video generation method described above.

[0013] Fifthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described model adjustment method or video generation method.

[0014] In a sixth aspect, this disclosure provides a computer program product that includes computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the model adjustment method or video generation method described above.

[0015] The embodiments provided in this disclosure can finely adjust the intensity of noise information in multiple rounds of noise prediction and denoising processing by using a noise intensity adjustment sequence during the process of multi-round noise prediction and denoising processing in a video generation model. This allows the video generation model to perform more precise multi-round noise prediction and denoising processing on the initial video noise using the adjusted noise information. Then, the video noise after the last round of denoising is decoded to obtain a higher-quality model-adjusted video. By adjusting the model parameters of the video generation model using a higher-quality model-adjusted video, a better-performing video generation model can be obtained, thereby improving the quality of the video generated by the video generation model.

[0016] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 A flowchart of a model adjustment method provided in an embodiment of this disclosure.

[0019] Figure 2 This is a schematic diagram illustrating the application of a model adjustment method provided in an embodiment of this disclosure.

[0020] Figure 3 This is a flowchart of a video generation method provided in an embodiment of the present disclosure.

[0021] Figure 4 This is a block diagram of a model adjustment device provided in an embodiment of the present disclosure.

[0022] Figure 5 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0025] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0027] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0028] The model adjustment method according to embodiments of this disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the method can be implemented by a processor calling computer-readable program instructions stored in memory. Alternatively, the method can be executed by a server.

[0029] The meanings of the terms used in the embodiments of this disclosure can be found in the following explanations:

[0030] GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm designed for model fine-tuning. It aims to solve problems such as high computational cost and unstable training when training models, and is suitable for improving the model's ability to perform complex reasoning, mathematical, and data generation tasks.

[0031] Diffusion Model: A generative artificial intelligence model that can generate high-quality images, audio, text, and video; the data generation method can be: starting from noise, removing noise step by step, and finally generating realistic new data.

[0032] The Diffusion library is a software library or framework used to implement inference and training of diffusion models.

[0033] DDPM (Denoising Diffusion Probabilistic Models) refers to a generative model that gradually adds noise to destroy the original data and then learns the reverse process to gradually denoise and restore the data.

[0034] DDIM (Denoising Diffusion Implicit Models) refers to a generative model that significantly improves sampling speed by introducing a non-Markovian diffusion process while maintaining the same training objective as DDPM (Denoising Diffusion Probability Model).

[0035] 3D consistency refers to the metrics used to measure whether a generated video appears natural in space, such as whether the movement of objects in the video is natural, whether the structure is consistent, and whether the frames are continuous.

[0036] Perceptual loss is a metric that measures the similarity between two images at the level of human visual perception. It calculates not pixel-level differences, but rather a similarity to human visual judgment. For example, a pre-trained neural network can be used to extract "features" from two images, and the differences between these features can be compared to obtain the perceptual loss.

[0037] Linear annealing is a process where a parameter decreases from a high value to a low value at a constant rate, like a straight line; for example, the learning rate starts at 1.0 and decreases to 0.0 after 1000 steps.

[0038] Cosine annealing: This process causes the parameters to decrease in the shape of a cosine function—slowly at first, then quickly, and then slowly again, like a smooth cosine curve.

[0039] KL Penalty is a regularization technique used in machine learning. It uses KL divergence as a "penalty" to constrain the model's behavior and prevent it from deviating excessively from a pre-defined, desired distribution.

[0040] Relative Advantage: This is a standardized score that measures how well a single video performs relative to a number of other videos; Relative Advantage can be directly used to guide the updates of the policy network.

[0041] PyTorch is an open-source deep learning framework for machine learning and deep learning; a PyTorch module refers to a software module within PyTorch.

[0042] SFT (Supervised Fine-Tuning) is a technique in deep learning used to retrain pre-trained models (such as diffusion models) using labeled data, adapting them to the needs of specific tasks or domains. SFT data refers to the labeled data used for supervised fine-tuning.

[0043] ODE (Ordinary Differential Equation): Ordinary differential equation.

[0044] With the continuous development of artificial intelligence technology, neural network models are widely used in various business scenarios to solve users' actual business needs. For example, in video generation scenarios, video generation models can be used to generate the videos required by users. Before application, the video generation models of related technologies may be trained using a relatively coarse and fixed training method, resulting in poor performance and consequently, low-quality videos generated by the video generation models.

[0045] For example, the solutions to related technologies can be summarized as two parallel but non-overlapping paths: the first approach is the conventional training method of the diffusion model in video generation, and the second approach is the subsequent optimization of the diffusion model by GRPO reinforcement learning.

[0046] The first approach treats the diffusion model as a generator with frozen weights, which is no longer updated after training. In other words, the diffusion model relies entirely on predefined noise (σ_t) scheduling during the sampling phase. For example, σ_t slides from 1.0 linear or cosine to 0.0. This noise is fixed throughout and has no room for adjustment.

[0047] The second approach treats the diffusion model as a policy network (πθ) and uses the group comparison mechanism of GRPO to continue fine-tuning after freezing the weights. The process includes: first, sampling the complete video trajectory using DDIM / DDPM with fixed noise (σ); second, calculating rewards such as 3D consistency and perceptual loss all at once; and finally, updating the policy network. The drawback of this second approach is that, since σ is treated as a constant hyperparameter throughout the training cycle, random noise will still overwhelm the generated details in the later stages, leading to a decrease in the quality of policy gradient estimation.

[0048] Based on the above, it can be seen that related technologies treat the diffusion model as a generator that is frozen after training. During the sampling phase, a preset and fixed noise schedule is used; for example, σ_t slides along a fixed curve from 1.0 to 0.0, and remains unadjusted throughout. This "one-size-fits-all" approach to noise intensity leads to the following problems: 1. In the later stages of model processing, near-clean frames, random perturbations are continuously injected, meaning random perturbations are continuously added to the noise. This directly overwhelms the high-frequency textures and geometric details already generated in the noise, leading to an increase in the 3D consistency error of subsequent videos. 2. Because the noise intensity is constant, the GRPO policy gradient always carries unnecessary random components, resulting in decreased training stability.

[0049] Furthermore, it should be noted that the annealing concept in related technologies is applied to the training phase. Its function is to decay the learning rate, KL penalty, thought reward weights, or data mixing ratios over time or exponentially during training. This annealing operation never addresses the noise intensity during the sampling phase. In other words, the annealing scheme in related technologies only affects training hyperparameters such as the learning rate and reward weights, and cannot address the sampling noise itself, thus failing to simultaneously optimize sample quality and training efficiency. Moreover, while related technologies involve adjusting σ using a variable η = 0 to 1 to accelerate inference, this is only used to shorten the number of steps and is not linked to reinforcement learning training.

[0050] Figure 1 A flowchart illustrating a model adjustment method provided in an embodiment of this disclosure. (Refer to...) Figure 1 The method includes: step S11: determining the initial video noise to be denoised through a video generation model.

[0051] In this context, the video generation model can be understood as a model that generates video using noise. For example, the video generation model can be a diffusion model, a large model, a transformer model, etc. In this embodiment, the video generation model can be a trained video generation model or an untrained video generation model. When the video generation model is untrained, the initial video noise can be training samples used to train the model; that is, the initial video noise is sample video noise. When the video generation model is trained, the initial video noise can be sample video noise used for fine-tuning or secondary training of the model, such as SFT data.

[0052] In this embodiment of the disclosure, determining the initial video noise to be denoised using a video generation model includes: generating random noise using the video generation model and using this random noise as the initial video noise to be denoised. The process of generating random noise can be: initializing a random noise for video generation; or generating a random noise of a preset size (i.e., the initial video noise) in a latent space (dimensionality compression). Subsequently, the video generation model can be used to perform multiple rounds of noise prediction and denoising on the initial video noise to obtain a noise-free video (such as a model-adjusted video).

[0053] This disclosure embodiment can also sample a noise tensor from a standard normal distribution and use the noise tensor as initial video noise, the size of which is the same as the preset size.

[0054] It should be noted that the above operation of determining the initial video noise to be denoised can be performed by the video generation model; or by the noise generation network layer in the video generation model; or by a noise generation module outside the video generation model (such as a client, server, or software module).

[0055] Taking the application of the model adjustment method provided in this disclosure in generating 3D static scene video scenes as an example, the model adjustment method is explained. This disclosure embodiment can use the video generation model to generate a random noise of a preset size in a latent space (dimensional compression) as the initial video noise.

[0056] Step S12: Based on the initial video noise and the noise intensity adjustment sequence, perform multiple rounds of noise prediction and denoising processing through the video generation model to obtain the video noise after the last round of denoising; wherein, the noise intensity adjustment sequence is generated based on random parameters and prediction progress parameters corresponding to multiple rounds of noise prediction.

[0057] Among them, noise prediction processing can be understood as the operation of the video generation model to predict the noise information added to the video noise in the current round, and denoising processing can be understood as the video generation model to denoise the video noise in the current round. It should be noted that each round of denoising processing will remove part of the noise in the video noise. Through multiple rounds of denoising processing, the noise information in the video noise can be removed to obtain clean noise.

[0058] The noise intensity adjustment sequence can be understood as a sequence used to adjust the noise information (i.e., the noise information to be adjusted) predicted by the video generation model during multiple rounds of noise prediction and denoising processing. The random parameter can be understood as a randomly generated parameter; for example, it can be a random vector or a random numerical value.

[0059] The prediction progress parameter can be understood as a parameter used to characterize the prediction progress during the noise prediction processing of the video generation model. This prediction progress parameter can be the round number, such as round 1, round 50, etc.; it can also be the time step (t) in the noise prediction processing of the video generation model, such as the 500th time step (t=500), the 200th time step (t=200), etc. It should be noted that each round can correspond to one time step. For example, if the total time step T=500 in the noise prediction process of the video generation model, 500 rounds of noise prediction and denoising processing can be performed, with each round of noise prediction and denoising processing corresponding to one time step; for example, the first round of noise prediction and denoising processing corresponds to time step t=500, and the second round corresponds to time step t=499.

[0060] Specifically, in step S12, the embodiments of this disclosure can input the initial video noise into the video generation model for multiple rounds of noise prediction and denoising processing. In each round of noise prediction and denoising processing, the video generation model will predict the noise information added to the video noise in the current round (such as the noise information to be adjusted below), and use the noise intensity adjustment sequence to adjust the noise information to obtain the adjusted noise information. Then, the adjusted noise information is used to denoise the video noise. The above noise prediction and denoising processing is iteratively executed until the video noise after the last round of denoising is obtained.

[0061] In some embodiments, the video generation model needs to use a target image for video generation during the video generation process (such as model-adjusted video). The target image can be understood as the image used to generate the video. The model-adjusted video generated by the video generation model is a video associated with the target image. For example, the target image could be an image containing a water glass, and the model-adjusted video could be a video of the water glass rotating clockwise; or, for another example, the target image could be an image containing a house, and the model-adjusted video could be a video showing the exterior of the house. In this process, the video generation model can use the target image as a video generation condition and use the video generation condition to guide the generation process of the model-adjusted video. That is, in the multi-round noise prediction and denoising process, the target image (such as the water glass image) can be used as a video generation condition, allowing the video generation model to perform multi-round noise prediction and denoising on the initial video noise based on the guidance of the video generation condition, thereby obtaining a noise-free video (such as the water glass rotation video).

[0062] For example, in the first round of noise prediction and denoising processing, the initial video noise and the target image are input into the video generation model. During the multiple rounds of noise prediction and denoising, the video generation model processes the video noise based on the target image and a noise intensity adjustment sequence to obtain the denoised video noise in the final round. That is, in step S12, the initial video noise, the target image, and the noise intensity adjustment sequence can be used by the video generation model to perform multiple rounds of noise prediction and denoising to obtain the denoised video noise in the final round.

[0063] In some embodiments, each round of processing in the multi-round noise prediction and denoising process using the video generation model includes: performing noise prediction on the video noise of the current round using the video generation model to obtain the noise information to be adjusted for the current round; wherein, the video noise of the first round is the initial video noise;

[0064] According to the noise intensity adjustment parameters corresponding to the current round in the noise intensity adjustment sequence, the noise information to be adjusted in the current round is adjusted to obtain the noise information of the current round; according to the noise information of the current round, the video noise of the current round is locally denoised to obtain the video noise of the next round; wherein, if the current round is the last round, the video noise of the next round is the video noise after denoising in the last round.

[0065] Here, noise information can be understood as the information of noise added to the video noise in the current round; the video generation model obtains the noise information added to the video noise in the current round through noise prediction operations; subsequently, the video noise in the current round can be denoised based on the noise information. Through multiple rounds of noise prediction and denoising processing, clean noise can be obtained, which is used by the generation model to adjust the video; the noise information to be adjusted is the noise information that needs to be adjusted; the noise intensity adjustment parameter can be understood as the parameter used to adjust the noise intensity of the noise information to be adjusted; local denoising can be understood as the operation of removing some noise from the video noise.

[0066] Specifically, in the process of performing multi-round noise prediction and denoising processing, the video noise of the current round is input into the video generation model as input data. The video generation model performs noise prediction based on the input data to obtain the noise information to be adjusted in the current round.

[0067] It should be noted that the input data may also include the prediction progress parameters corresponding to the current round and / or video generation prompts; these prompts can be understood as information guiding the video generation model to generate the video; the prompts may be a target image, for example, an image containing a water glass, used to guide the video generation model to generate a video of the water glass rotating based on video noise; the prompts may be prompt text, for example, "Generate a video of a water glass rotating clockwise"; or, the prompts may be motion trajectory information, for example, a trajectory information of a 360-degree clockwise rotation along an axis, used to guide the video generation model to generate a video of the water glass rotating 360 degrees clockwise.

[0068] The prediction progress parameter included in the input data can be used to control the intensity of noise information predicted by the video generation model. For example, the prediction progress parameter is a time step. The larger the time step, the greater the intensity of noise information predicted by the video generation model.

[0069] After obtaining the noise information to be adjusted for the current wheel, the noise intensity adjustment parameter corresponding to the current wheel can be selected from the noise intensity adjustment sequence according to the predicted progress parameter corresponding to the current wheel, and the noise intensity adjustment parameter can be used to adjust the noise information to be adjusted for the current wheel to obtain the noise information of the current wheel.

[0070] In this embodiment of the disclosure, the noise intensity adjustment parameter includes random noise information and enhanced noise information. The random noise information can be understood as randomly generated noise information, and the enhanced noise information is noise information used to enhance the noise intensity of the noise information to be adjusted. Based on this, the operation of adjusting the noise information to be adjusted in the current round can be: determining the product of the random noise information and the enhanced noise information, and adding the product to the noise information to be adjusted.

[0071] Finally, using a preset denoising formula, the video noise of the current round is locally denoised based on the noise information of the current round to obtain the video noise of the next round; wherein, the preset denoising formula can be a back diffusion formula.

[0072] Following the example above, in the process of performing multi-round noise prediction and denoising in this embodiment of the disclosure, the current round's time step t (e.g., 100) and video noise x_t (i.e., the noise state corresponding to time step t) can be input into the video generation model.

[0073] It should be noted that the video noise x_t in the first round is the initial video noise. This initial video noise can be a "random noise", but it is an "ordered noise" that has been obtained from x_T after multiple rounds of denoising and already contains some structural information, rather than being purely random.

[0074] The noise prediction is performed using a video generation model based on the time step t of the current round and the video noise x_t, to obtain the noise information to be adjusted for the current round.

[0075] Then, from the pre-generated monotonically decreasing sequence σ_schedule∈[1,0], determine the σ_schedule[t] (i.e., enhanced noise information) and the random term ε (i.e., random noise information) corresponding to the current round (time step t). In the sampling loop of DDIM / DDPM, σ_schedule[t] is directly multiplied by the random term, and the resulting product is added to the noise information to be adjusted in the current round to obtain the adjusted noise information. Then, the adjusted noise information is used to denoise the video noise x_t, and the denoised video noise is used as the video noise of the (t-1)th round (i.e., the noise state corresponding to time step t-1). For details, please refer to Formula 1 below.

[0076] x_t-1=μθ(x_t)+σ_schedule[t]×ε (ε~N(0,1)) Formula (1).

[0077] Where x_t refers to the video noise in the current round t; μθ(x_t) refers to the noise information to be adjusted obtained by using the video generation model (μθ) to predict the noise in the video noise of the round t; σ_schedule[t] is the noise to be enhanced in the noise information to be adjusted in the round t; ε is the random term; x_t-1 refers to the video noise in the round t-1 (the next round); based on the above formula, it can be seen that after adding μθ(x_t) to the product of σ_schedule[t] and ε, the enhanced noise information can be obtained, and then the enhanced noise information is used to denoise x_t to obtain x_t-1.

[0078] It should be noted that in this embodiment, the operation of "denoising using noise information" is performed by the "sampler". The denoising operation performed by the sampler can be described as follows: After the sampler obtains x_t, the adjusted noise information, and t, it queries a predefined schedule table and obtains fixed diffusion parameters based on t. These parameters, along with x_t and the adjusted noise information, are substituted into the back diffusion formula (such as the formulas for DDPM and DDIM) to calculate x_t-1, which is the video noise of the next round.

[0079] Thus, through multiple rounds of noise prediction and denoising processing, if the current round is the last round, the video noise in the next round will be the video noise after the last round of denoising.

[0080] The above-described operation of finely adjusting the noise information to be adjusted in the current round using noise intensity adjustment parameters can be understood as an annealing operation. Based on the above embodiments, it can be seen that the embodiments of this disclosure can perform dynamic annealing on the noise intensity during the sampling phase in GRPO training. That is to say, the embodiments of this disclosure provide an annealing method for "modifying the noise intensity during the sampling phase", thereby significantly improving the video detail fidelity and reducing training noise interference without retraining the video generation model or introducing additional reward penalties, and achieving an improvement in video generation quality and training efficiency.

[0081] In some embodiments, the random parameters include a random vector, the noise intensity adjustment parameters in the noise intensity adjustment sequence include random noise information and enhanced noise information, and the prediction progress parameters include time steps.

[0082] The method further includes: determining the time step of the target round in the multi-round noise prediction, wherein the target round is any round in the multi-round noise prediction; calculating the enhanced noise information corresponding to the target round using the enhanced noise calculation formula, based on the intensity control parameters and the time step corresponding to the target round; generating a random vector using a Gaussian distribution algorithm, and determining the random noise information corresponding to the target round based on the random vector; and determining the noise intensity adjustment sequence based on the time step corresponding to the multi-round noise prediction, the enhanced noise information, and the random noise information.

[0083] The enhanced noise calculation formula can be understood as a formula used to calculate enhanced noise information. For example, the enhanced noise calculation formula can be a linear function, a cosine function, or a piecewise function. The intensity control parameter can be understood as a parameter required by the enhanced noise calculation formula in the process of calculating enhanced noise information. The intensity control parameter is used to control the noise intensity of the enhanced noise information.

[0084] Following the example above, this disclosure provides a strategy for generating annealing schedules, which means that before multiple rounds of noise prediction and denoising, a monotonically decreasing sequence σ_schedule∈[1,0] (i.e., noise intensity adjustment sequence) is pre-generated based on the total number of steps T (total time steps); the monotonically decreasing sequence supports linear, cosine or piecewise functions, and can be changed in real time during the training / inference phase.

[0085] The method for determining the monotonically decreasing sequence includes: First, determining the time step of any round (hereinafter referred to as the target round) in the multi-round noise prediction, and using the enhanced noise calculation formula, calculating the enhanced noise information corresponding to the target round based on the intensity control parameters and the time step corresponding to the target round; Second, using the Gaussian distribution algorithm, generating a random vector that follows a Gaussian distribution for the target round, and using the random vector of the target round as the random noise information of the target round.

[0086] In this way, each round of the multi-round noise prediction is processed to obtain the enhanced noise information and random noise information of the multi-round noise prediction. Then, based on the time step corresponding to the multi-round noise prediction, the enhanced noise information, and the random noise information, a monotonically decreasing sequence σ_schedule is constructed.

[0087] As can be seen from the above embodiments, this disclosure pre-constructs a noise intensity adjustment sequence, which facilitates the addition of eta_schedule during the GRPO sampling process. It generates corresponding eta weights (i.e., enhanced noise information and random noise information) based on the sampling step size, controls the noise sampling level through the eta weights, and replaces the original fixed eta parameter (noise information to be adjusted). In other words, this disclosure allows for dynamic adjustment of noise intensity during the GRPO training process at the sampling stage. By using a predefined annealing schedule, the noise intensity adjustment parameters corresponding to the current round can be quickly determined, improving the efficiency of model adjustment and gradually reducing noise intensity to enhance video generation quality.

[0088] In some embodiments, the enhanced noise calculation formula is a linear noise calculation formula, used to determine linearly decreasing enhanced noise information, and the intensity control parameters include the start time step and the end time step to be adjusted, and the total time step corresponding to the multi-round noise prediction.

[0089] The step of using the enhanced noise calculation formula to calculate the enhanced noise information corresponding to the target wheel based on the intensity control parameters and the time step corresponding to the target wheel includes: using the linear noise calculation formula to determine the enhanced noise information corresponding to the target wheel based on the start time step, the end time step, the total time step, and the time step corresponding to the target wheel.

[0090] The linear noise calculation formula can be understood as a formula used to calculate enhanced noise information that conforms to linear decrease. For example, the linear noise calculation formula can be a linear function.

[0091] The start and end time steps can be any of the total time steps. These steps define a time step range, which refers to the range within which the noise information predicted by the video generation model needs adjustment. When the video generation model reaches this time step range, the noise information predicted by the model can be adjusted by enhancing the noise information. It should be noted that in this embodiment, the noise information to be adjusted in some rounds of multi-round noise prediction and denoising can be adjusted according to the needs of the actual application; for example, adjustments can be made in the noise prediction and denoising processes from the first to the fiftieth round. Based on this, the start and end time steps can be set during the process of determining the enhanced noise information. For rounds where enhanced noise information is not needed to adjust the noise information to be adjusted, the enhanced noise information corresponding to that round still needs to be calculated; however, this enhanced noise information cannot affect the noise information to be adjusted. For example, the enhanced noise information can be 0. When σ_schedule[t] = 0, dynamic adjustment sampling can automatically degenerate into deterministic sampling. The start and end time steps to be adjusted refer to the start and end time steps that are adjusted according to the needs of the actual application.

[0092] Following the example above, the formula for calculating linear noise can be found in the following formula (2).

[0093] σ_schedule[t]=σ_start+(σ_end-σ_start)(t / T) Formula (2).

[0094] Where σ_schedule[t] refers to the enhanced noise information of the current round, σ_start is the start time step, σ_end is the end time step, t is the current time step, and T is the total time step.

[0095] As can be seen from the above embodiments, the annealing operation in this disclosure can be linear annealing. By conforming to the linearly decreasing σ_schedule, the noise information predicted by the model can be dynamically adjusted in a fine manner. Furthermore, dynamic sampling is achieved in the early stage through the start time step and the end time step, and the later stage directly switches to deterministic sampling, thereby adjusting the noise information more flexibly.

[0096] In some embodiments, the enhanced noise calculation formula is a cosine noise calculation formula, used to determine the enhanced noise information with decreasing cosine. The intensity control parameters include the start time step and the end time step to be adjusted, and the total time step corresponding to the multi-round noise prediction.

[0097] The step of using the enhanced noise calculation formula to calculate the enhanced noise information corresponding to the target wheel according to the intensity control parameters and the time step corresponding to the target wheel includes: using the cosine noise calculation formula to calculate the cosine parameter according to the total time step and the time step corresponding to the target wheel, and determining the enhanced noise information corresponding to the target wheel based on the cosine parameter, the start time step, and the end time step.

[0098] The cosine noise calculation formula can be understood as a formula used to calculate enhanced noise information that conforms to cosine decreasing. For example, the cosine noise calculation formula can be a cosine function.

[0099] Following the example above, the formula for calculating cosine noise can be found in the following formula (3).

[0100] σ_schedule[t]=0.5(σ_start+σ_end)+0.5(σ_start+σ_end)cos(tπ / T) formula (3).

[0101] Here, cos(tπ / T) refers to the cosine parameter.

[0102] As can be seen from the above embodiments, the annealing operation in this disclosure can be cosine annealing. By conforming to the cosine-decreasing σ_schedule, the noise information predicted by the model can be dynamically adjusted in a fine manner. Furthermore, dynamic sampling in the early stage is achieved through the start time step and the end time step, and the later stage directly switches to deterministic sampling, thereby adjusting the noise information more flexibly.

[0103] Step S13: Decode the video noise after the last round of denoising to obtain the model-adjusted video.

[0104] In step S13, a decoder can be used to decode the video noise after the last round of denoising to obtain the model-adjusted video. The decoder can be the decoder in the video generation model or the decoder corresponding to the video generation model.

[0105] It should be noted that after multiple rounds of noise prediction and denoising, clean noise (i.e., the video noise after the final round of denoising) can be obtained. The video noise after the final round of denoising can be understood as a denoised video generation data, used to generate the model-adjusted video. This video generation data can be a vector, matrix, etc. The video dimensions (such as the number of frames, the width and height of the video frames, the number of channels in the video frames, etc.) of the model-adjusted video can be preset.

[0106] The model adjustment video can be understood as a video used to adjust the video adjustment model. For example, the model adjustment video could be a video of a water glass rotating clockwise; or it could be a video showing the exterior of a house.

[0107] Step S14: Adjust the video according to the model, adjust the model parameters of the video generation model, and obtain the adjusted video generation model.

[0108] In some embodiments, adjusting the video according to the model and adjusting the model parameters of the video generation model to obtain the adjusted video generation model includes steps S141 to S142.

[0109] In step S141, the model adjustment video is evaluated using multiple evaluation algorithms to determine multiple video evaluation information, and the model adjustment information is determined based on the multiple video evaluation information.

[0110] The evaluation algorithm can be understood as an algorithm used to evaluate the model-adjusted video. For example, the evaluation algorithm can be an algorithm for calculating 3D consistency, an algorithm for calculating perceptual loss, etc.

[0111] Video evaluation information can be understood as the information obtained by the evaluation algorithm after evaluating the model-adjusted video. This video evaluation information can be reward information calculated based on the model-adjusted video, and the reward information can be a reward score. This video evaluation information can be used to characterize the video quality of the model-adjusted video. For example, the video evaluation information can be 3D consistency, perceptual loss, etc.

[0112] Model tuning information can be understood as information used to adjust the model parameters in a video. For example, this model tuning information can be loss or relative advantage information.

[0113] In some embodiments, the model adjusts multiple videos, the model adjustment information includes relative advantage information, and the video generation model includes a diffusion model.

[0114] The step of evaluating the model adjustment video using multiple evaluation algorithms to determine multiple video evaluation information and determining model adjustment information based on the multiple video evaluation information includes: evaluating the target model adjustment video using the multiple evaluation algorithms to obtain multiple video evaluation information corresponding to the target model adjustment video, wherein the target model adjustment video is any one of multiple model adjustment videos; determining the reward information of the target model adjustment video based on the multiple video evaluation information corresponding to the target model adjustment video; and determining the relative advantage information of the target model adjustment video using the random noise distribution information corresponding to the multi-round noise prediction and the reward information of the target model adjustment video, wherein the random noise distribution information is used to represent the distribution state of random noise information in the multi-round noise prediction, and the random noise information is determined according to the Gaussian distribution algorithm.

[0115] Following the example above, after completing multiple rounds of noise prediction and denoising, the diffusion model in this embodiment performs trajectory collection and reward operations. These operations involve: first, collecting multiple videos (i.e., the complete annealed videos) generated by the diffusion model during the multiple rounds of noise prediction and denoising. Any one of these videos is generated by the diffusion model performing multiple rounds of noise prediction and denoising. Second, using a preset reward function (i.e., multiple evaluation algorithms), 3D consistency and perceptual loss (i.e., various video evaluation information) are calculated based on the annealed complete videos. 3D consistency and perceptual loss are used as rewards (i.e., reward information). Each video requires the calculation of rewards such as 3D consistency and perceptual loss, and this embodiment requires no additional penalty.

[0116] After completing the trajectory collection and reward operations, the GRPO policy update needs to be performed. Specifically, the log-prob (i.e., random noise distribution information) obtained from annealing sampling and the reward are input into the GRPO group relative advantage estimation to obtain the relative advantage information of each video. Subsequently, the loss can be calculated based on the relative advantage information, and the diffusion model can be adjusted based on the loss.

[0117] As can be seen from the above embodiments, the embodiments of this disclosure can evaluate the reward of each video in multiple dimensions through various reward functions, and use the reward information to determine the relative advantage information of each video. This relative advantage can effectively enhance the training weight of high-quality samples, thereby achieving stable and efficient updating of the policy network without introducing a complex penalty mechanism, reducing training complexity and improving convergence.

[0118] In step S142, the model parameters of the video generation model are adjusted using the model adjustment information to obtain the adjusted video generation model.

[0119] Here, model parameters can be understood as the parameters that need to be adjusted in the model adjustment video. For example, the model parameters can be model weights, activation functions in the model, etc.

[0120] In some embodiments, adjusting the model parameters of the video generation model using the model adjustment information to obtain an adjusted video generation model includes: adjusting the model weights of the diffusion model using the relative advantage information of the multiple model adjustment videos to obtain an adjusted diffusion model.

[0121] Following the example above, in the process of performing GRPO policy update in this embodiment, after obtaining the relative advantage information of each video, the policy network πθ (i.e., the diffusion model) will be updated based on the relative advantage information. That is, the model parameters in the diffusion model will be adjusted to obtain the adjusted diffusion model. In this embodiment, the annealing parameters themselves do not participate in backpropagation, thereby ensuring gradient stability.

[0122] Based on the above embodiments, it can be seen that the embodiments of this disclosure can stably and efficiently update the policy network through relative advantage information, reducing training complexity and improving convergence. It should be noted that in some embodiments, the stopping conditions for model adjustment operations on the video generation model include: the number of model adjustments reaching a preset threshold, or the loss of the video generation model reaching convergence, etc.

[0123] In some embodiments, the process of performing the operation of "based on the initial video noise and noise intensity adjustment sequence, performing multiple rounds of noise prediction and denoising processing through the video generation model to obtain the denoised video noise in the final round" can be implemented based on an annealing scheduler, a sampler, and a video generation model. The annealing scheduler is used to generate a noise intensity adjustment sequence and adjust the noise information predicted by the video generation model using the noise intensity adjustment sequence; the sampler is used to perform denoising processing. It should be noted that the annealing scheduler and sampler can be network layers or sub-models in the video generation model; or, the annealing scheduler and sampler can be models or software modules independent of the video generation model. For example, the annealing scheduler can be a lightweight PyTorch module, which can be a piece of code logic less than 30 lines. With system encapsulation of the annealing scheduler, it can be plug-and-play, meaning that the annealing scheduler can be embedded in any DDIM / DDPM pipeline based on the diffuses library without retraining.

[0124] In some embodiments, the noise intensity adjustment sequence in the model adjustment method of this disclosure can be a time step sequence; the noise intensity adjustment parameter can be a time step, and the noise information to be adjusted can be a noise weight. That is to say, this disclosure provides a scheme for noise weight adjustment based on time steps. Specifically, during GRPO training, the noise weight (i.e., the noise information to be adjusted) is dynamically adjusted according to the time step. For example, the noise weight is gradually reduced in the later stages of multi-round noise prediction and denoising processing.

[0125] Based on the above, the model adjustment method provided in this disclosure can be a model training method, a model fine-tuning method for a trained video generation model, or a reinforcement learning method for a trained video generation model.

[0126] As can be seen from the embodiments provided in this disclosure above, the embodiments of this disclosure can, during the process of multi-round noise prediction and denoising in the video generation model, finely adjust the intensity of noise information in multi-round noise prediction and denoising through a noise intensity adjustment sequence. This allows the video generation model to perform multi-round noise prediction and denoising on the initial video noise more precisely using the adjusted noise information. Then, the video noise after the last round of denoising is decoded to obtain a higher-quality model-adjusted video. By adjusting the model parameters of the video generation model through the higher-quality model-adjusted video, a better-performing video generation model can be obtained, thereby improving the quality of the video generated by the video generation model.

[0127] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0128] Taking the application of the model adjustment method provided in this disclosure in generating 3D static scene video scenes as an example, the model adjustment method will be described. Figure 2 This is a schematic diagram illustrating the application of a model adjustment method provided in an embodiment of this disclosure; based on Figure 2 As can be seen, the model adjustment method in this embodiment of the present disclosure specifically includes the following steps.

[0129] Step S21: Using a diffusion model, perform multiple rounds of noise prediction and denoising on the video noise to obtain a 3D static scene video.

[0130] The diffusion model is a video latent diffusion model, which can perform multi-round noise prediction and denoising on random noise through a Markov decision process based on ODE sampling.

[0131] The video noise is an initial random noise used to generate the video.

[0132] The specific execution method of step S21 includes the following.

[0133] 1. Based on the total number of steps T, pre-generate a monotonically decreasing sequence σ_schedule, and use the Gaussian distribution algorithm to determine the corresponding random item for each σ_schedule[t]; the specific content is as follows.

[0134] This disclosure provides a strategy for generating annealing schedules, which can pre-generate a monotonically decreasing sequence σ_schedule∈[1,0] based on the total number of steps T (total time steps); the monotonically decreasing sequence supports linear, cosine or piecewise functions, and can be changed on the fly during the training / inference phase.

[0135] The method for determining the monotonically decreasing sequence includes: First, determining the time step of any round in the multi-round noise prediction, and using the linear formula or cosine formula to calculate the enhanced noise information (σ_schedule[t]) corresponding to the target round; Second, using the Gaussian distribution algorithm to generate a random vector that follows a Gaussian distribution for any round, and using the random vector of any round as the random term ε; Finally, constructing the monotonically decreasing sequence σ_schedule based on the time step t, σ_schedule[t] and random term ε corresponding to the multi-round noise prediction.

[0136] The formula for calculating linear noise can be found in the following formula.

[0137] σ_schedule[t]=σ_start+(σ_end-σ_start)(t / T).

[0138] For an explanation of the above formula, please refer to the explanation of formula 2 in the above embodiments, which will not be repeated here.

[0139] The formula for calculating cosine noise can be found in the following formula.

[0140] σ_schedule[t]=0.5(σ_start+σ_end)+0.5(σ_start+σ_end)cos(tπ / T).

[0141] For an explanation of the above formula, please refer to the explanation of formula 3 in the above embodiments, which will not be repeated here.

[0142] 2. During each round of noise prediction and denoising processing for video noise, find σ_schedule[t] and the random term corresponding to the time step t of the current round from the monotonically decreasing sequence.

[0143] 3. Adjust the noise information predicted by the diffusion model using σ_schedule[t] and the random term, and use the adjusted noise information to denoise the video noise of the current round.

[0144] First, during the process of multi-round noise prediction and denoising, the diffusion model can input the current round's time step t (e.g., 100) and the video noise x_t (i.e., the noise state corresponding to time step t) into the diffusion model.

[0145] Secondly, the diffusion model is used to predict noise based on the time step t of the current round and the video noise x_t, so as to obtain the noise information to be adjusted in the current round.

[0146] Finally, from the pre-generated monotonically decreasing sequence σ_schedule∈[1,0], determine the σ_schedule[t] (i.e., enhanced noise information) and the random term ε (i.e., random noise information) corresponding to the current round (round t). In the sampling loop of DDIM / DDPM, σ_schedule[t] is directly multiplied by the random term, and the resulting product is added to the noise information to be adjusted in the current round to obtain the adjusted noise information. Then, the adjusted noise information is used to denoise the video noise x_t, and the denoised video noise is used as the video noise of the (t-1)th round (i.e., the noise state corresponding to time step t-1).

[0147] For specific implementation details, please refer to the formula below.

[0148] x_t-1=μθ(x_t)+σ_schedule[t]×ε (ε~N(0,1)).

[0149] For an explanation of the above formula, please refer to the explanation of Formula 1 in the above embodiments, which will not be repeated here.

[0150] Based on the above, it can be seen that the steps of each round of noise reduction processing in the embodiments of this disclosure can be simplified to the formula: dx_t=fθ(x)dt+σdw.

[0151] Where dx_t refers to the video noise in round t; fθ(x)dt refers to the noise information to be adjusted obtained by using the diffusion model to predict the video noise in round t; σdw refers to the above σ_schedule[t] and ε.

[0152] 4. Repeat the above steps until the last round is completed to obtain the denoised video noise (i.e., clean noise). Then, use the decoder to decode this clean noise of the preset size to generate a 3D still video.

[0153] It should be noted that the embodiments of this disclosure can generate multiple 3D static videos (i.e., complete videos after annealing), and each 3D static video is generated through the above-described diffusion model.

[0154] Step S22: Using the reward function, calculate the rewards such as 3D consistency and perceptual loss based on the complete annealed video, and then input the log-prob obtained from annealing sampling and the rewards into the GRPO group.

[0155] Step S23: The GRPO group estimates the relative advantage based on log-prob and reward, calculates the relative advantage information for each video, generates a loss based on the relative advantage information, and uses the loss to update the policy network (i.e., the diffusion model).

[0156] Among them, rewards refer to Figure 2 In this context, R1, R2, R3, ..., Rg; relative advantage information refers to... Figure 2 In the context of A1, A2, A3, ..., Ag; loss refers to... Figure 2 GRPO loss in GRPO; GRPO refers to Figure 2 Group relative policy optimization in [the context of policy optimization].

[0157] Based on the above steps, it can be seen that the model adjustment method in this embodiment is a video generation sampling annealing method for GRPO training. This method involves the intersection of reinforcement learning and diffusion models, and specifically provides a method and system for optimizing video generation quality through sampling stage annealing within the GRPO (Group Relative Policy Optimization) training framework.

[0158] The model adjustment method in this disclosure proposes a 3D static scene video generation method and system based on Group Relative Policy Optimization (GRPO), 3D Gaussian sputtering guidance, and a 3D reward function. This method uses 3D GS (Gaussian) rendered video as a high-quality reference, focuses on evaluating the 3D consistency of the generated video through a carefully designed 3D reward function, and fine-tunes the video generation diffusion model using GRPO, thereby generating 3D static scene videos with high geometric consistency and supporting temporal control of dynamic viewpoints.

[0159] The model adjustment method in this embodiment is a model training fine-tuning method. During the reinforcement learning sampling process, in addition to using the video generation model for denoising, extra noise (such as a noise intensity adjustment sequence) is added for model exploration, thus introducing randomness into the sampling process. This embodiment reduces randomness in the later stages of denoising by using early truncation (t_start (a certain time step) to t_end (a certain time step)) and an annealing algorithm, thereby avoiding image quality degradation caused by randomness.

[0160] This disclosure provides solutions to the following key problems encountered by GRPO in video diffusion model training: 1. Post-training noise degradation: In GRPO training, the diffusion sampler uses a fixed σ throughout, resulting in random noise being injected even when frames are nearly clean, obscuring high-frequency details and geometric consistency. 2. Training instability: Constant noise causes the policy gradient to carry redundant random components, reducing the reliability of GRPO's group comparison. 3. Limited scope of modification: Existing annealing only affects training hyperparameters (learning rate, reward weights), and cannot directly suppress noise intensity during the sampling phase, requiring model retraining or the introduction of additional loss.

[0161] To address the aforementioned problems, this disclosure provides corresponding solutions, including the following:

[0162] 1. Avoid post-processing noise from damaging image quality. Specifically, during GRPO training, the noise intensity during the sampling phase is dynamically adjusted to prevent post-processing noise from damaging already generated details, thereby stabilizing image quality. Specifically, as the image approaches a clean frame, the noise intensity gradually decreases to zero, ensuring that generated high-frequency details and geometric structures are preserved, avoiding detail blurring or geometric distortion caused by random noise.

[0163] 2. Stabilizing the training process specifically includes: improved gradient stability – due to reduced sampling noise in later stages, policy gradient estimation is more stable, avoiding gradient explosion or vanishing problems caused by random noise, thus improving the stability of GRPO training; and improved training efficiency – by reducing unnecessary noise interference, the training process is smoother, reducing the number of training steps and increasing training efficiency.

[0164] 3. Wide applicability: Specifically, it enhances task versatility, as the method of this disclosure can be extended to GRPO-based video generation tasks with any reward function. It also improves model versatility, supporting various video diffusion models (such as DDIM and DDPM) and reinforcement learning frameworks, exhibiting broad applicability and compatibility.

[0165] Based on the above, the differences between the model adjustment method in this embodiment and related technologies include: the model adjustment method in this embodiment proposes a method to optimize video generation quality by dynamically adjusting the noise intensity during the sampling stage within the GRPO (Group Relative Policy Optimization) training framework; including dynamic noise annealing during the sampling stage, system encapsulation and plug-and-play, the design and application of the annealing schedule table, and the noise intensity adjustment mechanism during the sampling stage.

[0166] Among them, dynamic noise annealing during the sampling phase refers to dynamically adjusting the noise intensity only during the sampling phase of GRPO training. By using a predefined annealing schedule (such as linear, cosine, or piecewise functions), the noise intensity is gradually reduced, thereby improving the quality of video generation.

[0167] Among them, system encapsulation and plug-and-play refer to encapsulating the annealing scheduler and sampler as independent modules, which are easy to integrate into the existing GRPO training framework, support multiple scheduling strategies, and have wide applicability and flexibility.

[0168] The design and application of the annealing schedule table refers to providing a method for generating the annealing schedule table (such as linear, cosine, or piecewise functions), which can be applied to the random terms of the diffusion sampler to achieve dynamic noise adjustment.

[0169] The noise intensity adjustment mechanism in the sampling phase refers to the specific implementation method for dynamically adjusting the noise intensity during the sampling phase, including how to multiply the annealing coefficient into the random term at each time step, and how to switch to deterministic sampling in the later stage.

[0170] As can be seen from the above, the model adjustment method in this embodiment provides an efficient, flexible and practical method to optimize the video generation quality in GRPO training while maintaining the stability and efficiency of the training process.

[0171] It should be noted that in related technologies, the scheme of fine-tuning the diffusion model by combining GRPO reinforcement learning usually uses a fixed noise intensity (σ) in the diffusion sampling phase and does not adjust the noise intensity in the sampling phase during training. Compared with this technical solution, the beneficial effects achieved by the model adjustment method in this disclosure embodiment include the following.

[0172] 1. Avoiding the destruction of details by post-processing noise: Specifically, related techniques typically use a fixed noise intensity (σ) during the diffusion sampling stage. This means that random noise is still injected even in the later stages, nearing clean frames. This fixed noise intensity can destroy the generated high-frequency details and geometry, thus reducing the quality of the generated video. The model adjustment method in this embodiment introduces a dynamic noise annealing mechanism to gradually reduce the noise intensity during the sampling stage, ultimately switching to deterministic sampling in the later stages. This method avoids the destruction of generated details by post-processing noise, significantly improving the 3D consistency and detail fidelity of the generated video.

[0173] 2. Enhanced training stability: Specifically, related techniques, due to the fixed noise intensity during the sampling phase, can cause policy gradient estimation to be affected by unnecessary random components, leading to instability in the training process and a tendency for gradient explosion or vanishing. The model adjustment method in this embodiment dynamically adjusts the noise intensity during the sampling phase, reducing the interference of later noise on the policy gradient and making the policy gradient estimation more stable. This not only improves training stability but also reduces the number of training steps, thus increasing training efficiency.

[0174] 3. Improved generation quality: Specifically, related technologies often result in poor high-frequency details and geometric consistency in generated videos due to post-production noise interference, leading to a decline in overall quality. The model adjustment method in this embodiment, through a dynamic noise annealing mechanism, produces videos with better high-frequency details and geometric consistency, significantly improving the overall quality of the generated video.

[0175] In summary, the model adjustment method in this embodiment can reduce SFT data dependence, enhance the video model generation effect by using reinforcement learning, and improve the 3D consistency of the video generation model by introducing 3D reconstruction rewards, thereby improving the scene video generation effect.

[0176] Figure 3 A flowchart illustrating a video generation method provided in an embodiment of this disclosure. (Refer to...) Figure 3 The method includes:

[0177] Step S31: Use the adjusted video generation model to perform multiple rounds of noise prediction and denoising on the video noise to be denoised, and obtain the video noise after the last round of denoising. Decode the video noise after the last round of denoising to obtain the video corresponding to the video noise to be denoised. The adjusted video generation model is determined according to the above model adjustment method.

[0178] The adjusted video generation model processes the video noise to be denoised in the same way as the initial video noise. For an explanation of the video noise to be denoised, please refer to the relevant content on the initial video noise to be denoised in the model adjustment method described above. For an explanation of the video, please refer to the relevant content on the model-adjusted video in the model adjustment method described above. For example, the video could be a video of a water glass rotating clockwise; or, for another example, a video showing the exterior of a house.

[0179] Based on the embodiments provided in this disclosure, it can be seen that the embodiments of this disclosure can utilize a better-performing, adjusted video generation model to perform multiple rounds of noise prediction and denoising on the video noise to be denoised, thereby obtaining a higher-quality video. The adjustment method of the adjusted video generation model includes: during the process of the video generation model performing multiple rounds of noise prediction and denoising, the intensity of the noise information in the multiple rounds of noise prediction and denoising is adjusted more finely through a noise intensity adjustment sequence, so that the video generation model can perform multiple rounds of noise prediction and denoising on the initial video noise more finely through the adjusted noise information, and then decode the video noise after the last round of denoising to obtain a higher-quality model-adjusted video; by adjusting the model parameters of the video generation model through the higher-quality model-adjusted video, a better-performing video generation model can be obtained.

[0180] Figure 4 This is a block diagram of a model adjustment device provided in an embodiment of the present disclosure.

[0181] Reference Figure 4 This disclosure provides a model adjustment device, which includes:

[0182] The noise determination module 402 is used to determine the initial video noise to be denoised through the video generation model.

[0183] The noise processing module 404 is used to perform multiple rounds of noise prediction and denoising processing through the video generation model based on the initial video noise and the noise intensity adjustment sequence to obtain the video noise after the last round of denoising. The noise intensity adjustment sequence is generated based on random parameters and prediction progress parameters corresponding to the multiple rounds of noise prediction.

[0184] The decoding module 406 is used to decode the video noise after the last round of denoising to obtain the model-adjusted video.

[0185] The model adjustment module 408 is used to adjust the model parameters of the video generation model according to the model to obtain the adjusted video generation model.

[0186] In some embodiments, the noise prediction module 404 is configured to: predict the video noise of the current round using the video generation model to obtain the noise information to be adjusted for the current round; wherein the video noise of the first round is the initial video noise; adjust the noise information to be adjusted for the current round according to the noise intensity adjustment parameters corresponding to the current round in the noise intensity adjustment sequence to obtain the noise information of the current round; perform local denoising on the video noise of the current round according to the noise information of the current round to obtain the video noise of the next round; wherein, if the current round is the last round, the video noise of the next round is the denoised video noise of the last round.

[0187] In some embodiments, the random parameters include a random vector, the noise intensity adjustment parameters in the noise intensity adjustment sequence include random noise information and enhanced noise information, and the prediction progress parameter includes a time step; the device further includes a sequence generation module, configured to: determine the time step of the target round in the multi-round noise prediction, wherein the target round is any round in the multi-round noise prediction; calculate the enhanced noise information corresponding to the target round using the enhanced noise calculation formula, based on the intensity control parameters and the time step corresponding to the target round; generate a random vector using a Gaussian distribution algorithm, and determine the random noise information corresponding to the target round based on the random vector; and determine the noise intensity adjustment sequence based on the time step, enhanced noise information, and random noise information corresponding to the multi-round noise prediction.

[0188] In some embodiments, the enhanced noise calculation formula is a linear noise calculation formula, used to determine linearly decreasing enhanced noise information. The intensity control parameters include the start time step and the end time step to be adjusted, and the total time step corresponding to the multi-round noise prediction. The sequence generation module is used to: use the linear noise calculation formula to determine the enhanced noise information corresponding to the target round based on the start time step, the end time step, the total time step, and the time step corresponding to the target round.

[0189] In some embodiments, the enhanced noise calculation formula is a cosine noise calculation formula, used to determine the enhanced noise information with decreasing cosine. The intensity control parameters include the start time step and the end time step to be adjusted, and the total time step corresponding to the multi-round noise prediction. The sequence generation module is used to: calculate the cosine parameter based on the total time step and the time step corresponding to the target round using the cosine noise calculation formula, and determine the enhanced noise information corresponding to the target round based on the cosine parameter, the start time step, and the end time step.

[0190] In some embodiments, the model adjustment module 408 is configured to: evaluate the model-adjusted video using multiple evaluation algorithms, determine multiple video evaluation information, and determine model adjustment information based on the multiple video evaluation information; adjust the model parameters of the video generation model using the model adjustment information to obtain the adjusted video generation model.

[0191] In some embodiments, there are multiple model adjustment videos, the model adjustment information includes relative advantage information, and the video generation model includes a diffusion model; the model adjustment module 408 is configured to: evaluate the target model adjustment video using the multiple evaluation algorithms to obtain multiple video evaluation information corresponding to the target model adjustment video, wherein the target model adjustment video is any one of the multiple model adjustment videos; determine the reward information of the target model adjustment video based on the multiple video evaluation information corresponding to the target model adjustment video; determine the relative advantage information of the target model adjustment video using the random noise distribution information corresponding to the multi-round noise prediction and the reward information of the target model adjustment video, wherein the random noise distribution information is used to represent the distribution state of random noise information in the multi-round noise prediction, and the random noise information is determined according to the Gaussian distribution algorithm; the model adjustment module 408 is configured to: adjust the model weights of the diffusion model using the relative advantage information of the multiple model adjustment videos to obtain an adjusted diffusion model.

[0192] The model adjustment device in the embodiments provided in this disclosure can finely adjust the intensity of noise information in multiple rounds of noise prediction and denoising processing through a noise intensity adjustment sequence during the process of multi-round noise prediction and denoising processing of the video generation model. This allows the video generation model to perform more precise multi-round noise prediction and denoising processing on the initial video noise using the adjusted noise information. Then, the video noise after the last round of denoising is decoded to obtain a higher quality model-adjusted video. By adjusting the model parameters of the video generation model through the higher quality model-adjusted video, a better-performing video generation model can be obtained, thereby improving the quality of the video generated by the video generation model.

[0193] Figure 5 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0194] Reference Figure 5This disclosure provides an electronic device, which includes: at least one processor 501; at least one memory 502; and one or more I / O interfaces 503 connected between the processor 501 and the memory 502; wherein the memory 502 stores one or more computer programs that can be executed by the at least one processor 501, and the one or more computer programs are executed by the at least one processor 501 to enable the at least one processor 501 to perform the above-described model adjustment method or video generation method.

[0195] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the model adjustment method or video generation method described above. The computer-readable storage medium may be volatile or non-volatile.

[0196] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described model adjustment method or video generation method.

[0197] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0198] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0199] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0200] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0201] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0202] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0203] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0204] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0205] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0206] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A model adjustment method characterized by comprising: The method comprises: determining initial video noise to be denoised by a video generation model; performing multiple rounds of noise prediction and denoising processing on the initial video noise by the video generation model according to a noise intensity adjustment sequence, to obtain video noise after the last round of denoising, wherein the noise intensity adjustment sequence is generated based on a random parameter and a prediction progress parameter corresponding to the multiple rounds of noise prediction, and a noise intensity adjustment parameter in the noise intensity adjustment sequence comprises enhanced noise information calculated by using an enhanced noise calculation formula, the enhanced noise calculation formula comprises a linear noise calculation formula or a cosine noise calculation formula, the linear noise calculation formula is used to determine linearly decreasing enhanced noise information, and the cosine noise calculation formula is used to determine cosine-decreasing enhanced noise information; decoding the video noise after the last round of denoising to obtain a model adjustment video; adjusting model parameters of the video generation model according to the model adjustment video, to obtain an adjusted video generation model.

2. The method of claim 1, wherein, Each round of processing in the multiple rounds of noise prediction and denoising processing by the video generation model comprises: performing noise prediction on video noise of a current round by the video generation model, to obtain to-be-adjusted noise information of the current round; wherein the video noise of the first round is the initial video noise; adjusting the to-be-adjusted noise information of the current round according to a noise intensity adjustment parameter corresponding to the current round in the noise intensity adjustment sequence, to obtain noise information of the current round; performing local denoising on the video noise of the current round according to the noise information of the current round, to obtain video noise of a next round; wherein, in a case where the current round is the last round, the video noise of the next round is the video noise after the last round of denoising.

3. The model adjustment method according to claim 1 or 2, characterized by, The random parameter comprises a random vector, the noise intensity adjustment parameter in the noise intensity adjustment sequence comprises random noise information and enhanced noise information, and the prediction progress parameter comprises a time step; The method further comprises: determining a time step of a target round in the multiple rounds of noise prediction, wherein the target round is any round in the multiple rounds of noise prediction; calculating enhanced noise information corresponding to the target round by using an enhanced noise calculation formula according to an intensity control parameter and a time step corresponding to the target round; generating a random vector by using a Gaussian distribution algorithm, and determining random noise information corresponding to the target round according to the random vector; determining the noise intensity adjustment sequence according to the time step corresponding to the multiple rounds of noise prediction, the enhanced noise information, and the random noise information.

4. The model adjustment method according to claim 3, characterized by, The enhanced noise calculation formula is a linear noise calculation formula, and the intensity control parameter comprises a to-be-adjusted starting time step and a to-be-adjusted ending time step and a total time step corresponding to the multiple rounds of noise prediction; The calculation of the enhanced noise information corresponding to the target round by using the enhanced noise calculation formula according to the intensity control parameter and the time step corresponding to the target round comprises: The linear noise calculation formula is used to determine the enhanced noise information corresponding to the target wheel according to the start time step, the end time step, the total time step, and the time step corresponding to the target wheel.

5. The model adjustment method according to claim 3, characterized by, The enhanced noise calculation formula is a cosine noise calculation formula, and the intensity control parameter includes a start time step and an end time step to be adjusted, and a total time step corresponding to the multi-wheel noise prediction; The enhanced noise information corresponding to the target wheel is calculated by using the enhanced noise calculation formula according to the intensity control parameter and the time step corresponding to the target wheel, including: The cosine noise calculation formula is used to calculate a cosine parameter according to the total time step and the time step corresponding to the target wheel, and to determine the enhanced noise information corresponding to the target wheel based on the cosine parameter, the start time step, and the end time step.

6. The model adjustment method according to claim 1 or 2, characterized by, The model adjustment video is adjusted according to the model, the model parameter of the video generation model is adjusted, and an adjusted video generation model is obtained. The model adjustment video is evaluated by using a plurality of evaluation algorithms, a plurality of video evaluation information is determined, and model adjustment information is determined according to the plurality of video evaluation information; The model parameter of the video generation model is adjusted by using the model adjustment information, and an adjusted video generation model is obtained.

7. The model adjustment method according to claim 6, characterized by, The model adjustment video is a plurality of model adjustment videos, the model adjustment information includes relative advantage information, and the video generation model includes a diffusion model; The model adjustment video is evaluated by using a plurality of evaluation algorithms, a plurality of video evaluation information is determined, and model adjustment information is determined according to the plurality of video evaluation information, including: The target model adjustment video is evaluated by using the plurality of evaluation algorithms, and a plurality of video evaluation information corresponding to the target model adjustment video is obtained, wherein the target model adjustment video is any one of a plurality of model adjustment videos; The reward information of the target model adjustment video is calculated according to the plurality of video evaluation information corresponding to the target model adjustment video; The relative advantage information of the target model adjustment video is determined by using the random noise distribution information of the multi-wheel noise prediction and the reward information of the target model adjustment video, wherein the random noise distribution information is used to represent the distribution state of the random noise information in the multi-wheel noise prediction, and the random noise information is determined according to a Gaussian distribution algorithm; The model parameter of the video generation model is adjusted by using the model adjustment information, and an adjusted video generation model is obtained, including: The model weight of the diffusion model is adjusted by using the relative advantage information of the plurality of model adjustment videos, and an adjusted diffusion model is obtained.

8. A method of video generation, the method comprising: Including: The last round of denoised video noise is obtained by using the adjusted video generation model to perform multi-wheel noise prediction and denoising processing on the video noise to be denoised, and the video corresponding to the video noise to be denoised is obtained by decoding the last round of denoised video noise, wherein the adjusted video generation model is determined according to the model adjustment method of any one of the above claims 1-7.

9. A model adjustment device characterized by comprising: Including: The noise determination module is configured to determine initial video noise to be denoised by the video generation model; The noise processing module is configured to perform multi-round noise prediction and denoising processing on the initial video noise and a noise intensity adjustment sequence by the video generation model to obtain video noise after the last round of denoising, wherein the noise intensity adjustment sequence is generated based on a random parameter and a prediction progress parameter corresponding to the multi-round noise prediction, and a noise intensity adjustment parameter in the noise intensity adjustment sequence includes enhanced noise information calculated by using an enhanced noise calculation formula, the enhanced noise calculation formula includes a linear noise calculation formula or a cosine noise calculation formula, the linear noise calculation formula is used to determine linearly decreasing enhanced noise information, and the cosine noise calculation formula is used to determine cosine-decreasing enhanced noise information. The decoding module is configured to decode the video noise after the last round of denoising to obtain a model adjustment video. The model adjustment module is configured to adjust model parameters of the video generation model according to the model adjustment video to obtain an adjusted video generation model.

10. An electronic device, comprising: comprise: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the method of any one of claims 1-8.

12. A computer program product, characterised in that, comprise computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in the processor of an electronic device, the processor in the electronic device performs the method of any one of claims 1-8.

Images