Single-step video generation diffusion model construction method and device, computer equipment and computer readable storage medium

Abstract

The invention relates to a single-step video generation diffusion model construction method. The method comprises the following steps: acquiring a teacher model, a first student model and a second student model; video description text features and real noise are acquired, and the video description text features and the real noise are used as input to generate an initial generation video by using the first student model; performing multi-step iterative noise addition on the initially generated video to obtain a noise-added generated video; taking the video description text features and the noise-added video as input, and predicting first video noise by using a teacher model; taking the video description text features and the noise-added generated video as input, and predicting second video noise by using a second student model; performing alternative iterative optimization on the first student model and the second student model according to the first video noise, the second video noise and the real noise; and determining the iteratively optimized first student model as a single-step video generation diffusion model. By adopting the method, both the video generation effect and the video generation efficiency can be considered.

Description

Technical Field

[0001] This application relates to the field of video generation technology, and in particular to a method, apparatus, computer device, and computer-readable storage medium for constructing a single-step video generation diffusion model. Background Technology

[0002] With the development of video generation technology, video generation methods based on diffusion models have gradually emerged.

[0003] However, this diffusion-based video generation method can generate video data based on a stepwise denoising process, and the generated video data is smoother and more natural, that is, the video generation effect is better. However, it requires multiple (large) model inference processes to generate video data, resulting in low video generation efficiency. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer device, and computer-readable storage medium for constructing a single-step video generation diffusion model that can balance video generation effect and video generation efficiency, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for constructing a single-step video generation diffusion model, including:

[0006] A multi-step video generation diffusion model is obtained as the teacher model, and the first student model and the second student model corresponding to the teacher model are determined.

[0007] The video description text features and real noise are obtained, and the video description text features and real noise are used as inputs to perform single-step video generation using the first student model to obtain the initial generated video.

[0008] The initially generated video is subjected to multi-step iterative noise addition to obtain a noisy generated video;

[0009] Using the video description text features and the noise-generated video as input, the teacher model is used to predict video noise to obtain the first video noise.

[0010] Using the video description text features and the noise-generated video as input, the second student model is used to predict video noise to obtain the second video noise.

[0011] Based on the differences between the first video noise, the second video noise, and the real noise, the first student model and the second student model are alternately and iteratively optimized.

[0012] The first student model, after iterative optimization, is determined as a single-step video generation diffusion model.

[0013] In one embodiment, the step of iteratively optimizing the first student model and the second student model based on the differences between the first video noise, the second video noise, and the real noise includes:

[0014] Based on the first video noise and the second video noise, a first model loss is generated corresponding to the first student model, wherein the first model loss is used to minimize the difference between the first video noise and the second video noise;

[0015] Based on the second video noise and the real noise, a second model loss corresponding to the second student model is generated, wherein the first model loss is used to minimize the difference between the second video noise and the real noise;

[0016] Based on the first model loss and the second model loss, the first student model and the second student model are iteratively optimized alternately.

[0017] In one embodiment, the step of iteratively optimizing the first student model and the second student model based on the first model loss and the second model loss includes:

[0018] If the first model loss and the second model loss converge, then it is determined that the first student model and the second student model have completed the iterative optimization.

[0019] If the first model loss and the second model loss do not converge, then while keeping the model parameters of the teacher model and the second student model unchanged, backpropagation is performed to update the LoRA branch in the first student model based on the first model loss; or while keeping the model parameters of the teacher model and the first student model unchanged, backpropagation is performed to update the LoRA branch in the second student model based on the second model loss.

[0020] Return to the steps of obtaining video description text features and real noise, until the generated first model loss and second model loss converge.

[0021] In one embodiment, determining the first student model and the second student model corresponding to the teacher model includes:

[0022] By adding a preset LoRA branch to the teacher model, a first student model and a second student model are generated.

[0023] In one embodiment, the step of using the video description text features and the real noise as input, and performing single-step video generation using the first student model to obtain an initial generated video, includes:

[0024] Using the video description text features and the real noise as input, the first student model is used to predict video noise to obtain the third video noise; based on the third video noise and the real noise, an initial generated video is generated.

[0025] In one embodiment, the step of using the video description text features and the noisy generated video as input, and using the teacher model to predict video noise to obtain the first video noise, includes:

[0026] Obtain the number of iterations in the multi-step iterative noise addition process; based on the video description text features, the number of iterations, and the noise-generated video, predict the first video noise by performing multi-step iterative backdiffusion processing using the teacher model.

[0027] In one embodiment, the step of using the video description text features and the noisy generated video as input, and using the second student model to predict video noise to obtain second video noise, includes:

[0028] Obtain the number of iterations in the multi-step iterative noise addition process; based on the video description text features, the number of iterations, and the noise-generated video, predict the second video noise by performing multi-step iterative backdiffusion processing using the second student model.

[0029] Secondly, this application also provides a single-step video generation diffusion model construction apparatus, comprising:

[0030] The first determining module is used to obtain a multi-step video generation diffusion model as a teacher model, and determine the first student model and the second student model corresponding to the teacher model.

[0031] The single-step video generation module is used to acquire video description text features and real noise, and use the video description text features and real noise as input to perform single-step video generation using the first student model to obtain an initial generated video.

[0032] The noise-adding module is used to perform multi-step iterative noise addition on the initially generated video to obtain a noise-added generated video.

[0033] The first noise prediction module is used to predict video noise by taking the video description text features and the noise-generated video as inputs and using the teacher model to obtain the first video noise.

[0034] The second noise prediction module is used to predict video noise by taking the video description text features and the noisy generated video as inputs and using the second student model to obtain the second video noise.

[0035] The alternating iterative optimization module is used to perform alternating iterative optimization of the first student model and the second student model based on the differences between the first video noise, the second video noise, and the real noise.

[0036] The second determining module is used to determine the first student model, after iterative optimization, as a single-step video generation diffusion model.

[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0038] A multi-step video generation diffusion model is obtained as the teacher model, and a first student model and a second student model corresponding to the teacher model are determined. Video description text features and real noise are obtained, and using the video description text features and the real noise as input, the first student model is used for single-step video generation to obtain an initial generated video. The initial generated video is then subjected to multi-step iterative noise addition to obtain a noisy generated video. Using the video description text features and the noisy generated video as input, the teacher model is used to predict video noise to obtain first video noise. The second student model is used to predict video noise to obtain second video noise. Based on the differences between the first video noise, the second video noise, and the real noise, the first student model and the second student model are iteratively optimized alternately. The first student model after iterative optimization is determined as the single-step video generation diffusion model.

[0039] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0040] A multi-step video generation diffusion model is obtained as the teacher model, and a first student model and a second student model corresponding to the teacher model are determined. Video description text features and real noise are obtained, and using the video description text features and the real noise as input, the first student model is used for single-step video generation to obtain an initial generated video. The initial generated video is then subjected to multi-step iterative noise addition to obtain a noisy generated video. Using the video description text features and the noisy generated video as input, the teacher model is used to predict video noise to obtain first video noise. The second student model is used to predict video noise to obtain second video noise. Based on the differences between the first video noise, the second video noise, and the real noise, the first student model and the second student model are iteratively optimized alternately. The first student model after iterative optimization is determined as the single-step video generation diffusion model.

[0041] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0042] A multi-step video generation diffusion model is obtained as the teacher model, and a first student model and a second student model corresponding to the teacher model are determined. Video description text features and real noise are obtained, and using the video description text features and the real noise as input, the first student model is used for single-step video generation to obtain an initial generated video. The initial generated video is then subjected to multi-step iterative noise addition to obtain a noisy generated video. Using the video description text features and the noisy generated video as input, the teacher model is used to predict video noise to obtain first video noise. The second student model is used to predict video noise to obtain second video noise. Based on the differences between the first video noise, the second video noise, and the real noise, the first student model and the second student model are iteratively optimized alternately. The first student model after iterative optimization is determined as the single-step video generation diffusion model.

[0043] The aforementioned method, apparatus, computer equipment, computer-readable storage medium, and computer program product for constructing a single-step video generation diffusion model can first obtain a multi-step video generation diffusion model as the teacher, and determine the first and second student models corresponding to the teacher model; obtain video description text features and real noise, so that the video description text features and real noise can be used as inputs to the first student model for single-step video generation to obtain an initial generated video; perform multi-step iterative noise addition on the initial generated video to obtain a noisy generated video; thus, using the video description text features and the noisy generated video as inputs, the teacher model can be used to predict video noise to obtain the first video noise, and the second student model can be used to predict video noise to obtain the second video noise; thus, in this application... Based on the differences between the first video noise, the second video noise, and the real noise, the first student model and the second student model can be iteratively optimized alternately. During the iterative optimization process, the first student model can indirectly learn the prior knowledge of multi-step video generation in the teacher model through the second student model via model distillation. In this way, the first student model after iterative optimization is determined as a single-step video generation diffusion model. This single-step video generation diffusion model can generate video data in a single step, with higher video generation efficiency. Moreover, the video data generated in a single step can achieve the same or similar video generation effect as the video data generated by multi-step iterative generation based on the diffusion model. Therefore, the single-step video generation diffusion model finally obtained in this application can balance video generation effect and video generation efficiency. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating a single-step video generation diffusion model construction method in one embodiment of this application;

[0046] Figure 2 This is a schematic diagram illustrating the process of iteratively optimizing the first student model and the second student model in one embodiment of this application;

[0047] Figure 3 This is a schematic diagram of the model architecture consisting of a teacher model, a first student model, and a second student model in one embodiment of this application;

[0048] Figure 4 This is a structural block diagram of a single-step video generation diffusion model construction device in one embodiment of this application;

[0049] Figure 5 This is an internal structural diagram of a computer device in one embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0052] In one exemplary embodiment, such as Figure 1 As shown, a method for constructing a single-step video generation diffusion model is provided, including the following steps 202 to 214. Wherein:

[0053] Step 202: Obtain the multi-step video generation diffusion model as the teacher model, and determine the first student model and the second student model corresponding to the teacher model.

[0054] Among them, the multi-step video generation diffusion model can be a diffusion-based video generation model. This multi-step video generation diffusion model can perform multi-step iterative forward diffusion or backward diffusion processing based on the input data to obtain generated video data. The generated video data can be a video image sequence, and the input data can be video description text features and video noise. The video description text features can be extracted from the video description text.

[0055] As an example, the video description text features mentioned above can be embedding vectors extracted from the video description text.

[0056] As an example, the first student model can be a diffusion-based single-step video generation model, while the second student model and the teacher model can be diffusion-based multi-step video generation models. It should be noted that the diffusion process in the diffusion-based single-step video generation model is a single-step diffusion process, while the final diffusion process in the diffusion-based multi-step video generation model is a multi-step iterative diffusion process.

[0057] As an example, determining the first and second student models corresponding to the teacher model includes:

[0058] By adding a preset LoRA branch to the teacher model, the first student model and the second student model are generated.

[0059] In this embodiment, a preset LoRA branch can be added to the teacher model to obtain the first student model and the second student model. The difference is that the diffusion process in the first student model has 1 iteration, while the diffusion process in the second student model has t iterations, where t is greater than 1.

[0060] It should be noted that the default LoRA branch is a low-rank matrix branch. The low-rank matrix branch is a pluggable module that achieves efficient parameter training through low-rank matrix decomposition in the fine-tuning of large models. This lays the foundation for subsequent fine-tuning of the first student model and the second student model.

[0061] As an example, the teacher model described above can be obtained through iterative training using video description text and real videos as training data. For example, the training data could be... ,in, Describe the text for the video. This is a real video.

[0062] Step 204: Obtain video description text features and real noise, and use the video description text features and real noise as input to perform single-step video generation using the first student model to obtain the initial generated video.

[0063] As an example, step 204 includes: acquiring video description text features and real noise; adjusting and extracting the real noise to obtain real noise features, and fusing the video description text features and real noise features to obtain a first fused feature; inputting the first fused feature into a first student model, and performing single-step video generation through the first student model to obtain an initial generated video. The initial generated video can be a sequence of video images.

[0064] As an example, using video description text features and real noise as input, a single-step video generation is performed using the first student model to obtain an initial generated video, including:

[0065] Using video description text features and real noise as input, the first student model is used to predict video noise to obtain the third video noise; based on the third video noise and the real noise, an initial generated video is generated.

[0066] In this embodiment, the real noise can be adjusted and extracted to obtain real noise features, and the video description text features and real noise features can be fused to obtain the first fused feature. Then, using the first fused feature as input, the video noise prediction based on single-iteration diffusion using the first student model is used to obtain the third video noise. Then, the difference between the real noise and the third video noise is calculated to obtain the initial generated video.

[0067] Step 206: Perform multi-step iterative noise addition on the initially generated video to obtain a noisy generated video.

[0068] In this embodiment, a Markov chain can be used to iteratively add noise to the initially generated video in multiple steps to obtain a noisy generated video.

[0069] As an example, the process of adding noise to the initially generated video in multiple iterative steps described above is as follows:

[0070]

[0071] in, For the process The noise-generated video is obtained by iterative noise addition. For the process The noisy video obtained by iterative noisening is generated in the first step of the noisy addition process. For the initial video generation, For the added noise, where, During iterative noise addition, the noise added in each iteration can be sampled from a Gaussian distribution. and For the first Hyperparameters during iterative noise addition.

[0072] Step 208: Using the video description text features and the noise-generated video as input, the teacher model is used to predict video noise and obtain the first video noise.

[0073] As an example, step 208 includes: extracting features from the noisy video to obtain noisy video features; then using the noisy video features and video description text features as input, using the teacher model to perform noise prediction based on multi-step iterative diffusion to obtain the first video noise.

[0074] As an example, taking video description text features and a noisy generated video as input, a teacher model is used to predict video noise, resulting in the first video noise, including:

[0075] Obtain the number of iterations in the multi-step iterative noise addition process; using video description text features, the number of iterations, and the noise-generated video as input, predict the first video noise by performing multi-step iterative backdiffusion processing using a teacher model.

[0076] Among them, the above-mentioned multi-step iterative noise addition process is the forward diffusion process, and the back diffusion process is the reverse process of the forward diffusion process. The forward diffusion process is the process of iteratively adding noise to the video data, while the back diffusion process is the process of iteratively removing noise from the video data.

[0077] Specifically, the number of iterations in the multi-step iterative noise addition process can be obtained first; features can be extracted from the noise-generated video to obtain noise-added video features, and the video description text features and noise-added video features can be fused to obtain a second fused feature; using the second fused feature as input, the teacher model can be used to perform multi-step iterative backdiffusion processing based on the above iteration number to predict the first video noise.

[0078] Step 210: Using the video description text features and the noise-generated video as input, the second student model is used to predict video noise and obtain the second video noise.

[0079] As an example, step 210 includes: extracting features from the noisy video to obtain noisy video features; then using the noisy video features and video description text features as input, performing noise prediction based on multi-step iterative diffusion using a second student model to obtain second video noise.

[0080] As an example, taking video description text features and a noisy generated video as input, a second student model is used to predict video noise, resulting in second video noise, including:

[0081] Obtain the number of iterations in the multi-step iterative noise addition process; generate a video based on the video description text features, the number of iterations, and the noise addition; and predict the second video noise by using a second student model for multi-step iterative backdiffusion processing.

[0082] Among them, the above-mentioned multi-step iterative noise addition process is the forward diffusion process, and the back diffusion process is the reverse process of the forward diffusion process. The forward diffusion process is the process of iteratively adding noise to the video data, while the back diffusion process is the process of iteratively removing noise from the video data.

[0083] Specifically, the number of iterations in the multi-step iterative noise addition process can be obtained first; features can be extracted from the noise-generated video to obtain noise-added video features, and the video description text features and noise-added video features can be fused to obtain a second fused feature; using the second fused feature as input, the second student model can be used to perform multi-step iterative backdiffusion processing based on the above number of iterations to predict the second video noise.

[0084] Step 212: Based on the differences between the first video noise, the second video noise, and the real noise, the first student model and the second student model are iteratively optimized alternately.

[0085] As an example, step 212 includes: generating a first model loss based on the difference between the first video noise and the second video noise; generating a second model loss based on the difference between the second video noise and the real noise; and iteratively optimizing the first student model and the second student model alternately based on the first model loss and the second model loss, while freezing the model parameters of the teacher model, so that the first student model learns the prior knowledge of the teacher model.

[0086] Step 214: The first student model, after iterative optimization, is determined as the single-step video generation diffusion model.

[0087] The conditions for completing iterative optimization can include model loss convergence or reaching the maximum number of iterations.

[0088] After obtaining the single-step video generation diffusion model, the descriptive text of the video to be processed and the actual noise can be acquired. Then, using the descriptive text and actual noise as input data to the model, the single-step video generation diffusion model is used to predict video noise based on a single iteration of diffusion, obtaining the predicted video noise. The difference between the actual noise and the predicted video noise can then be calculated to obtain the generated video. This video generation process eliminates the need for multi-step iterative diffusion calculations, resulting in higher efficiency. Furthermore, since the single-step video generation diffusion model has learned prior knowledge of the multi-step diffusion process from the teacher model, video data generated based on single-step diffusion can be as smooth and natural as video data generated based on multi-step diffusion, ensuring the quality of single-step video generation.

[0089] In the above-mentioned method for constructing a single-step video generation diffusion model, a multi-step video generation diffusion model can be obtained first as the teacher, and the first and second student models corresponding to the teacher model can be determined. Video description text features and real noise can be obtained, so that the video description text features and real noise can be used as input to the first student model for single-step video generation, resulting in an initial generated video. The initial generated video is then subjected to multi-step iterative noise addition to obtain a noisy generated video. Therefore, using the video description text features and the noisy generated video as input, the teacher model can be used to predict video noise, resulting in the first video noise, and the second student model can be used to predict video noise, resulting in the second video noise. Thus, in this application, the first video noise and the second video noise can be predicted based on the second student model. The differences between frequency noise, real noise, and other noise factors are used to iteratively optimize the first and second student models. This allows the first student model to indirectly learn the prior knowledge of multi-step video generation from the teacher model through model distillation during the iterative optimization process. The optimized first student model is then identified as a single-step video generation diffusion model. This model can generate video data in a single step, resulting in higher video generation efficiency. Furthermore, the single-step generated video data can achieve the same or similar video generation effect as the multi-step iteratively generated video data based on the diffusion model. Therefore, the single-step video generation diffusion model obtained in this embodiment can balance video generation effect and video generation efficiency.

[0090] In one exemplary embodiment, such as Figure 2 As shown, based on the differences between the first video noise, the second video noise, and the real noise, the first student model and the second student model are iteratively optimized alternately, including:

[0091] Step 302: Generate a first model loss corresponding to the first student model based on the first video noise and the second video noise, wherein the first model loss is used to minimize the difference between the first video noise and the second video noise.

[0092] As an example, step 302 includes: calculating a first noise difference between the first video noise and the second video noise, and constructing a first model loss corresponding to the first student model based on the first noise difference, wherein the first model loss is used to minimize the difference between the first video noise and the second video noise.

[0093] As an example, the formula for the loss of the first model is as follows:

[0094]

[0095] in, The loss of the first model. The first type of video noise, This is the second type of video noise.

[0096] Step 304: Based on the second video noise and the real noise, generate the second model loss corresponding to the second student model, wherein the first model loss is used to minimize the difference between the second video noise and the real noise.

[0097] As an example, step 304 includes: calculating a second noise difference between the second video noise and the real noise, and constructing a second model loss corresponding to the second student model based on the second noise difference, wherein the second model loss is used to minimize the difference between the second video noise and the real noise.

[0098] As an example, the formula for the loss of the second model is as follows:

[0099]

[0100] in, For the second model loss, This is the second type of video noise. This is real noise.

[0101] Step 306: Based on the first model loss and the second model loss, perform alternating iterative optimization on the first student model and the second student model.

[0102] As an example, step 306 includes: alternating the following steps in each iteration round during the model iterative optimization process:

[0103] Based on the loss of the first model, the first student model is updated by backpropagation;

[0104] Based on the loss of the second model, the second student model is updated through backpropagation.

[0105] As an example, the first student model and the second student model are iteratively optimized alternately based on the first model loss and the second model loss, including:

[0106] If the first model loss and the second model loss converge, it is determined that the first student model and the second student model have completed iterative optimization. If the first model loss and the second model loss do not converge, while keeping the model parameters of the teacher model and the second student model unchanged, backpropagation is performed on the LoRA branch in the first student model based on the first model loss. Alternatively, while keeping the model parameters of the teacher model and the first student model unchanged, backpropagation is performed on the LoRA branch in the second student model based on the second model loss. The process then returns to the steps of obtaining video description text features and real noise until the generated first model loss and second model loss converge.

[0107] Specifically, it checks whether both the first model loss and the second model loss have converged. If both converge, it determines that the first student model and the second student model have completed their iterative optimization. The second model loss is used to enable the second student model to learn the prior knowledge of the teacher model, while the first model loss is used to enable the first student model to learn the prior knowledge of the second student model. If neither converges, the following two steps are executed alternately in each iteration:

[0108] While keeping the model parameters of the teacher model and the second student model unchanged, the LoRA branch in the first student model is updated by backpropagation based on the first model loss, so as to fine-tune the first student model; while keeping the model parameters of the teacher model and the first student model unchanged, the LoRA branch in the second student model is updated by backpropagation based on the second model loss, so as to fine-tune the second student model.

[0109] Return to the steps of obtaining video description text features and real noise, until the generated first model loss and second model loss converge.

[0110] In this embodiment, since both the first and second student models have LoRA branches, the second model loss is used to encourage the second student model to learn the prior knowledge of the multi-step diffusion process of the teacher model into its LoRA branch. The first model loss is used to encourage the first student model to learn the parameters of the second model's LoRA branch into its own LoRA branch. In this way, the first student model can also indirectly learn the prior knowledge of the teacher model's multi-step diffusion process through the second student model. The video data generated based on the iteratively optimized first student model can achieve the same or similar video generation effect as the video data generated based on the teacher model. Therefore, the video data generated step by step based on the first student model will be smoother and more natural.

[0111] It should also be noted that since LoRA branches are added to both the first and second student models, when backpropagating to update the first and second student models, it is not necessary to update the backbone network parameters of the first and second student models. Only the parameters of the LoRA branches need to be updated. This allows for fine-tuning updates of the first and second student models, resulting in higher training efficiency and lower computational resources required during training.

[0112] In some embodiments, parameters Figure 3 , Figure 3 This embodiment illustrates the model architecture comprised of the teacher model, the first student model, and the second student model, based on... Figure 3The model architecture shown can be iteratively optimized to form a single-step video generation diffusion model.

[0113] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0114] Based on the same inventive concept, this application also provides a single-step video generation diffusion model construction apparatus for implementing the single-step video generation diffusion model construction method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method. Therefore, the specific limitations in one or more embodiments of the single-step video generation diffusion model construction apparatus provided below can be found in the limitations of the single-step video generation diffusion model construction method described above, and will not be repeated here.

[0115] In one exemplary embodiment, such as Figure 4 As shown, a single-step video generation diffusion model construction device is provided, comprising: a first determining module 402, a single-step video generation module 404, a noise adding module 406, a first noise prediction module 408, a second noise prediction module 410, an alternating iterative optimization module 412, and a second determining module 414, wherein:

[0116] The first determining module 402 is used to obtain a multi-step video generation diffusion model as a teacher model, and determine the first student model and the second student model corresponding to the teacher model.

[0117] The single-step video generation module 404 is used to acquire video description text features and real noise, and use the video description text features and real noise as input to perform single-step video generation using the first student model to obtain an initial generated video.

[0118] The noise-adding module 406 is used to perform multi-step iterative noise addition on the initially generated video to obtain a noise-added generated video.

[0119] The first noise prediction module 408 is used to predict video noise using the video description text features and the noise-generated video as input, and to obtain the first video noise.

[0120] The second noise prediction module 410 is used to predict video noise using the video description text features and the noise-generated video as input, and to obtain the second video noise by using the second student model.

[0121] The alternating iterative optimization module 412 is used to perform alternating iterative optimization on the first student model and the second student model based on the differences between the first video noise, the second video noise and the real noise.

[0122] The second determining module 414 is used to determine the first student model after iterative optimization as a single-step video generation diffusion model.

[0123] In one embodiment, the alternating iterative optimization module is further configured to:

[0124] Based on the first video noise and the second video noise, a first model loss is generated corresponding to the first student model, wherein the first model loss is used to minimize the difference between the first video noise and the second video noise; based on the second video noise and the real noise, a second model loss is generated corresponding to the second student model, wherein the first model loss is used to minimize the difference between the second video noise and the real noise; based on the first model loss and the second model loss, the first student model and the second student model are iteratively optimized alternately.

[0125] In one embodiment, the alternating iterative optimization module is further configured to:

[0126] If the first model loss and the second model loss converge, it is determined that the first student model and the second student model have completed iterative optimization. If the first model loss and the second model loss do not converge, while keeping the model parameters of the teacher model and the second student model unchanged, backpropagation is performed on the LoRA branch in the first student model based on the first model loss; or while keeping the model parameters of the teacher model and the first student model unchanged, backpropagation is performed on the LoRA branch in the second student model based on the second model loss. The process returns to the step of obtaining video description text features and real noise until the generated first model loss and second model loss converge.

[0127] In one embodiment, the first determining module is further configured to:

[0128] By adding a preset LoRA branch to the teacher model, a first student model and a second student model are generated.

[0129] In one embodiment, the single-step video generation module is further configured to:

[0130] Using the video description text features and the real noise as input, the first student model is used to predict video noise to obtain the third video noise; based on the third video noise and the real noise, an initial generated video is generated.

[0131] In one embodiment, the first noise prediction module is further configured to:

[0132] Obtain the number of iterations in the multi-step iterative noise addition process; based on the video description text features, the number of iterations, and the noise-generated video, predict the first video noise by performing multi-step iterative backdiffusion processing using the teacher model.

[0133] In one embodiment, the second noise prediction module is further configured to:

[0134] Obtain the number of iterations in the multi-step iterative noise addition process; based on the video description text features, the number of iterations, and the noise-generated video, predict the second video noise by performing multi-step iterative backdiffusion processing using the second student model.

[0135] Each module in the aforementioned single-step video generation diffusion model construction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0136] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores video data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When executed by the processor, the computer program implements a single-step video generation diffusion model construction method.

[0137] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0138] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0139] A multi-step video generation diffusion model is obtained as the teacher model, and a first student model and a second student model corresponding to the teacher model are determined. Video description text features and real noise are obtained, and using the video description text features and the real noise as input, the first student model is used for single-step video generation to obtain an initial generated video. The initial generated video is then subjected to multi-step iterative noise addition to obtain a noisy generated video. Using the video description text features and the noisy generated video as input, the teacher model is used to predict video noise to obtain first video noise. The second student model is used to predict video noise to obtain second video noise. Based on the differences between the first video noise, the second video noise, and the real noise, the first student model and the second student model are iteratively optimized alternately. The first student model after iterative optimization is determined as the single-step video generation diffusion model.

[0140] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0141] Based on the first video noise and the second video noise, a first model loss is generated corresponding to the first student model, wherein the first model loss is used to minimize the difference between the first video noise and the second video noise; based on the second video noise and the real noise, a second model loss is generated corresponding to the second student model, wherein the first model loss is used to minimize the difference between the second video noise and the real noise; based on the first model loss and the second model loss, the first student model and the second student model are iteratively optimized alternately.

[0142] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0143] If the first model loss and the second model loss converge, it is determined that the first student model and the second student model have completed iterative optimization. If the first model loss and the second model loss do not converge, while keeping the model parameters of the teacher model and the second student model unchanged, backpropagation is performed on the LoRA branch in the first student model based on the first model loss; or while keeping the model parameters of the teacher model and the first student model unchanged, backpropagation is performed on the LoRA branch in the second student model based on the second model loss. The process returns to the step of obtaining video description text features and real noise until the generated first model loss and second model loss converge.

[0144] In one embodiment, the teacher model includes a first video generation module and a first diffusion module; the processor, when executing the computer program, further implements the following steps:

[0145] By adding a preset LoRA branch to the teacher model, a first student model and a second student model are generated.

[0146] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0147] Using the video description text features and the real noise as input, the first student model is used to predict video noise to obtain the third video noise; based on the third video noise and the real noise, an initial generated video is generated.

[0148] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0149] Obtain the number of iterations in the multi-step iterative noise addition process; based on the video description text features, the number of iterations, and the noise-generated video, predict the first video noise by performing multi-step iterative backdiffusion processing using the teacher model.

[0150] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0151] Obtain the number of iterations in the multi-step iterative noise addition process; based on the video description text features, the number of iterations, and the noise-generated video, predict the second video noise by performing multi-step iterative backdiffusion processing using the second student model.

[0152] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0153] A multi-step video generation diffusion model is obtained as the teacher model, and a first student model and a second student model corresponding to the teacher model are determined. Video description text features and real noise are obtained, and using the video description text features and the real noise as input, the first student model is used for single-step video generation to obtain an initial generated video. The initial generated video is then subjected to multi-step iterative noise addition to obtain a noisy generated video. Using the video description text features and the noisy generated video as input, the teacher model is used to predict video noise to obtain first video noise. The second student model is used to predict video noise to obtain second video noise. Based on the differences between the first video noise, the second video noise, and the real noise, the first student model and the second student model are iteratively optimized alternately. The first student model after iterative optimization is determined as the single-step video generation diffusion model.

[0154] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0155] Based on the first video noise and the second video noise, a first model loss is generated corresponding to the first student model, wherein the first model loss is used to minimize the difference between the first video noise and the second video noise; based on the second video noise and the real noise, a second model loss is generated corresponding to the second student model, wherein the first model loss is used to minimize the difference between the second video noise and the real noise; based on the first model loss and the second model loss, the first student model and the second student model are iteratively optimized alternately.

[0156] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0157] If the first model loss and the second model loss converge, it is determined that the first student model and the second student model have completed iterative optimization. If the first model loss and the second model loss do not converge, while keeping the model parameters of the teacher model and the second student model unchanged, backpropagation is performed on the LoRA branch in the first student model based on the first model loss; or while keeping the model parameters of the teacher model and the first student model unchanged, backpropagation is performed on the LoRA branch in the second student model based on the second model loss. The process returns to the step of obtaining video description text features and real noise until the generated first model loss and second model loss converge.

[0158] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0159] By adding a preset LoRA branch to the teacher model, a first student model and a second student model are generated.

[0160] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0161] Using the video description text features and the real noise as input, the first student model is used to predict video noise to obtain the third video noise; based on the third video noise and the real noise, an initial generated video is generated.

[0162] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0163] Obtain the number of iterations in the multi-step iterative noise addition process; based on the video description text features, the number of iterations, and the noise-generated video, predict the first video noise by performing multi-step iterative backdiffusion processing using the teacher model.

[0164] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0165] Obtain the number of iterations in the multi-step iterative noise addition process; based on the video description text features, the number of iterations, and the noise-generated video, predict the second video noise by performing multi-step iterative backdiffusion processing using the second student model.

[0166] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0167] A multi-step video generation diffusion model is obtained as the teacher model, and a first student model and a second student model corresponding to the teacher model are determined. Video description text features and real noise are obtained, and using the video description text features and the real noise as input, the first student model is used for single-step video generation to obtain an initial generated video. The initial generated video is then subjected to multi-step iterative noise addition to obtain a noisy generated video. Using the video description text features and the noisy generated video as input, the teacher model is used to predict video noise to obtain first video noise. The second student model is used to predict video noise to obtain second video noise. Based on the differences between the first video noise, the second video noise, and the real noise, the first student model and the second student model are iteratively optimized alternately. The first student model after iterative optimization is determined as the single-step video generation diffusion model.

[0168] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0169] Based on the first video noise and the second video noise, a first model loss is generated corresponding to the first student model, wherein the first model loss is used to minimize the difference between the first video noise and the second video noise; based on the second video noise and the real noise, a second model loss is generated corresponding to the second student model, wherein the first model loss is used to minimize the difference between the second video noise and the real noise; based on the first model loss and the second model loss, the first student model and the second student model are iteratively optimized alternately.

[0170] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0171] If the first model loss and the second model loss converge, it is determined that the first student model and the second student model have completed iterative optimization. If the first model loss and the second model loss do not converge, while keeping the model parameters of the teacher model and the second student model unchanged, backpropagation is performed on the LoRA branch in the first student model based on the first model loss; or while keeping the model parameters of the teacher model and the first student model unchanged, backpropagation is performed on the LoRA branch in the second student model based on the second model loss. The process returns to the step of obtaining video description text features and real noise until the generated first model loss and second model loss converge.

[0172] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0173] By adding a preset LoRA branch to the teacher model, a first student model and a second student model are generated.

[0174] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0175] Using the video description text features and the real noise as input, the first student model is used to predict video noise to obtain the third video noise; based on the third video noise and the real noise, an initial generated video is generated.

[0176] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0177] Obtain the number of iterations in the multi-step iterative noise addition process; based on the video description text features, the number of iterations, and the noise-generated video, predict the first video noise by performing multi-step iterative backdiffusion processing using the teacher model.

[0178] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0179] Obtain the number of iterations in the multi-step iterative noise addition process; based on the video description text features, the number of iterations, and the noise-generated video, predict the second video noise by performing multi-step iterative backdiffusion processing using the second student model.

[0180] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0181] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0182] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for constructing a single-step video generation diffusion model, characterized in that, The method includes: A multi-step video generation diffusion model is obtained as the teacher model, and the first student model and the second student model corresponding to the teacher model are determined. The video description text features and real noise are obtained, and the video description text features and real noise are used as inputs to perform single-step video generation using the first student model to obtain the initial generated video. The initially generated video is subjected to multi-step iterative noise addition to obtain a noisy generated video; Using the video description text features and the noise-generated video as input, the teacher model is used to predict video noise to obtain the first video noise. Using the video description text features and the noise-generated video as input, the second student model is used to predict video noise to obtain the second video noise. Based on the differences between the first video noise, the second video noise, and the real noise, the first student model and the second student model are alternately and iteratively optimized. The first student model, after iterative optimization, is determined as a single-step video generation diffusion model.

2. The method according to claim 1, characterized in that, The step of iteratively optimizing the first student model and the second student model based on the differences between the first video noise, the second video noise, and the real noise includes: Based on the first video noise and the second video noise, a first model loss is generated corresponding to the first student model, wherein the first model loss is used to minimize the difference between the first video noise and the second video noise; Based on the second video noise and the real noise, a second model loss corresponding to the second student model is generated, wherein the first model loss is used to minimize the difference between the second video noise and the real noise; Based on the first model loss and the second model loss, the first student model and the second student model are iteratively optimized alternately.

3. The method according to claim 2, characterized in that, The step of iteratively optimizing the first student model and the second student model based on the first model loss and the second model loss includes: If the first model loss and the second model loss converge, then it is determined that the first student model and the second student model have completed the iterative optimization. If the first model loss and the second model loss do not converge, then while keeping the model parameters of the teacher model and the second student model unchanged, backpropagation is performed to update the LoRA branch in the first student model based on the first model loss; or while keeping the model parameters of the teacher model and the first student model unchanged, backpropagation is performed to update the LoRA branch in the second student model based on the second model loss. Return to the steps of obtaining video description text features and real noise, until the generated first model loss and second model loss converge.

4. The method according to claim 1, characterized in that, Determining the first student model and the second student model corresponding to the teacher model includes: By adding a preset LoRA branch to the teacher model, a first student model and a second student model are generated.

5. The method according to claim 1, characterized in that, The step of generating an initial video by using the video description text features and the real noise as input and performing a single-step video generation using the first student model includes: Using the video description text features and the real noise as input, the first student model is used to predict the video noise to obtain the third video noise; An initial generated video is generated based on the third video noise and the real noise.

6. The method according to claim 1, characterized in that, The step of using the video description text features and the noisy generated video as input, and using the teacher model to predict video noise to obtain the first video noise includes: Obtain the number of iterations in the multi-step iterative noise addition process; Based on the video description text features, the number of iterations, and the noise-generated video, the first video noise is predicted by performing multi-step iterative backdiffusion processing using the teacher model.

7. The method according to claim 1, characterized in that, The step of using the video description text features and the noisy generated video as input, and using the second student model to predict video noise to obtain the second video noise includes: Obtain the number of iterations in the multi-step iterative noise addition process; Based on the video description text features, the number of iterations, and the noise-generated video, the second video noise is predicted by performing multi-step iterative backdiffusion processing using the second student model.

8. A device for constructing a single-step video generation diffusion model, characterized in that, The device includes: The first determining module is used to obtain a multi-step video generation diffusion model as a teacher model, and determine the first student model and the second student model corresponding to the teacher model. The single-step video generation module is used to acquire video description text features and real noise, and use the video description text features and real noise as input to perform single-step video generation using the first student model to obtain an initial generated video. The noise-adding module is used to perform multi-step iterative noise addition on the initially generated video to obtain a noise-added generated video. The first noise prediction module is used to predict video noise by taking the video description text features and the noise-generated video as inputs and using the teacher model to obtain the first video noise. The second noise prediction module is used to predict video noise by taking the video description text features and the noisy generated video as inputs and using the second student model to obtain the second video noise. The alternating iterative optimization module is used to perform alternating iterative optimization of the first student model and the second student model based on the differences between the first video noise, the second video noise, and the real noise. The second determining module is used to determine the first student model, after iterative optimization, as a single-step video generation diffusion model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images