Video generation method and device, equipment, medium and program product

Abstract

The invention discloses a video generation method and device, equipment, a medium and a program product, and relates to the field of data processing, in particular to the technical field of artificial intelligence. The specific implementation scheme is as follows: acquiring video generation information; wherein the video generation information comprises at least one of an image, a video and text description; inputting the video generation information into a video generation model; wherein the video generation model adopts an autoregressive diffusion architecture of a streaming sliding window; dividing a video generation process into a plurality of blocks through the video generation model; and for each block, determining a historical reference frame from the generated video frames, and generating a target video based on the historical reference frame and the anchor point frame.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data processing, in particular to the technical field of artificial intelligence, and specifically to a video generation method, device, equipment, medium and program product. BACKGROUND

[0002] In the related art, the AI video generation technology generates a video with a length usually within 5-10 seconds. With the continuous development and wide application of short video technology, higher requirements are put forward for the continuous duration of the video. In the related art, the video duration is usually extended by using the first and last frame continuation method. SUMMARY

[0003] The present disclosure provides a video generation method, device, equipment, medium and program product.

[0004] According to an aspect of the present disclosure, a video generation method is provided, comprising: obtaining video generation information; wherein the video generation information comprises at least one of an image, a video, and a text description; inputting the video generation information into a video generation model; wherein the video generation model adopts a streaming sliding window autoregressive diffusion architecture; dividing, by the video generation model, a video generation process into multiple blocks; for each of the blocks, determining a historical reference frame from the generated video frames, and generating a target video based on the historical reference frame and an anchor frame.

[0005] According to another aspect of the present disclosure, a video generation device is provided, comprising: an obtaining module configured to obtain video generation information; wherein the video generation information comprises at least one of an image, a video, and a text description; an inputting module configured to input the video generation information into a video generation model; wherein the video generation model adopts a streaming sliding window autoregressive diffusion architecture; a dividing module configured to divide, by the video generation model, a video generation process into multiple blocks; a generating module configured to, for each of the blocks, determine a historical reference frame from the generated video frames, and generate a target video based on the historical reference frame and an anchor frame.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the video generation method according to any one of the aspects.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the video generation method according to any one of the aspects.

[0008] According to another aspect of the present disclosure, a computer program product is provided, comprising computer programs / instructions, which, when executed by a processor, implement the steps in the video generation method according to any one of the aspects.

[0009] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them: Figure 1 is a flowchart of a video generation method according to a first embodiment of the present disclosure; Figure 2 is a structural diagram of a DiT module in a streaming sliding window autoregressive diffusion architecture according to a second embodiment of the present disclosure; Figure 3 is a schematic diagram of zero-noise patch introduction according to a third embodiment of the present disclosure; Figure 4 is a structural diagram of a video generation device according to a fourth embodiment of the present disclosure; Figure 5 is a block diagram of an electronic device for implementing the video generation method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0011] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, descriptions of well-known functions and structures are omitted in the following description.

[0012] In the technical scheme of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical scheme comply with the relevant legal regulations and do not violate public order and good customs.

[0013] In the technical solution of the present disclosure, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0014] As can be known from the background, in the related art, the first and last frame continuation writing technology is usually used to realize video extension. Although this can fill the time gap, it is easy to cause the video to lack coherence, the quality and details of the picture are unstable, and it is difficult to meet the complex creation requirements. At the same time, the first and last frame continuation writing requires the user to upload pictures and prompt words for each shot, but since a shot usually has 1-6 seconds, if a video of several tens of seconds is generated, it may require more than 10 groups of pictures and prompt words. This way, the operation threshold is very high, and it is difficult to achieve infinite time generation. Therefore, the related art at least has the following technical problems: memory forgetting problem, the model is difficult to maintain long-term memory of the content of the starting frame when generating a continuous long video, resulting in inconsistent content in the time dimension; cumulative drift problem: the error generated will gradually accumulate with the frame sequence, causing the quality of the generated picture to gradually decrease; as the video length increases, the simple frame continuation writing strategy will exacerbate the cumulative error, causing the generated video quality to decay and the subject consistency to be lost; high computational cost, real-time interaction challenge: limited by the quadratic computational complexity of the Transformer architecture in the video generation model, the computational overhead increases with the generation length, directly training or reasoning for longer videos requires higher requirements for GPU (Graphics Processing Unit, graphics processing unit) memory and computational efficiency, resulting in a sharp increase in cost and difficulty in achieving real-time interaction requirements.

[0015] Therefore, the present disclosure provides a video generation method, which can realize low-cost infinite extrapolation and real-time generation by introducing an autoregressive diffusion model (Autoregressive Diffusion Models) and using a flow sliding window-based autoregressive diffusion architecture.

[0016] The video generation method, device, equipment, medium and program product of the present disclosure will be described below in conjunction with the following embodiments.

[0017] Figure 1 is a flowchart of a video generation method provided by an embodiment of the present disclosure. As shown in Figure 1 , the video generation method can include the following processing: S110, obtaining video generation information.

[0018] The video generation information includes at least one of images, videos, and text descriptions. For example, the image can be a static image input; the video can be a video clip that provides motion information; and the text description can guide the generation of specific video content from the image or video. This multimodal input method can greatly expand the application scope of video generation.

[0019] In embodiments of this disclosure, when generating a video, video generation information can be obtained first. This information may include at least one of an image, a video clip, or a text description. As an example, a user can upload an image and attach a description, requesting the generation of a video where the image and text description are related. For example, an image containing a person can be entered, along with the text description 'Have the person make a heart shape with their hands'.

[0020] S120, input the video generation information into the video generation model.

[0021] The video generation model employs a streaming sliding window autoregressive diffusion architecture. For example, this architecture decomposes the video generation process into multiple consecutive chunks, each processed independently, thus achieving streaming processing of the video generation process.

[0022] In embodiments of this disclosure, after acquiring video generation information, this information can be input into a pre-trained video generation model as initial conditions for generating the target video. For example, the video generation model can be constructed using a streaming sliding window autoregressive diffusion architecture, which can effectively handle long video sequences and avoid the computational burden of processing the entire video at once.

[0023] S130 divides the video generation process into multiple blocks using a video generation model.

[0024] In the embodiments of this disclosure, after the video generation information is input into the video generation model, the model can decompose the video generation process into multiple consecutive blocks, each block containing a certain number of video frames. This allows the video generation model to generate video content block by block, avoiding the computational burden of processing the entire video at once and improving video generation efficiency.

[0025] S140, for each block, determine the historical reference frame from the generated video frames, and generate the target video based on the historical reference frame and the anchor frame.

[0026] In an embodiment of the present disclosure, for each sub-block, the video generation model can determine a historical reference frame from the generated video frames, and generate a final video frame sequence in combination with an anchor frame to obtain a target video. Illustratively, the historical reference frame can provide short-term memory to ensure the coherence of the video content, while the anchor frame can provide long-term memory to ensure the consistency of the video content. As an example, a global attention mechanism can be used to determine the anchor frame, for example, at the beginning of each sub-block, the video generation model can scan the generated video frames through the global attention mechanism to identify a key frame as the anchor frame; wherein the key frame can generally contain important visual information, such as the significant position of the subject or the key action, etc.

[0027] In an embodiment of the present disclosure, by obtaining video generation information; wherein the video generation information includes at least one of an image, a video, and a text description; inputting the video generation information into a video generation model; wherein the video generation model adopts a streaming sliding window autoregressive diffusion architecture; dividing the video generation process into multiple sub-blocks by the video generation model; for each sub-block, determining a historical reference frame from the generated video frames, and generating a target video based on the historical reference frame and an anchor frame. In this way, on the one hand, by adopting the streaming sliding window autoregressive diffusion architecture, the memory requirement can be reduced and the processing efficiency can be improved through sub-block processing, thereby improving the video generation efficiency. On the other hand, by combining the historical reference frame and the anchor frame, the coherence and consistency of the generated target video can be ensured, thereby effectively improving the video quality.

[0028] In some possible implementations, before obtaining the video generation information, the method further includes: constructing a streaming sliding window autoregressive diffusion architecture to decompose a long video sequence into multiple continuous sub-blocks for streaming processing; wherein the streaming sliding window autoregressive diffusion architecture is trained and optimized using a student-teacher model; demonstrating a denoising trajectory for each sub-block by the teacher model to generate a standard video frame; training a student model based on the teacher model, introducing an anchor frame in the training process, and determining a historical reference frame from the generated video frames; training the student model based on the anchor frame and the historical reference frame; determining the trained student model as the video generation model.

[0029] In embodiments of the present disclosure, the training of the video generation model can be performed in advance, and called when video generation processing is needed. For example, a streaming sliding window autoregressive diffusion architecture can be constructed, which can decompose a long video sequence into multiple consecutive blocks for streaming processing. Moreover, a student-teacher model mechanism can also be used for training and optimization of the video generation model. The teacher model can demonstrate a denoising trajectory for each block as a standard denoising trajectory to generate a standard video frame, and the student model is trained based on these denoising trajectories. In the process of training the student model based on the teacher model, anchor frames can be introduced to establish global attention anchors, and video frames can be dynamically selected from the generated video frames as historical reference frames. Then, the student model can be trained by learning the standard denoising trajectory demonstrated by the teacher model, combined with the anchor frames and the historical reference frames. This training mechanism can ensure that the student model can generate high-quality and coherent video content similar to the teacher model. After the student model is trained, the student model can be determined as the video generation model for actual video generation tasks. In this way, on the one hand, the pre-training of the video generation model can further improve the video generation efficiency by directly calling it when used; on the other hand, the training process through the streaming sliding window autoregressive diffusion architecture and the student-teacher model mechanism can effectively improve the accuracy and efficiency of the student model, thereby further improving the quality and efficiency of video generation.

[0030] In some possible implementations, the teacher model demonstrates a denoising trajectory for each block to generate a standard video frame, including: allocating, by the teacher model, an independent noise intensity for each block based on a preset noise scheduler curve; performing denoising processing based on the independent noise intensity allocated for each block to form a denoising trajectory and generate a standard video frame.

[0031] In an embodiment of the present disclosure, the teacher model can assign an independent noise intensity to each patch using a preset noise scheduler curve. The noise scheduler curve can define the noise intensity at each time step in the denoising process, ensuring the step-by-step and controllable nature of the denoising process. The noise scheduler curve can be pre-designed according to specific application requirements, such as linear, cosine, or other forms of curves. Then, the teacher model can gradually remove noise according to the assigned noise intensity, forming a denoising trajectory. The denoising trajectory of each patch is the result of the teacher model performing denoising operations at a specific noise intensity. In this way, the teacher model can generate high-quality standard video frames through denoising operations, which can serve as learning objectives for the student model. In this way, by pre-setting the noise scheduler curve, the teacher model can accurately control the noise intensity at each time step, ensuring the step-by-step and controllable nature of the denoising process, so that the student model can provide high-quality and high-standard denoising trajectories, thereby further improving training efficiency and training quality, and further improving the generation efficiency and quality of the video generation model.

[0032] In some possible implementations, the student model is trained based on the teacher model, including: The standard video frames generated by the teacher model are re-injected into the latent space vector of the student model after reverse noise initialization, and the student model is guided to learn the denoising trajectory based on the latent space vector through the student-teacher mechanism.

[0033] In an embodiment of the present disclosure, when training the student model based on the teacher model, the standard video frames generated by the teacher model can be reverse noise initialized, for example, re-injected with noise to restore a state similar to the original input noise. These re-injected noise video frames can be injected into the latent space vector of the student model, so that the student model can be trained under similar noise conditions as the teacher model. Then, the student model can learn the denoising trajectory demonstrated by the teacher model through the student-teacher mechanism during the training process, and can gradually remove the noise in the latent space vector to finally generate high-quality video frames. In this way, by reverse noise initialization and re-injection, the student model can be trained under similar noise conditions as the teacher model, reducing the cumulative error between training and inference, thereby enhancing the robustness to noise and improving the stability and video quality of the generated video.

[0034] In some possible implementations, the student model is trained based on the teacher model, further including: During the training of the student model, the input historical reference frames in each patch are subjected to probability disturbance enhancement.

[0035] In the embodiments of the present disclosure, the training of the student model based on the teacher model can also perform probability perturbation enhancement on the historical reference frame input in each sub-block. For example, random noise can be applied to the historical reference frame, and the strength and distribution of the noise can be controlled by a preset probability distribution, for example, Gaussian distribution or other suitable distribution can be used to generate random noise. In this way, through the probability perturbation enhancement, the student model can better learn how to perform denoising processing and video generation in the presence of noise during the training process, thereby not only improving the self-correction ability of the video generation model, but also enhancing the robustness of the video generation model to noise.

[0036] In some possible implementations, the historical reference frame is determined from the generated video frames, including: The historical reference frame is obtained by sampling and compressing the generated video frames according to the time proximity and importance.

[0037] In the embodiments of the present disclosure, the historical reference frame can be determined according to the time proximity and importance of the video frames. For example, the time proximity can be the closeness of the video frames in the time sequence. Since the most recently generated frame usually contains more information related to the currently generated frame in the video generation process, a video frame close in time can be selected as the historical reference frame. The frame importance can be the importance of the video frame in the content. For example, the importance can be evaluated by the content features (such as motion information, subject position, etc.) of the video frame. Important frames may, for example, contain key visual information such as significant positions or key actions of the subject. Based on this, a part of the video can be selected from the generated video frames according to the time proximity and importance, and compressed to obtain the historical reference frame. In this way, by selecting video frames with high time proximity and frame importance as historical reference frames, the coherence and consistency of the generated video can be further ensured, and the generated video can be avoided to be inconsistent with the video generation information. On the other hand, by performing sampling and compression processing, the number of historical reference frames can be reduced, the video generation efficiency can be improved, and the consumption of computing resources can be reduced.

[0038] In some possible implementations, the student model is trained based on the anchor frame and the historical reference frame, including: The anchor frame is introduced into each sub-block to establish a global attention anchor, and the historical reference frame is trained as conditional information together with the target frame by using zero noise patch embedding.

[0039] In an embodiment of the present disclosure, when training the student model based on the anchor frame and the historical reference frame, the anchor frame can be introduced in each patch to establish a global attention anchor. During the training process, the historical reference frame can also be embedded as conditional information into the denoising process of the student model, and zero noise can be applied when embedding (i.e., no noise is added). After the historical reference frame is embedded through the zero noise patch, it can be used as conditional information to participate in the training of the student model together with the target frame (i.e., the current video frame to be denoised and generated). In this way, when learning the denoising trajectory, the student model can not only consider the denoising process of the current target frame, but also generate zero noise in combination with the contextual information provided by the historical reference frame, thereby not only improving the temporal coherence of the generated video and making the action transition natural and smooth, but also effectively reducing the cumulative error in the autoregressive generation, further improving the quality and coherence of the generated video.

[0040] In some possible implementations, the trained student model is determined as a video generation model, including: The trained student model is distilled using a DMD step distillation technique to obtain a distilled student model; The distilled student model is used as a video generation model.

[0041] In an embodiment of the present disclosure, the DMD (Deterministic Model Distillation) step distillation technique is a deterministic model distillation method that can accelerate the inference process of the diffusion video generation model. The DMD step distillation technique can compress the multi-step denoising process of the teacher model into fewer steps through knowledge distillation while maintaining the generation quality. For example, the trained student model can be distilled using the DMD step distillation technique to reduce the inference steps, for example, the denoising process originally requiring tens of steps or even hundreds of steps can be compressed to only a few steps. The distilled student model is determined as the final video generation model and is deployed for actual video generation tasks. In this way, through the DMD step distillation technique, the inference steps can be effectively reduced while ensuring the video generation quality, thereby further reducing resource consumption and improving video generation efficiency.

[0042] In some possible implementations, the target video is generated based on the historical reference frame and the anchor frame, including: The historical reference frame and the anchor frame are dynamically selected as contextual information for generating the target video through an attention gating mechanism; The target video is generated according to the contextual information.

[0043] In an embodiment of the present disclosure, when generating the target video based on the historical reference frames and the anchor frames, an attention gating mechanism can be employed to dynamically select the historical reference frames and the anchor frames. For example, a correlation score between the current generated frame and each historical reference frame and anchor frame can be calculated, which is based on multi-dimensional features such as content similarity, temporal proximity, etc. Based on the correlation score, the gating mechanism can dynamically filter out irrelevant or low-correlation historical video frames, and only keep the high-correlation historical video frames as valid context information. Alternatively, the remaining historical reference frames and anchor frames can also be attention-weighted, giving different weights to different frames, so that the model can focus on the information most valuable to the current generated frame. Then, the context information can be input into the denoising process of the video generation model as conditional information to guide the frame-by-frame generation of the target video. In this way, the attention gating mechanism can dynamically select the historical reference frames and the anchor frames, thereby constructing efficient context information, and further achieving high-quality and efficient video generation, and further improving the video generation efficiency and video quality.

[0044] In some possible implementations, the video length of the target video is greater than the video length in the video generation information.

[0045] In an embodiment of the present disclosure, the target video output by the video generation model has a video length greater than the video length in the video generation information input by the user. As an example, the received video generation information can be a short video containing initial frames or first and last frames, which serves as a starting point or constraint condition for generating the target video, and the length thereof is usually short. The video generation model employing the streaming sliding window self-recursive diffusion architecture can organize the video generation process into multiple continuous blocks for streaming processing in sequence, and each block generates a certain length of video content. The length of the target video finally generated by each block can be much greater than the length of the video in the video generation information. The number of blocks can be divided according to actual needs. In this way, the streaming sliding window self-recursive diffusion architecture and related mechanisms can effectively extend the video length while maintaining the quality of the generated video, significantly improving the practicability and user experience of video generation, and better meeting user needs. In some possible implementations, the video generation model employs a window attention mechanism, and the attention range of each block is the historical frames within a preset window.

[0046] In an embodiment of the present disclosure, the video generation model can employ a window attention mechanism. For example, a sliding window attention mechanism is employed, in which, during processing of each patch, attention calculation for the current video frame is no longer performed with all historical frames, but only with historical video frames within a preset window size. As an example, the preset window size is a fixed value (e.g., 16 or 32, etc.), and specific data can be configured according to hardware resources and real-time requirements. As the generated patch advances, the window can slide forward along the time axis, always keeping the latest few historical frames within the window. In this way, the window attention mechanism can limit the attention range to historical frames within the preset window, thereby further ensuring computational efficiency and generation quality, and further improving the real-time performance, efficiency and quality of video generation.

[0047] It can be understood that, after the target video is generated, the following steps can also be included: receiving new video generation information, wherein the new video generation information at least includes a text description; inputting the new video generation information into the video generation model; generating a new target video by the video generation model.

[0048] In an embodiment of the present disclosure, after the target video is output, the user can also generate a new video based on the target video, i.e., a new target video. For example, the user can input new video generation information again, which can be used to describe the adjustment the user wants to make to the target video. Then, the new video generation information can be input into the video generation model. The video generation model can further process the target video based on the new video generation information, generate a new target video and output it. It can be understood that the process of the video generation model generating a new target video based on the new video generation information is similar to the process of generating a target video in the above method embodiment, and will not be described here. Moreover, this generation process can be one, two or more times. In this way, interaction with the user can be achieved, allowing the user to adjust again or multiple times according to the generated video to generate a final satisfactory video, thereby effectively improving the user experience.

[0049] To make the video generation method provided by the embodiments of the present disclosure clearer, the following examples are combined for illustration. The video generation method provided by the embodiments of the present disclosure can include the following contents: 1. Infrastructure renovation: the present disclosure introduces an autoregressive diffusion model, employs an autoregressive diffusion architecture based on streaming sliding windows, to achieve low-cost infinite extrapolation and real-time generation, Figure 2Figure 1 is a structural diagram of a DiT (Diffusion Transformer) module in a flow window-based autoregressive diffusion architecture. Figure 2 In the figure, FFN (Feed-Forward Network) is a feed-forward network that can perform nonlinear transformation on each position to enhance the model expression capability; Self Attention is self-attention that can calculate the correlation between positions in the sequence to enable interaction between video frames / image blocks and capture spatial / temporal dependency; Cross Attention is cross-attention that can calculate the correlation between the sequence and external conditions (such as text, image) to combine the denoising process with video generation information (such as text description, image guidance) and realize video generation.

[0050] In the figure, a ladder independent noise structure can be adopted: based on time step sampling, different intensities of noise are applied to each frame of the video diffusion model (video generation model). According to the noise scheduler curve, each prediction segment is assigned a noise level that meets the noise scheduling strategy in the inference stage to enhance the stability of temporal modeling.

[0051] 2. Training bias elimination: solve the problem of cumulative error and attenuation.

[0052] The traditional sliding window method has a serious error accumulation problem, and small errors of each video frame will gradually accumulate, eventually leading to a serious decline in the quality of the generated video, and color shift, unnatural motion, and subject deformation. These problems are essentially caused by training and inference inconsistency. Training is based on clean video plus denoising, but when inference is performed, the reference sequence contains error-prone generated video results. Therefore, the training process is improved as follows in the embodiments of the present disclosure: Noise re-injection: use the generated result samples of the teacher model for reverse noise initialization, re-add noise to the hidden space vector after denoising of the student model, and use it as the starting noise. Through the student-teacher model, the student model is ensured to denoise on the trajectory correctly constructed by the given noise plan, and the ability of the teacher model is migrated.

[0053] Historical frame disturbance enhancement: probability disturbance of historical frames can be performed to improve the self-correction ability of the model, make the model more robust, and alleviate the cumulative error problem of the autoregressive model.

[0054] 3. Consistency optimization: in order to solve the continuity and consistency problem when generating long videos, a global planning and local reference combined optimization method can be used, including: Introducing anchor frame guidance to ensure global memory: key frames are introduced to establish global attention anchors to ensure the long-term memory capability of the model to improve the global consistency and role retention of the generated video.

[0055] Introduce historical reference frames to ensure continuous generation: see Figure 3 To ensure the continuity of the generated video segment and the previous video content, historical reference frames can be introduced as context information, so that the model has short-term memory capability, making the motion transition natural and smooth.

[0056] "Zero" noise clip introduction: based on the idea of noise as masking, historical reference frames can be directly introduced into the training process together with the generated target frame to improve the continuity of the generation.

[0057] History frame compression: sampling and compression according to time proximity and frame importance to improve the effective control of global historical video frames on current video generation.

[0058] Attention gate mechanism: dynamically select relevant historical frames for reference according to the content of the current frame to avoid irrelevant information interference and improve memory efficiency.

[0059] In this way, based on the streaming sliding window autoregressive diffusion architecture, the high compression ratio generation technology can be broken through, the streaming inference performance of the diffusion model can be greatly improved, the extreme balance of effect and efficiency can be ensured, and real-time interaction can be met. Streaming inference: thanks to the streaming sliding window autoregressive diffusion architecture, the steam engine model can simplify the long window into a sequence composed of smaller chunk blocks to realize streaming inference; Window attention upgrade: the computational complexity can be reduced from quadratic complexity to linear complexity, greatly improving the computational efficiency; Model distillation technology: using dmd step distillation technology can further reduce inference time.

[0060] The specific implementation and technical effects of each step in this embodiment are similar to those of the above method embodiment, and will not be repeated again.

[0061] According to embodiments of the present disclosure, a video generation device is also provided, see Figure 4 The video generation device 400 includes: The acquisition module 410 is configured to acquire video generation information; wherein the video generation information includes at least one of an image, a video, and a text description; The input module 420 is configured to input the video generation information to a video generation model; wherein the video generation model adopts a streaming sliding window autoregressive diffusion architecture; The division module 430 is configured to divide the video generation process into multiple blocks by the video generation model; The generation module 440 is configured to determine a historical reference frame from the generated video frames for each block, and generate a target video based on the historical reference frame and an anchor frame.

[0062] Further, the model training module comprises: The construction unit is configured to construct a streaming sliding window autoregressive diffusion architecture to decompose a long video sequence into multiple continuous subblocks for streaming processing, wherein the streaming sliding window autoregressive diffusion architecture is trained and optimized by using a student-teacher model. The demonstration unit is configured to demonstrate a denoising trajectory for each subblock by using the teacher model to generate a standard video frame. The introduction unit is configured to train a student model based on the teacher model, introduce an anchor frame during the training process, and determine a historical reference frame from the generated video frame. The training unit is configured to train the student model based on the anchor frame and the historical reference frame. The determination unit is configured to determine the trained student model as a video generation model.

[0063] Further, the demonstration unit comprises: The allocation subunit is configured to allocate an independent noise intensity to each subblock based on a preset noise scheduler curve by using the teacher model. The demonstration subunit is configured to perform denoising processing based on the independent noise intensity allocated to each subblock to form the denoising trajectory and generate a standard video frame.

[0064] Further, the introduction unit comprises: The re-injection subunit is configured to re-inject the standard video frame generated by the teacher model after reverse noise initialization into a hidden space vector of the student model, and guide the student model to learn the denoising trajectory based on the hidden space vector by using a student-teacher mechanism.

[0065] Further, the introduction unit further comprises: The perturbation subunit is configured to perform probabilistic perturbation enhancement on the input historical reference frame in each subblock during the training of the student model.

[0066] Further, the introduction unit further comprises: The determination subunit is configured to sample and compress the generated video frame according to time proximity and importance to obtain the historical reference frame.

[0067] Further, the training unit is configured to: Introduce the anchor frame for each subblock, establish a global attention anchor, and use a zero noise slice embedding method to train the historical reference frame as conditional information together with the target frame.

[0068] Further, the determination unit comprises: a distillation subunit configured to distill the trained student model by using a DMD step number distillation technique to obtain a distilled student model; a determination subunit configured to determine the distilled student model as the video generation model.

[0069] Further, the generation module 440 includes: a selection unit configured to dynamically select the historical reference frames and the anchor frames as context information for generating the target video by using an attention gate mechanism; a generation unit configured to generate the target video according to the context information.

[0070] Further, a video duration of the target video is greater than a video duration in the video generation information.

[0071] Further, the video generation model uses a window attention mechanism, and an attention range of each of the subblocks is a historical frame within a preset window.

[0072] The specific implementation and technical effects of each module of the embodiments of the present disclosure are similar to those of the above method embodiments, and will not be described here.

[0073] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0074] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0075] As shown in Figure 5 The electronic device 500 includes a computing unit 501 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 502 or a computer program loaded into a random access memory (RAM) 503 from a storage unit 508. Various programs and data required for the operation of the device 500 can also be stored in the RAM 503. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0076] A plurality of components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0077] The computing unit 501 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs various methods and processes described above, such as the video generation method. For example, in some embodiments, the video generation method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded onto the RAM 503 and executed by the computing unit 501, one or more steps of the video generation method described above can be performed. Alternatively, in other embodiments, the computing unit 501 can be configured to perform the video generation method by any other appropriate means, such as by means of firmware.

[0078] The various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0079] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0080] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0081] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0082] The systems and techniques described herein can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0083] The computer system can include clients and servers. This relationship can be. The servers are generally remote from the users and can be accessed via the Internet using a communication network. The relationship can be facilitated by a computer program running on the computers and having a client-server relationship to one another. The servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are a host product in the cloud computing service system. The servers can also be servers of a distributed system or servers combined with a blockchain.

[0084] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, in series, or in a different order, without departing from the desired results of the technical solutions of the present disclosure, and this is not limited herein.

[0085] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A video generation method, comprising: Obtain video generation information; wherein, the video generation information includes at least one of image, video, and text description; The video generation information is input into the video generation model; wherein the video generation model adopts a streaming sliding window autoregressive diffusion architecture; The video generation model divides the video generation process into multiple blocks. For each block, a historical reference frame is determined from the generated video frames, and a target video is generated based on the historical reference frame and the anchor frame.

2. The method according to claim 1, wherein, Before obtaining the video generation information, the process also includes: An autoregressive diffusion architecture for streaming sliding windows is constructed to decompose long video sequences into multiple consecutive blocks for streaming processing; wherein, the autoregressive diffusion architecture for streaming sliding windows is trained and optimized using a student-teacher model; The teacher model is used to demonstrate the denoised trajectory for each segment, generating standard video frames. The student model is trained based on the teacher model, and anchor frames are introduced during the training process, and historical reference frames are determined from the generated video frames. The student model is trained based on the anchor frame and the historical reference frame; The trained student model is then designated as the video generation model.

3. The method according to claim 2, wherein, The step of demonstrating a denoising trajectory for each of the aforementioned blocks using the teacher model and generating standard video frames includes: Using the teacher model, an independent noise intensity is assigned to each block based on a preset noise scheduler curve; Denoising is performed based on the independent noise intensity assigned to each block to form the denoising trajectory and generate a standard video frame.

4. The method according to claim 3, wherein, The process of training the student model based on the teacher model includes: The standard video frames generated by the teacher model are initialized with inverse noise and then re-injected into the latent space vector of the student model. The student model is then guided to learn the denoised trajectory based on the latent space vector through a student-teacher mechanism.

5. The method according to claim 2, wherein, The process of training the student model based on the teacher model also includes: During the training of the student model, the historical reference frames input in each block are subjected to probability perturbation enhancement.

6. The method according to claim 2, wherein, Determining historical reference frames from generated video frames includes: The generated video frames are sampled and compressed based on temporal proximity and importance to obtain the historical reference frames.

7. The method according to claim 2, wherein, The training of the student model based on the anchor frame and the historical reference frame includes: For each block, the anchor frame is introduced to establish a global attention anchor. The historical reference frame is used as conditional information and trained together with the target frame using a zero-noise patch embedding method.

8. The method according to claim 1, wherein, The step of determining the trained student model as a video generation model includes: The trained student model is distilled using the DMD step distillation technique to obtain the distilled student model. The distilled student model is used as the video generation model.

9. The method according to claim 1, wherein, The generation of the target video based on the historical reference frames and anchor frames includes: The historical reference frame and the anchor frame are dynamically selected through an attention gating mechanism as context information for generating the target video; The target video is generated based on the context information.

10. The method according to claim 1, characterized in that, The duration of the target video is greater than the duration of the video in the video generation information.

11. The method according to claim 1, wherein, The video generation model adopts a window attention mechanism, and the attention range of each block is the historical frames within a preset window.

12. A video generation apparatus, comprising: An acquisition module is used to acquire video generation information; wherein, the video generation information includes at least one of images, videos, and text descriptions; An input module is used to input the video generation information into the video generation model; wherein the video generation model adopts a streaming sliding window autoregressive diffusion architecture; A segmentation module is used to divide the video generation process into multiple blocks using the video generation model; The generation module is used to determine historical reference frames from the generated video frames for each block, and generate target video based on the historical reference frames and anchor frames.

13. The apparatus according to claim 12, wherein, It also includes a model training module, which includes: The construction unit is used to construct the autoregressive diffusion architecture of the streaming sliding window, which decomposes the long video sequence into multiple consecutive blocks for streaming processing; wherein, the autoregressive diffusion architecture of the streaming sliding window is trained and optimized using a student-teacher model; The demonstration unit is used to demonstrate a denoised trajectory for each of the blocks using the teacher model, and generate standard video frames. An introduction unit is used to train a student model based on the teacher model, introduce anchor frames during the training process, and determine historical reference frames from the generated video frames. The training unit is used to train the student model based on the anchor frame and the historical reference frame. A determining unit is used to determine the trained student model as a video generation model.

14. The apparatus according to claim 13, wherein, The demonstration unit includes: The allocation subunit is used to allocate an independent noise intensity to each of the blocks based on a preset noise scheduler curve using the teacher model. The demonstration subunit is used to perform denoising processing based on the independent noise intensity allocated to each of the blocks, form the denoising trajectory, and generate a standard video frame.

15. The apparatus according to claim 14, wherein, The introducing unit includes: The re-injection subunit is used to re-inject the standard video frames generated by the teacher model into the latent space vector of the student model after inverse noise initialization, and guide the student model to learn the denoised trajectory based on the latent space vector through the student-teacher mechanism.

16. The apparatus according to claim 13, wherein, The introducing unit further includes: The perturbation subunit is used to perform probabilistic perturbation enhancement on the historical reference frames input in each block during the training of the student model.

17. The apparatus according to claim 13, wherein, The introducing unit further includes: A subunit is determined to perform sampling and compression processing on the generated video frames based on temporal proximity and importance to obtain the historical reference frames.

18. The apparatus according to claim 2, wherein, The training unit is used for: For each block, the anchor frame is introduced to establish a global attention anchor. The historical reference frame is used as conditional information and trained together with the target frame using a zero-noise patch embedding method.

19. The apparatus according to claim 12, wherein, The determining unit includes: The distillation subunit is used to perform distillation processing on the trained student model using the DMD step distillation technique to obtain the distilled student model. A sub-unit is defined for using the distilled student model as the video generation model.

20. The apparatus according to claim 12, wherein, The generation module includes: The selection unit is used to dynamically select the historical reference frame and the anchor frame through an attention gating mechanism as context information for generating the target video; The generation unit is used to guide the generation of the target video based on the context information.

21. The apparatus according to claim 12, characterized in that, The duration of the target video is greater than the duration of the video in the video generation information.

22. The apparatus according to claim 12, wherein, The video generation model adopts a window attention mechanism, and the attention range of each block is the historical frames within a preset window.

23. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

25. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-11.

Images