Video generation method, video model training method, and video generation apparatus

Abstract

The application discloses a video generation method, a video model training method and a video generation device, and relates to the technical field of computers. The method comprises the following steps: obtaining a video generation request, wherein the video generation request comprises a target resolution and at least one of a text prompt word, a reference picture, reference audio, and reference video; determining at least one intermediate resolution based on the target resolution in the video generation request; generating a first video hidden vector of an intermediate resolution based on the video generation request through a target video model; taking the first video hidden vector as a conditional constraint, generating a second video hidden vector of a target resolution through the target video model, and the target resolution is higher than the intermediate resolution; and decoding the second video hidden vector to obtain a target video. The above technical feature can guarantee the generation quality of a high-resolution video, reduce the calculation cost, and reduce the training and deployment cost.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a video generation method, a video model training method, and a video generation device. Background Technology

[0002] In recent years, video generation models have been widely used in film and television production, virtual reality, and intelligent creation. Existing high-resolution video generation methods are mainly divided into two categories: one is to directly generate high-resolution videos end-to-end using a large base model, and the other is to use a cascaded generation method, where a base model first generates a low-resolution video, and then a separate super-resolution model is used to upscale the low-resolution video to high resolution.

[0003] However, in end-to-end generation, the computational cost of the self-attention mechanism increases quadratically with the number of tokens. Directly generating high-resolution videos significantly increases memory usage and inference time, resulting in excessive computational overhead. While cascaded generation reduces computation, it requires separate training of the base model and the super-resolution model. Training capabilities cannot be reused, training time and costs increase, and deployment and scheduling complexity rises. Furthermore, the super-resolution model has a smaller number of parameters, and the generated high-resolution videos often fall short in terms of detail richness and image quality. Summary of the Invention

[0004] This invention provides a video generation method, a video model training method, and a video generation device, which reduce computational overhead and training and deployment costs while ensuring high-resolution video generation quality.

[0005] In a first aspect, embodiments of the present invention provide a video generation method, including: Obtain a video generation request, wherein the video generation request includes a target resolution and at least one of text prompts, reference images, reference audio, and reference video; At least one intermediate resolution is determined based on the target resolution in the video generation request; Based on the video generation request, a first video latent vector at intermediate resolution is generated using the target video model; Using the first video latent vector as a conditional constraint, a second video latent vector with a target resolution is generated through the target video model, wherein the target resolution is higher than the intermediate resolution; The target video is obtained by decoding the second video latent vector.

[0006] Secondly, embodiments of the present invention provide a video model training method, including: Acquire video generation training data, which includes at least one of target resolution, training target video, and text prompts, reference images, reference audio, and reference videos. At least one intermediate resolution is determined based on the target resolution in the video generation training data; Based on the video generation training data, the first video latent vector at the intermediate resolution is generated through the target video model; Using the first video latent vector as a conditional constraint, a second video latent vector with a target resolution is generated through the target video model, wherein the target resolution is higher than the intermediate resolution; The target video is obtained by decoding the second video latent vector; Determine the second loss value based on the target video and the training target video; The target video model is determined based on a first loss value and / or a second loss value, wherein the first loss value is used to train the target video model to generate first video latent vectors at different intermediate resolutions.

[0007] Thirdly, embodiments of the present invention provide a video generation apparatus, comprising: The request acquisition module is used to acquire a video generation request, wherein the video generation request includes a target resolution and at least one of text prompts, reference images, reference audio, and reference video. The first resolution determination module is used to determine at least one intermediate resolution based on the target resolution in the video generation request; The first generation module is used to generate a first video latent vector at an intermediate resolution based on the video generation request and the target video model. The second generation module is used to generate a second video latent vector with a target resolution by using the first video latent vector as a conditional constraint and the target video model. The target resolution is higher than the intermediate resolution. The first decoding module decodes the second video latent vector to obtain the target video.

[0008] Fourthly, embodiments of the present invention provide a video model training apparatus, comprising: The data acquisition module is used to acquire video generation training data, which includes at least one of the following: target resolution, training target video, text prompts, reference images, reference audio, and reference video. The second resolution determination module is used to determine at least one intermediate resolution based on the target resolution in the video-generated training data. The third generation module is used to generate training data based on the video and generate the first video latent vector of the intermediate resolution through the target video model. The fourth generation module is used to generate a second video latent vector with a target resolution by using the first video latent vector as a conditional constraint and the target video model, wherein the target resolution is higher than the intermediate resolution. The second decoding module is used to decode the second video latent vector to obtain the target video; The loss value determination module is used to determine a second loss value based on the target video and the training target video; The model training module is used to determine a target video model based on a first loss value and / or a second loss value, wherein the first loss value is used to train the target video model to generate first video latent vectors at different intermediate resolutions.

[0009] Fifthly, embodiments of the present invention provide an electronic device, comprising: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores a computer program that can be executed by at least one processor, such that the at least one processor can perform a video generation method provided in the first aspect embodiment or a video model training method provided in the second aspect embodiment.

[0010] In a sixth aspect, embodiments of the present invention provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute a video generation method provided in the first aspect embodiment or a video model training method provided in the second aspect embodiment.

[0011] In a seventh aspect, embodiments of the present invention provide a computer program product, the computer program product comprising a computer program, which, when executed by a processor, implements a video generation method provided in the first aspect embodiment or a video model training method provided in the second aspect embodiment.

[0012] The technical solution of this invention includes obtaining a video generation request, wherein the video generation request includes a target resolution and at least one of text prompts, reference images, reference audio, and reference video; determining at least one intermediate resolution based on the target resolution in the video generation request; generating a first video latent vector of the intermediate resolution using a target video model based on the video generation request; using the first video latent vector as a conditional constraint, generating a second video latent vector of the target resolution using the target video model, wherein the target resolution is higher than the intermediate resolution; and decoding the second video latent vector to obtain the target video. The above technical feature involves performing at least two generation operations using the same target video model: first generating a low-resolution first video latent vector based on the video generation request, and then using the first video latent vector as a conditional constraint to generate a high-resolution second video latent vector using the same target video model. The two generation processes share the same model, eliminating the need for additional super-resolution model training and reducing training and deployment costs. The first generation generates a lower-resolution video latent vector, while the second generation is performed under the constraint of the lower-resolution latent vector, reducing the number of generation steps. This avoids the quadratic computational increase caused by the increased number of generation steps in end-to-end direct generation of high-resolution video, thus reducing computational overhead. At the same time, the first video latent vector serves as a conditional constraint to ensure semantic consistency between the two generation processes, ensuring the generation quality of high-resolution video.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a video generation method provided in an embodiment of the present invention; Figure 2 This is a flowchart of a video model training method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a video generation device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a video model training device provided in an embodiment of the present invention; Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] In one embodiment, Figure 1 This is a flowchart of a video generation method provided by an embodiment of the present invention. This embodiment is applicable to high-resolution video generation. The method can be executed by a video generation device, which can be implemented in hardware and / or software.

[0019] like Figure 1 As shown, the method includes: S101. Obtain a video generation request. The video generation request includes a target resolution and at least one of the following: text prompts, reference images, reference audio, and reference video.

[0020] In this embodiment, the video generation request can be understood as an instruction or data used to trigger a video generation task, provided by the user or automatically invoked by the system. It includes conditional information required to generate the target video, such as at least one of the following: target resolution, text prompts, reference images, reference audio, and reference video. The target resolution can be understood as the resolution of the high-resolution video the user expects to output, such as 1080P, 2K, 4K, or 8K. The text prompts can be understood as natural language descriptions serving as references for target video generation, used to indicate the semantic content of the target video. Reference images can be understood as single-frame images serving as references for target video generation, used to provide an appearance reference for people, objects, or scenes in the target video. Reference audio can be understood as audio signals serving as references for target video generation, used to provide audio conditions such as dubbing, sound effects, timbre, or background music for the target video. Reference videos can be understood as existing video clips serving as references for target video generation, used to provide a timeline structure, image frame reference, motion structure, or style reference for the target video. Optionally, the video generation request may also include video aspect ratio and / or video duration requirements. The video aspect ratio and / or video duration requirements can be specified by the user or automatically invoked by the system in the video generation request, or they can be excluded from the video generation request and left as a default setting by the model.

[0021] Specifically, after obtaining a video generation request, it can be categorized into different video generation task types based on the types of input conditions contained in the request. For example, if the generation request only contains text prompts, it is a Text-to-Video (T2V) task, and the target video model can generate a video based on the text prompts. If the generation request contains both text prompts and reference images, it is a Text-Image-to-Video (TI2V) task, and the target video model needs to generate a video by referring to both the description of the text prompts and the appearance information in the reference images. If the generation request contains both text prompts and reference videos, it is a video editing or reference video generation task, and the target video model needs to generate a new video by referring to the temporal or motion structure of the reference video. If the generation request contains both text prompts and reference audio, it is an audio-visual generation task, and the target video model needs to generate synchronized video content based on the audio signal. Different combinations of input conditions correspond to different attention processing methods to adapt to the input requirements of various video generation task types.

[0022] S102. Determine at least one intermediate resolution based on the target resolution in the video generation request.

[0023] In this embodiment, the intermediate resolution can be understood as a transitional resolution determined based on the target resolution and used for the first generation. The intermediate resolution is smaller than the target resolution. The relationship between the target resolution and the intermediate resolution can be one-to-one or one-to-many, meaning that one or more intermediate resolutions can be determined for one target resolution. It can also be many-to-one or many-to-many, meaning that one or more intermediate resolutions can be determined for multiple target resolutions.

[0024] In one implementation, there is a single-level correspondence between the target resolution and the intermediate resolution, that is, an intermediate resolution is determined based on the target resolution. For example, when the target resolution is 2K, an intermediate resolution of 480P is determined.

[0025] In another implementation, there is a multi-level correspondence between the target resolution and the intermediate resolutions. That is, multiple intermediate resolutions are determined based on the target resolution, and the multiple intermediate resolutions are arranged in order from low to high resolution to form a progressively increasing resolution sequence. For example, when the target resolution is 8K, two intermediate resolutions, 480P and 2K, are determined, where 480P is the lowest level intermediate resolution and 2K is the highest level intermediate resolution.

[0026] The number of intermediate resolutions can be determined based on the target resolution, computing resources, or a preset scaling strategy. The higher the target resolution, the more intermediate resolutions can be determined. The method for determining intermediate resolutions can be set based on preset fixed values, the target resolution, the model's computing resources, or a preset scaling strategy. This embodiment does not impose any limitations on this, the purpose of which is to ensure that the first generation is completed at a lower resolution to reduce the amount of computation.

[0027] S103. Based on the video generation request, generate the first video latent vector at intermediate resolution using the target video model.

[0028] In this embodiment, the target video model can be understood as a deep learning model for video generation, capable of generating video latent vectors based on input conditions. The target video model can be simply a Diffusion Transformer (DiT) model used to generate video latent vectors, encoded and decoded by an existing Variational Autoencoder (VAE); alternatively, the target video model can include a Diffusion Transformer (DiT), an encoder, and a decoder for end-to-end video content generation. The encoder and decoder can employ a block-based encoding training method, encoding video frames into a video term (e.g., encoding one video term with every 4×4 pixels) according to a preset number of pixels, and then converting the video term into latent vectors in the latent space through a word embedding layer. Therefore, the target video model naturally supports video generation with different temporal and spatial dimensions. The core of the DiT model is a noise prediction network (noise predictor) based on the Transformer architecture. During the training phase, noisy latent vector samples are constructed by applying noise at different time steps to the clear video latent vectors, and the noise prediction network is trained to predict the added noise based on the current noisy latent vector and the time step. During the inference (generation) phase, starting with a random noise latent vector, in each iteration, the current noisy latent vector and the time step are input into the noise prediction network. Text prompts are injected into the network through Adaptive Layer Normalization (AdaLN) to control the generation direction. The network captures spatiotemporal dependencies through a self-attention mechanism and outputs predicted noise, then uses the predicted noise to update the current latent vector, achieving progressive denoising. After repeating the iteration for a preset number of steps, a clear video latent vector is obtained. During the denoising process, text prompts (and optional reference conditions) guide the denoising direction through the conditional injection mechanism of the target video model, ensuring that the generated first video latent vector is consistent with the semantic content in the video generation request. After denoising is completed, a first video latent vector at an intermediate resolution is obtained.

[0029] Video latent vectors can be understood as compressed representations of video data in the latent space, and are low-dimensional feature vector sequences of video frames. The first video latent vector can be understood as the video latent vector corresponding to the intermediate resolution generated by the target video model before generating the target resolution video.

[0030] Specifically, based on intermediate resolution and at least one of the video aspect ratio and video duration requirements, a first noisy latent vector at intermediate resolution is generated. Using the target video model, under the constraints of at least one of the text prompts, reference images, reference audio, and reference video in the video generation request, this first noisy latent vector is progressively denoised to obtain a first video latent vector representing a clear, intermediate-resolution first video. This first video latent vector corresponds to the intermediate resolution. During the generation process, a sparse attention mechanism can be used to reduce computational load.

[0031] Optionally, after generating the first video latent vector, the first video latent vector can be decoded by the decoder of the variational autoencoder (VAE) of the target video model to obtain a low-resolution video (LR) at intermediate resolution, which can be used for preview or as an intermediate result output.

[0032] S104. Using the first video latent vector as a conditional constraint, generate a second video latent vector with a target resolution higher than the intermediate resolution through the target video model.

[0033] In this embodiment, the second video latent vector can be understood as the video latent vector finally generated by the target video model. The target resolution can be understood as the resolution of the high-resolution video that the user ultimately expects, which is included in the video generation request.

[0034] Specifically, based on the target resolution and at least one of the video aspect ratio and video duration requirements, a second noisy latent vector at the target resolution is generated. The first video latent vector and the second noisy latent vector are aligned on the time axis, so that their frame sequences correspond one-to-one in time. Based on the aligned first video latent vector and constrained by at least one of the text prompts, reference images, reference audio, and reference videos in the video generation request, the second noisy latent vector is progressively denoised. That is, using the existing semantic and structural information in the low-resolution video latent vector, as well as the overall constraints and guidance information in the video generation request, the denoising process of the target video model generating the high-resolution video is constrained. This ensures that the second noisy latent vector is guided and constrained by the first video latent vector and the video generation request during denoising, thereby generating a second video latent vector that is semantically consistent with the first video latent vector but has a higher resolution. Since the first video latent vector and the second noisy latent vector have been aligned on the time axis during the generation process, the generated second video latent vector naturally remains aligned with the first video latent vector on the time axis.

[0035] During the generation process, a sparse attention mechanism can be used to reduce the amount of computation.

[0036] S105. Decode the second video latent vector to obtain the target video.

[0037] In this embodiment, the target video can be understood as the final output high-resolution video.

[0038] Specifically, the second video latent vector is decoded using the VAE decoder of the target video model, restoring it to a sequence of video frames in pixel space, thus obtaining a high-resolution (HR) video at the target resolution. Since the second video latent vector is generated under the conditional constraints of the first video latent vector, the decoded target video is semantically consistent with the low-resolution video corresponding to the first video latent vector, but has a higher resolution and richer details.

[0039] This invention provides a video generation method, comprising: acquiring a video generation request, the video generation request including a target resolution and at least one of text prompts, reference images, reference audio, and reference video; determining at least one intermediate resolution based on the target resolution in the video generation request; generating a first video latent vector of the intermediate resolution using a target video model based on the video generation request; using the first video latent vector as a conditional constraint, generating a second video latent vector of the target resolution using the target video model, the target resolution being higher than the intermediate resolution; and decoding the second video latent vector to obtain the target video. This technical solution, through the same target video model, completes the output of a high-resolution target video in two generation steps: first, a low-resolution first video latent vector is generated based on the video generation request; then, using the first video latent vector as a conditional constraint, a high-resolution second video latent vector is generated using the same target video model. The two generation processes share the same model, eliminating the need for additional super-resolution model training and reducing training and deployment costs. The first generation generates a lower-resolution video latent vector, while the second generation is performed under the constraint of the lower-resolution latent vector, avoiding the quadratic computational increase caused by directly generating high-resolution videos end-to-end and reducing computational overhead. At the same time, the first video latent vector serves as a conditional constraint to ensure semantic consistency between the two generation processes, ensuring the generation quality of high-resolution videos.

[0040] As a first optional embodiment of this example, based on the video generation request and the intermediate resolution, a first video latent vector at the intermediate resolution is generated through the target video model, including: a1. Generate the first noisy latent vector based on the video generation request and intermediate resolution.

[0041] In this embodiment, the first noisy latent vector can be understood as an initial latent vector with noise generated based on the video generation request and the intermediate resolution. The video generation request includes the video aspect ratio and / or video duration requirements.

[0042] Specifically, intermediate noisy latent vectors are automatically generated based on the intermediate resolution. For example, if the intermediate resolution is 512×512, the video generation request requires the generation of a 1-second video, the preset video frame rate is 20 frames per second, and each 16×16 pixels is encoded as a video word, then based on the intermediate resolution and the video generation request, a first noisy latent vector of 20×512×512 / (16×16) random noises can be generated.

[0043] b1. Denoise the first noisy latent vector using the target video model to generate the first video latent vector at an intermediate resolution.

[0044] In this embodiment, the first noisy latent vector is input into the target video model. Under the constraint of at least one of the text prompts, reference images, reference audio, and reference video in the video generation request, the target video model restores the noisy latent vector to a clear first video latent vector through a stepwise denoising method, with the corresponding resolution being intermediate resolution.

[0045] As a second optional embodiment of this example, based on the video generation request and the intermediate resolution, a first video latent vector at the intermediate resolution is generated through the target video model, including: a2. If the number of intermediate resolutions is greater than 1, sort them according to their size and determine the intermediate resolution with the smallest resolution as the current resolution.

[0046] In this embodiment, the number of intermediate resolutions can be one or more. When there is only one intermediate resolution, it indicates that the target resolution is low or multi-level generation is not required. The determined intermediate resolution is directly used as the current resolution, and the current intermediate latent vector corresponding to the current resolution is generated based on the video generation request. This current intermediate latent vector is then used as the first video latent vector. When the number of intermediate resolutions is greater than one, it indicates that the target resolution is high or that the resolution needs to be gradually increased through multi-level generation. The current resolution can be understood as the resolution level currently being processed during the multi-level generation process.

[0047] Specifically, if the number of intermediate resolutions is greater than one, they are sorted according to their size, and the intermediate resolution with the smallest resolution is determined as the current resolution, serving as the starting level for multi-level generation. For example, if the intermediate resolutions include three levels: 480P, 768P, and 1080P, 480P is determined as the current resolution, and generation begins from the lowest level.

[0048] b2. Based on the video generation request, generate the current intermediate latent vector corresponding to the current resolution through the target video model.

[0049] In this embodiment, the current intermediate latent vector can be understood as the video latent vector corresponding to the current resolution level.

[0050] Specifically, if the number of intermediate resolutions is 1, then the current intermediate latent vector is the first video latent vector. If the number of intermediate resolutions is greater than 1, then the current resolution is the smallest intermediate resolution. Based on the current resolution, and at least one of the video aspect ratio and video duration requirements, a noisy latent vector is generated. Under the guidance of the target video model and at least one of the constraints of text prompts, reference images, reference audio, and reference video in the video generation request, the noisy latent vector is gradually denoised to obtain the current intermediate latent vector corresponding to the current resolution. This current intermediate latent vector is the lowest level latent vector in the multi-level generation.

[0051] c2. Using the current intermediate latent vector as a conditional constraint, determine the next intermediate resolution of the current resolution in the sorting as the new current resolution, and generate the current intermediate latent vector corresponding to the current resolution through the target video model.

[0052] In this embodiment, the new current resolution can be understood as the next level resolution in the sorting. For example, the intermediate resolutions include 480P, 768P and 1080P. After the current intermediate latent vector for 480P has been generated, 768P is used as the new current resolution to generate the corresponding current intermediate latent vector.

[0053] Specifically, when the number of intermediate resolutions is greater than one, the current intermediate latent vector is used as a conditional constraint input to the target video model. The next intermediate resolution in the sorting is determined as the new current resolution, and the target video model generates the current intermediate latent vector corresponding to the new current resolution. During the generation process, the current intermediate latent vector and the noisy latent vector corresponding to the new current resolution are aligned on the time axis. The aligned current intermediate latent vector guides the target video model in the denoising process of the noisy latent vector, so that the resolution of the current intermediate latent vector corresponding to the new current resolution is improved while maintaining semantic consistency with the previous level.

[0054] d2. Continue until the current intermediate latent vector corresponding to the intermediate resolution with the highest resolution is generated, and use it as the first video latent vector.

[0055] In this embodiment, if the number of intermediate resolutions is greater than 1, it is determined whether the current resolution corresponding to the currently generated current intermediate latent vector is the intermediate resolution with the largest resolution in the sort. If so, the current intermediate latent vector is determined as the first video latent vector; otherwise, it returns to re-execute c2, uses the current intermediate latent vector as a conditional constraint input to the target video model, determines the next intermediate resolution of the current resolution in the sort as the new current resolution, generates the current intermediate latent vector corresponding to the new current resolution through the target video model, until the current intermediate latent vector corresponding to the intermediate resolution with the largest resolution is generated, and uses it as the first video latent vector.

[0056] Through the multi-level generation process described above, the first video latent vector is generated as the resolution is progressively increased. Each level's generation uses the output of the previous level as a constraint, ensuring that the final first video latent vector semantically matches the video generation request and corresponds to the intermediate resolution of the highest level. The final generated first video latent vector is then used to generate the second video latent vector at the target resolution.

[0057] As a third optional embodiment of this example, the first video latent vector is used as a conditional constraint to generate a second video latent vector at the target resolution through the target video model, including: S1031. Generate a second noisy latent vector based on the video generation request and the target resolution.

[0058] In this embodiment, the second noisy latent vector can be understood as an initial latent vector with noise generated based on the video generation request and the target resolution. The video generation request includes the video aspect ratio and / or video duration requirements.

[0059] Specifically, a second noisy latent vector corresponding to the target resolution can be generated based on at least one of the video aspect ratio and video duration requirements in the video generation request, and the target resolution. The spatial size of the second noisy latent vector is determined by the target resolution, and it is essentially a noisy sequence of latent vectors consisting of pure noise or random noise.

[0060] S1032. Align the first video latent vector with the second noisy latent vector on the time axis.

[0061] In this embodiment, the first video latent vector corresponds to a low-resolution video at an intermediate resolution, and the second noisy latent vector corresponds to a high-resolution video at the target resolution; both have the same number of frames. Timeline alignment establishes a correspondence between each frame of the first video latent vector and the corresponding frame in the second noisy latent vector, providing a foundation for subsequent denoising based on the aligned first video latent vector to guide and constrain the second noisy latent vector.

[0062] Through the alignment operation described above, the second video latent vector generated subsequently will naturally remain aligned with the first video latent vector on the time axis.

[0063] S1033. Based on the video generation request and the aligned first video latent vector, denoise the second noisy latent vector to generate the second video latent vector.

[0064] In this embodiment, the aligned first video latent vector is used as a constraint and input into the target video model. Guided by constraints from at least one of the text prompts, reference images, reference audio, and reference video in the video generation request, the target video model progressively denoises the second noisy latent vector. During denoising, the first video latent vector provides semantic and structural information about the low-resolution video. The video generation request provides indicated generation conditions through the target resolution, at least one of the text prompts, reference images, reference audio, and reference video, and at least one of the optional video aspect ratio and video duration requirements. Both conditions jointly guide and constrain the denoising direction of the target video model on the second noisy latent vector, ensuring that it restores high-resolution details while maintaining semantic and structural consistency with the first video latent vector and conforming to the content indicated by the video generation request. After denoising, a second video latent vector with semantic consistency but higher resolution is obtained.

[0065] As a fourth optional embodiment of this example, the target video model adopts a block coding training method, and the attention calculation of the target video model is sparsified during the process of generating the first video latent vector and / or the second video latent vector.

[0066] In this embodiment, the block encoding training method encodes video frames into video words based on fixed pixels (e.g., each 16×16 pixel is encoded into a video word), or adaptively encoded into video words based on semantic structure (e.g., ensuring that a complete eye falls within the same block). Then, a word embedding layer transforms the video words into latent vectors in the latent space. Since both the encoder and decoder use block encoding training, they can encode and decode videos of different resolutions. That is, the model inherently possesses the ability to encode and decode both high-resolution and low-resolution videos, and the word sequence lengths corresponding to different resolutions are different. Simultaneously, the target video model calculates attention at the word level, which is not limited by a fixed spatial size, thus naturally supporting the generation of videos with different temporal and spatial dimensions.

[0067] Sparsity processing is a method in which a subset of words are selected for attention calculation by first filtering them through index branches, thereby reducing the amount of computation.

[0068] Specifically, in generating the first and / or second video latent vectors, firstly, an input sequence is obtained, which includes at least visual lexical units; based on the attention conditions satisfied by each lexical unit in the input sequence, a corresponding attention strategy is assigned to each lexical unit, which includes at least dense processing strategies and sparse processing strategies. Lexical units that satisfy global semantic conditions are assigned to dense processing strategies, and these lexical units include at least text or audio lexical units; for visual lexical units that do not satisfy global semantic conditions, a corresponding attention strategy is assigned based on whether each visual lexical unit satisfies temporal and visual structure conditions. Visual lexical units include at least one of target video lexical units, reference video lexical units, frame conditional lexical units, and reference image lexical units; sparse processing strategies include joint sparse topology, visual sparse topology, and independent sparse topology. Frame conditional terms belonging to the target timeline are assigned to the joint sparse topology, target video terms belonging to the target timeline are assigned to the visual sparse topology, reference video terms that do not belong to the target timeline but have independent visual structures are assigned to the independent sparse topology, and reference image terms that do not belong to the target timeline and do not have independent visual structures are assigned to the dense processing strategy.

[0069] Then, the visual lexical units assigned to the sparse processing strategy are grouped according to the temporal and spatial dimensions to generate at least one lexical cube structure. Each lexical cube structure preserves the adjacency relationships of visual lexical units in at least two directions. Each lexical cube structure is scored, and the target lexical cube structure is determined based on the scoring results.

[0070] Finally, attention calculation is performed based on the target words in the target word cube structure and the words assigned to the dense processing strategy, and then the noisy video latent vector is denoised and output based on the attention calculation results.

[0071] By employing the above method, since the target video model computes attention at the word level, it naturally supports video generation with different temporal and spatial dimensions. Therefore, the same model can handle low-resolution videos in the first generation and high-resolution videos in the second. Simultaneously, combining it with block-based encoding training allows the same model to encode and decode videos of different resolutions, further enhancing its processing capabilities across various resolutions. Through sparsity processing, attention computation is performed only on a selected subset of words, significantly reducing computational load, memory usage, and inference time, thus improving video generation efficiency. This ensures high-resolution video generation quality while reducing computational overhead and deployment costs.

[0072] As a fifth optional embodiment of this example, the method further includes: a3. Based on the video generation requirements and / or request task type in the video generation request, determine the target parameter template of the target video model.

[0073] In this embodiment, the video generation requirement can be understood as the information carried in the video generation request that indicates the configuration requirements of the generation process, such as requirements for generation quality, generation speed, or computing resources. The request task type can be understood as the task category corresponding to the video generation request, such as text-to-video (T2V), image-to-text video (TI2V), video editing, or audio-visual generation. The target parameter template can be understood as the hyperparameter template of the target video model, used to control the behavior of the target video model during the generation process, including sampling strategies, denoising steps, and time-step sampling distribution. Under different hyperparameter configurations, the video quality generated by the target video model varies. For example, under certain parameter configurations, the target video model regenerates a larger proportion during the second generation, meaning the conditional constraints of the first video latent vector are weaker, suitable for text-to-video tasks; under certain parameter configurations, the video sharpening effect is better, suitable for scenarios requiring high video contrast; the more denoising steps, the higher the quality of the generated video, but the longer the time and the higher the cost.

[0074] In one implementation, a target parameter template is determined based on the type of the requested task. For example, a text-based video task corresponds to a first parameter template, under which the regeneration ratio of the second generation is relatively large and the conditional constraints on the first video latent vector are relatively weak; a video editing task corresponds to a second parameter template, under which the conditional constraints on the first video latent vector are relatively strong.

[0075] In another implementation, a target parameter template is determined based on the video generation requirements. For example, if video contrast is a requirement, a parameter template with better sharpening effect is selected; if high generation quality is required, a parameter template with more denoising steps is selected; if high generation speed is required, a parameter template with fewer denoising steps is selected. Optionally, the number of denoising steps can also be determined based on computing power and / or task priority. For example, if computing power is sufficient or task priority is high, more denoising steps are used to improve generation quality; if computing power is limited or task priority is low, fewer denoising steps are used to reduce generation cost.

[0076] After determining the target parameter template, it is used to control the generation process of the first video latent vector and / or the second video latent vector.

[0077] b3. Generate the first video latent vector and / or the second video latent vector based on the target parameter template.

[0078] In this embodiment, when generating the first video latent vector, the target video model performs a denoising process on the first noisy latent vector according to the hyperparameters, such as the sampling strategy, number of denoising steps, and time-step sampling distribution, as indicated by the target parameter template, to obtain the first video latent vector. When generating the second video latent vector, the target video model performs a denoising process on the second noisy latent vector under the constraints of the video generation request and the aligned first video latent vector, according to the hyperparameters, such as the sampling strategy, number of denoising steps, and time-step sampling distribution, as indicated by the target parameter template, to obtain the second video latent vector.

[0079] The target parameter template can be used only to generate the first video latent vector, or only to generate the second video latent vector, or it can be used to generate both the first and second video latent vectors simultaneously. The specific method can be determined based on the preset configuration and the video generation requirements and / or the request task type in the video generation request.

[0080] By employing the above methods, the target video model can be configured with matching hyperparameters under different task scenarios, achieving a balance between generation quality and generation efficiency.

[0081] This invention provides a video generation method that re-inputs the latent vectors of low-resolution videos as conditions into the same video model, ensuring strict consistency between the timelines of the low-resolution and high-resolution videos. This allows the video model to generate high-resolution videos under the strict constraints of the low-resolution videos. Compared to end-to-end direct generation of high-resolution videos, this method reduces the difficulty of high-resolution video generation, decreases the number of denoising steps, thereby reducing computational load, lowering costs, and improving efficiency. Compared to two-stage cascaded methods for generating high-resolution videos, this method reduces the number of models, shortens model training time, improves the performance of the model in generating high-resolution videos, and reduces the difficulty of model deployment and scheduling. Furthermore, combining sparse attention and video segmentation schemes can further reduce computational load and improve video generation quality.

[0082] In one embodiment, Figure 2 This is a flowchart of a video model training method provided in an embodiment of the present invention. This embodiment is applicable to the training of a target video model. The method can be executed by a video model training device, which can be implemented in hardware and / or software.

[0083] like Figure 2 As shown, the method includes: S201. Obtain video generation training data. The video generation training data includes a video generation request and a training target video. The video generation request includes a target resolution and at least one of the following: text prompts, reference images, reference audio, and reference video.

[0084] In this embodiment, the video generation training data can be understood as data used to train the target video model, including a video generation request and a training target video. The video generation request includes a target resolution and at least one of text prompts, reference images, reference audio, and reference videos. Optionally, the video generation request may also include at least one of video aspect ratio and video duration requirements. The training target video can be understood as a high-resolution video used as a training label, that is, the target video that the video generation request expects the target video model to generate.

[0085] Specifically, after acquiring the video to generate training data, it is used as input for the target video model training process. The target video is used to calculate the second loss value, the target resolution is used to determine the intermediate resolution, and at least one of the following—text prompts, reference images, reference audio, and reference video—is used as prompts and generation conditions.

[0086] S202. Determine at least one intermediate resolution based on the target resolution in the video generation request.

[0087] The processing logic is the same as S102, and will not be repeated here.

[0088] S203. Based on the video generation request, generate the first video latent vector at intermediate resolution using the target video model.

[0089] The processing logic is the same as that of S103, and will not be repeated here.

[0090] S204. Using the first video latent vector as a conditional constraint, generate a second video latent vector with a target resolution higher than the intermediate resolution through the target video model.

[0091] The processing logic is the same as that of S104, and will not be repeated here.

[0092] S205. Decode the second video latent vector to obtain the target video.

[0093] The processing logic is the same as that of S105, and will not be repeated here.

[0094] The target video is a predicted video generated by the target video model during training, used for comparison with the training target video.

[0095] S206. Determine the second loss value based on the target video and the training target video.

[0096] In this embodiment, the second loss value can be understood as a loss value determined based on the difference between the target video generated by the target video model and the training target video, and is used to measure the deviation between the target video generated by the target video model and the training target video.

[0097] Specifically, a second loss value is determined based on the target video generated by the target video model and the training target video in the video generation training data. The second loss value can be calculated using pixel contrast loss, mean squared error, perceptual loss, or other loss functions. The second loss value directly optimizes the ability of the target video model to generate target videos at the target resolution. Through backpropagation of gradients, the ability of the target video model to generate first video latent vectors at intermediate resolution can also be trained simultaneously.

[0098] In this process, the target resolution serves as the direct optimization objective, enabling the target video model to clearly define what kind of target video should be generated to reduce the second loss value. In contrast, the intermediate resolution may not have a direct reference. The target video model does not aim to generate a specific intermediate resolution video as its optimization objective. Instead, it learns and adjusts parameters to generate a first video latent vector that can make the final target video better.

[0099] S207. Train the target video model based on the first loss value and / or the second loss value, wherein the first loss value is used to train the target video model's ability to generate a first video latent vector at an intermediate resolution.

[0100] In this embodiment, the first loss value can be understood as the loss value used to train the target video model to generate the first video latent vector at an intermediate resolution. The first loss value can be determined based on the difference between the first video at an intermediate resolution generated by decoding the first video latent vector and the training intermediate video downsampled from the training target video to the intermediate resolution. The first loss value can be calculated using pixel contrast loss, mean squared error, perceptual loss, or other loss functions. The first loss value takes the generation of the first video at an intermediate resolution by the training target video model as the direct optimization objective. Through backpropagation of gradients, the ability of the target video model to generate the target video at the target resolution can also be trained simultaneously.

[0101] The first and second loss values ​​work in the same way, with the direct optimization goal being to generate a video of a specific resolution from the target video model. Through training, the target video model is made capable of generating the final target video.

[0102] In one implementation, the target video model is trained solely based on a first loss value. The parameters of the target video model are updated according to the first loss value, enabling the target video model to learn to generate a first video at an intermediate resolution as the direct optimization objective. This gives the target video model the ability to denoise and generate high-quality intermediate-resolution videos, as well as the ability to generate videos at the target resolution.

[0103] In one implementation, the target video model is trained solely based on a second loss value. The parameters of the target video model are updated according to the second loss value, enabling the model to learn to generate the target video at the target resolution as the direct optimization objective. This gives the target video model the ability to denoise and generate high-quality target-resolution videos, as well as the ability to generate intermediate-resolution videos.

[0104] In one implementation, the target video model is trained jointly based on a first loss value and a second loss value. The parameters of the target video model are updated according to the weighted sum of the first and second loss values, enabling the target video model to simultaneously learn the target ability to generate a high-quality intermediate-resolution first video and the target ability to generate a high-quality target-resolution target video.

[0105] In one implementation, the target video model is first trained based on a first loss value, enabling it to have the basic ability to generate a first video at an intermediate resolution. Then, the target video model is trained again based on the first and second loss values, enabling it to further learn the ability to generate a target video at the target resolution under the latent vector constraints of a low-resolution video, while still possessing the basic capabilities.

[0106] By using any of the above training methods, the target video model can be made capable of generating both a first video at an intermediate resolution and a target video at the target resolution under the latent vector constraints of a low-resolution video.

[0107] This invention provides a video model training method, comprising: acquiring video generation training data, which includes a video generation request and a training target video; the video generation request including a target resolution and at least one of text prompts, reference images, reference audio, and reference videos; determining at least one intermediate resolution based on the target resolution in the video generation request; generating a first video latent vector at the intermediate resolution using a target video model based on the video generation request; using the first video latent vector as a constraint, generating a second video latent vector at the target resolution using the target video model, where the target resolution is higher than the intermediate resolution; decoding the second video latent vector to obtain the target video; determining a second loss value based on the target video and the training target video; and training the target video model based on the first loss value and / or the second loss value, where the first loss value is used to train the target video model's ability to generate the first video latent vector at the intermediate resolution. This technical feature, by sequentially generating the first video latent vector at the intermediate resolution and the second video latent vector at the target resolution using the same target video model, enables the model to achieve high-resolution video generation without relying on an independently trained super-resolution model, reducing training and deployment costs. The ability to generate intermediate-resolution videos by training the model with a first loss value and constraining the generated target video to be consistent with the training target video by a second loss value improves the quality of high-resolution video generation. Training the target video model based on the first and / or second loss values ​​supports joint training or individual training, enhancing the flexibility and stability of the training process.

[0108] As a first optional embodiment of this method, the method further includes: a4. Obtain training data of various resolutions and train the target video model separately, so that the target video model has the ability to independently denoise and generate videos of the corresponding resolution without relying on conditional constraints.

[0109] In this embodiment, the conditional constraint can be understood as the constraint of the first video latent vector on the generation process of the second video latent vector in the aforementioned embodiment, that is, using the low-resolution video latent vector to guide the denoising process of the high-resolution video.

[0110] Specifically, training data of various resolutions, such as 480P, 1080P, 2K, and 4K, are acquired and used to train the target video model. During training, for each resolution of training data, the noisy latent vector corresponding to that resolution is input into the target video model. The target video model then denoises this noisy latent vector, generating a video latent vector for the corresponding resolution. The generated video at the corresponding resolution is then decoded based on this video latent vector. A loss value is determined based on the generated video and the target video in the training data, and the parameters of the target video model are updated according to the loss value. In this training process, the target video model does not rely on the first video latent vector as a constraint; instead, it independently performs denoising and generation based on the input conditions, thus enabling the target video model to independently generate videos at the corresponding resolution.

[0111] Through the above training, the target video model can generate video latent vectors of corresponding resolutions in subsequent training stages in response to video generation requests of different resolutions.

[0112] As a second optional embodiment of this example, the method further includes: a5. Downsample the training target video to a preset resolution to obtain the downsampled video.

[0113] In this embodiment, the preset resolution can be understood as a pre-set intermediate resolution used in the intermediate generation process of the video model, and its value is lower than the resolution of the training target video. The downsampled video can be understood as a low-resolution video obtained by downsampling the training target video to the preset resolution. The specific value of the preset resolution can be determined based on the intermediate resolution that the target video model can directly generate. For example, if the base model can directly generate a 480P intermediate resolution video, then the preset resolution can be set to 480P. Downsampling leads to the loss of high-frequency texture details; the downsampled video only retains the basic structure consistent with the text semantics.

[0114] Specifically, the training target video is downsampled to a preset resolution to obtain a downsampled video. Downsampling can be achieved through bilinear interpolation, nearest neighbor interpolation, etc., which can convert high-resolution training target videos into low-resolution versions, serving as input for subsequent degradation processing.

[0115] b5. The downsampled video is sequentially encoded, denoised, denoised, and decoded to obtain a degraded reference video. The degraded reference video is used to simulate the degradation characteristics of the target video model to generate a video at a preset resolution.

[0116] In this embodiment, encoding is the process of mapping the downsampled video from pixel space to latent space, which can be performed by the 3D VAE encoder of the target video model. Noise addition is the process of adding noise to the latent vectors of the downsampled video. Denoising is the process of removing noise from the noisy latent vectors using the target video model. Decoding is the process of mapping the denoised latent vectors back from latent space to pixel space, which can be performed by the 3D VAE decoder of the target video model. The degraded reference video can be understood as a low-resolution video obtained after degrading the downsampled video, used to simulate the degradation features of the intermediate-resolution video generated by the target video model.

[0117] Degradation features can be categorized into two types: the first type is the loss of high-frequency texture details due to downsampling, retaining only the basic structure consistent with the text semantics; the second type is the imperfect ability of the target video model to coordinate text and visual references, leading to distortion of identity and local structure or insufficient response to reference conditions. Traditional degradation methods such as adding noise, blurring, and compression can only simulate the first type of problem, not the second type, and are insufficient to train the target video model to correct generative biases in intermediate resolution videos and generate high-quality target resolution videos.

[0118] Therefore, the downsampled video is first encoded into the latent space using a 3D VAE encoder to obtain downsampled latent vectors. Noise is added to the downsampled latent vectors during the forward diffusion process, with the noise addition steps k randomly sampled from a preset range [K1, K2], where K1 represents no noise and K2 represents the maximum noise threshold for preserving the main structural information of the input video. Then, the noisy downsampled latent vectors are reverse-denoised using the target video model, with k denoising steps. During denoising, only text prompts are used as conditions, without introducing visual reference conditions such as reference images or videos. This limits the denoising result to a range that still preserves the main structure, thus simulating the degradation scenario where the target video model fails to respond adequately to visual reference conditions when generating intermediate-resolution videos based solely on text conditions—the second type of degradation mentioned above. After denoising, the denoised latent vectors are decoded using a 3D VAE decoder to obtain a low-resolution video whose structure is reinterpreted by the target video model. Finally, traditional synthetic degradation, including blurring, noise, and compression, can be superimposed on the decoded low-resolution video to introduce high-frequency quality degradation, namely the first type of degradation features mentioned above, to obtain the final degraded reference video.

[0119] The degraded reference video obtained through the above process includes both local structural and identity biases caused by the denoising and generation of the target video model, as well as high-frequency quality degradation caused by traditional synthesis and compression. Therefore, it can more realistically simulate the degradation characteristics of intermediate-resolution videos generated by the target video model. This degraded reference video is used as an intermediate-resolution input video constraint when training the target video model to generate high-resolution target videos, enabling the target video model to learn to generate correct high-resolution target videos under its constraints even when the intermediate-resolution videos exhibit the aforementioned degradation.

[0120] Accordingly, acquiring training data at various resolutions and training the target video model accordingly also includes: a41. Using the latent vector corresponding to the degraded reference video as a conditional constraint, generate a second video latent vector with a target resolution higher than the preset resolution through the target video model.

[0121] In this embodiment, the latent vector corresponding to the degraded reference video can be understood as the low-resolution video latent vector obtained after encoding the degraded reference video. Its resolution corresponds to a preset resolution and includes the aforementioned two types of degradation features.

[0122] Specifically, a second noisy latent vector is first generated based on the target resolution, video aspect ratio, and video duration requirements. The latent vector corresponding to the degraded reference video is then aligned with the second noisy latent vector on the time axis. Using the aligned latent vector of the degraded reference video as a constraint, and guided by at least one of the constraints in the video generation request (text prompts, reference images, reference audio, and reference video), the target video model progressively denoises the second noisy latent vector to generate a second video latent vector at the target resolution. Since the degraded reference video contains both local structural and identity biases caused by the denoising generation of the target training model and high-frequency quality degradation caused by traditional synthesis and compression, the target video model, trained under these conditions, can learn to generate a correct high-resolution video even when the low-resolution input video exhibits the aforementioned degradation. For example, even when the first video latent vector at an intermediate resolution generated by the target video model exhibits the aforementioned degradation, it can still generate a correct high-resolution video.

[0123] a42. Decode the latent vector of the second video to obtain the target video, and train the target video model based on the target video and the training target video.

[0124] In this embodiment, the second video latent vector is decoded to obtain the target video, the target video is compared with the training target video, the loss value is calculated, and the parameters of the target video model are updated based on the loss value.

[0125] With the aforementioned technical features, the target video model learns to generate a high-resolution video consistent with the training target video under the condition of a degraded reference video, thereby improving the robustness of the target video model to the degradation features of low-resolution videos.

[0126] In one embodiment, Figure 3 This is a schematic diagram of the structure of a video generation device provided in an embodiment of the present invention. Figure 3 As shown, the device includes: The request acquisition module 31 is used to acquire a video generation request, wherein the video generation request includes a target resolution and at least one of text prompts, reference images, reference audio, and reference video. The first resolution determination module 32 determines at least one intermediate resolution based on the target resolution in the video generation request; The first generation module 33 is used to generate a first video latent vector at an intermediate resolution based on the video generation request and the target video model. The second generation module 34 is used to generate a second video latent vector with a target resolution by using the first video latent vector as a conditional constraint and the target video model, wherein the target resolution is higher than the intermediate resolution. The first decoding module 35 decodes the second video latent vector to obtain the target video.

[0127] The video generation device used in this technical solution reduces computational overhead and lowers training and deployment costs while ensuring high-resolution video generation quality.

[0128] Optionally, the first generation module 33 is specifically used for: Based on the video generation request and the intermediate resolution, a first noisy latent vector is generated; The first noisy latent vector is denoised using the target video model to generate a first video latent vector at an intermediate resolution.

[0129] Optionally, the first generation module 33 is specifically used for: If the number of intermediate resolutions is greater than 1, sort them according to their size, and determine the intermediate resolution with the smallest resolution as the current resolution; Based on the video generation request, the current intermediate latent vector corresponding to the current resolution is generated through the target video model; Using the current intermediate latent vector as a conditional constraint, the next intermediate resolution of the current resolution in the sorting is determined as the new current resolution, and the current intermediate latent vector corresponding to the current resolution is generated through the target video model; The process continues until the current intermediate latent vector corresponding to the highest resolution is generated, which is then used as the first video latent vector.

[0130] Optionally, the second generation module 34 is specifically used for: A second noisy latent vector is generated based on the video generation request and the target resolution; Align the first video latent vector with the second noisy latent vector on the time axis; Based on the aligned first video latent vector, the second noisy latent vector is denoised to generate the second video latent vector.

[0131] Optionally, the target video model adopts a block-based coding training method, and the attention calculation of the target video model is sparsified during the process of generating the first video latent vector and / or the second video latent vector.

[0132] Optionally, the device further includes a latent vector generation module, specifically used for: Based on the video generation requirements and / or request task type in the video generation request, determine the target parameter template of the target video model; The first video latent vector and / or the second video latent vector are generated based on the target parameter template.

[0133] The video generation apparatus provided in this embodiment of the invention can execute the video generation method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0134] In one embodiment, Figure 4 This is a schematic diagram of the structure of a video model training device provided in an embodiment of the present invention. Figure 4 As shown, the device includes: Data acquisition module 41 is used to acquire video generation training data, the video generation training data including video generation request and training target video, the video generation request including target resolution, training target video and at least one of text prompt words, reference image, reference audio and reference video; The second resolution determination module 42 is used to determine at least one intermediate resolution based on the target resolution in the video generation request; The third generation module 43 is used to generate the first video latent vector of the intermediate resolution based on the video generation request and the target video model. The fourth generation module 44 is used to generate a second video latent vector with a target resolution by using the first video latent vector as a conditional constraint and through the target video model, wherein the target resolution is higher than the intermediate resolution; The second decoding module 45 is used to decode the second video latent vector to obtain the target video; The loss value determination module 46 is used to determine a second loss value based on the target video and the training target video; The model training module 47 is used to train the target video model based on a first loss value and / or a second loss value, wherein the first loss value is used to train the target video model to generate the first video latent vector at the intermediate resolution.

[0135] The video model training device used in this technical solution reduces training and deployment costs, improves the generation quality of high-resolution videos, and enhances the flexibility and stability of training.

[0136] Optionally, the device further includes an independent training module, specifically used for: Multiple training data of different resolutions are acquired and used to train the target video model, enabling the target video model to independently denoise and generate videos of the corresponding resolutions without relying on conditional constraints.

[0137] Optionally, the device further includes a degradation module, specifically used for: The training target video is downsampled to a preset resolution to obtain the downsampled video; The downsampled video is sequentially encoded, denoised, denoised, and decoded to obtain a degraded reference video. The degraded reference video is used to simulate the degradation features of the target video model to generate a video at a preset resolution.

[0138] Accordingly, the independent training module is specifically used for: Using the latent vector corresponding to the degraded reference video as a conditional constraint, a second video latent vector with a target resolution higher than the preset resolution is generated through the target video model. The second video latent vector is decoded to obtain the target video, and the target video model is trained based on the target video and the training target video.

[0139] The video generation apparatus provided in this embodiment of the invention can execute the video model training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0140] In one embodiment, Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. For example... Figure 5The diagram illustrates a schematic representation of an electronic device 10 that can be used to implement embodiments of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 10 may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0141] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0142] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0143] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, central processing units, graphics processing units, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, digital signal processors, and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as video generation methods or video model training methods.

[0144] In some embodiments, the video generation method or video model training method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the video generation method or video model training method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the video generation method or video model training method by any other suitable means (e.g., by means of firmware).

[0145] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays, application-specific integrated circuits (ASICs), application-specific standard products (ASICs), systems-on-a-chip (SoCs), payload programmable logic devices, computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0147] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical fibers, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0148] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a cathode ray tube or liquid crystal display) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0149] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0150] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product within the cloud computing service system to address the shortcomings of traditional physical host and virtual machine services, such as high management difficulty and weak business scalability.

[0151] This invention also provides a computer program product, including a computer program that, when executed by a processor, can implement the video generation method or video model training method provided in any embodiment of this invention.

[0152] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0153] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0154] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A video generation method, characterized in that, include: Obtain a video generation request, wherein the video generation request includes a target resolution and at least one of text prompts, reference images, reference audio, and reference video; At least one intermediate resolution is determined based on the target resolution in the video generation request; Based on the video generation request, the first video latent vector of the intermediate resolution is generated through the target video model; Using the first video latent vector as a conditional constraint, a second video latent vector with a target resolution is generated through the target video model, wherein the target resolution is higher than the intermediate resolution; The target video is obtained by decoding the second video latent vector.

2. The method according to claim 1, characterized in that, The step of generating the first video latent vector at the intermediate resolution based on the video generation request and using the target video model includes: Based on the video generation request and the intermediate resolution, a first noisy latent vector is generated; The first noisy latent vector is denoised using the target video model to generate the first video latent vector at the intermediate resolution.

3. The method according to claim 1, characterized in that, The step of generating the first video latent vector at the intermediate resolution based on the video generation request and using the target video model includes: If the number of intermediate resolutions is greater than 1, sort them according to their size, and determine the intermediate resolution with the smallest resolution as the current resolution; Based on the video generation request, the current intermediate latent vector corresponding to the current resolution is generated through the target video model; Using the current intermediate latent vector as a conditional constraint, the next intermediate resolution of the current resolution in the sorting is determined as the new current resolution, and the current intermediate latent vector corresponding to the current resolution is generated through the target video model; The process continues until the current intermediate latent vector corresponding to the highest resolution is generated, which is then used as the first video latent vector.

4. The method according to claim 1, characterized in that, The step of using the first video latent vector as a conditional constraint to generate a second video latent vector at the target resolution through the target video model includes: A second noisy latent vector is generated based on the video generation request and the target resolution; Align the first video latent vector with the second noisy latent vector on the time axis; Based on the video generation request and the aligned first video latent vector, the second noisy latent vector is denoised using the target video model to generate the second video latent vector.

5. The method according to claim 1, characterized in that, The target video model is trained using a block-based encoding method. During the generation of the first video latent vector and / or the second video latent vector, the attention calculation of the target video model is sparsified.

6. The method according to claim 1, characterized in that, Also includes: Based on the video generation requirements and / or request task type in the video generation request, determine the target parameter template of the target video model; The first video latent vector and / or the second video latent vector are generated based on the target parameter template.

7. A video model training method, characterized in that, include: Acquire video generation training data, which includes video generation requests and training target videos. The video generation requests include target resolution and at least one of text prompts, reference images, reference audio, and reference videos. At least one intermediate resolution is determined based on the target resolution in the video generation request; Based on the video generation request, the first video latent vector of the intermediate resolution is generated through the target video model; Using the first video latent vector as a conditional constraint, a second video latent vector with a target resolution is generated through the target video model, wherein the target resolution is higher than the intermediate resolution; The target video is obtained by decoding the second video latent vector; Determine the second loss value based on the target video and the training target video; The target video model is trained based on a first loss value and / or a second loss value, wherein the first loss value is used to train the target video model to generate the first video latent vector at the intermediate resolution.

8. The method according to claim 7, characterized in that, Also includes: Multiple training data of different resolutions are acquired and used to train the target video model, enabling the target video model to independently denoise and generate videos of the corresponding resolutions without relying on conditional constraints.

9. The method according to claim 8, characterized in that, Also includes: The training target video is downsampled to a preset resolution to obtain the downsampled video; The downsampled video is sequentially encoded, denoised, denoised, and decoded to obtain a degraded reference video. The degraded reference video is used to simulate the degradation features of the target video model to generate a video at a preset resolution. The step of acquiring training data at multiple different resolutions and training the target video model accordingly also includes: Using the latent vector corresponding to the degraded reference video as a conditional constraint, a second video latent vector with a target resolution higher than the preset resolution is generated through the target video model. The second video latent vector is decoded to obtain the target video, and the target video model is trained based on the target video and the training target video.

10. A video generation apparatus, characterized in that, include: The request acquisition module is used to acquire a video generation request, wherein the video generation request includes a target resolution and at least one of text prompts, reference images, reference audio, and reference video. The first resolution determination module is used to determine at least one intermediate resolution based on the target resolution in the video generation request; The first generation module is used to generate a first video latent vector at an intermediate resolution based on the video generation request and the target video model. The second generation module is used to generate a second video latent vector with a target resolution by using the first video latent vector as a conditional constraint and the target video model. The target resolution is higher than the intermediate resolution. The first decoding module decodes the second video latent vector to obtain the target video.

Images