Video generation methods, apparatus, equipment, media and products

By expanding and extracting key and transitional segments from the user-input text content, and generating and splicing video frames and segments, the problem of low efficiency and disjointed content in the existing technology for generating long videos is solved, and the efficient generation of coherent and consistent long videos is achieved.

CN122093636APending Publication Date: 2026-05-26CHINA MOBILE JIUTIAN ARTIFICIAL INTELLIGENCE TECHNOLOGY (BEIJING) CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610133110.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-01-30
Publication Date
2026-05-26

AI Technical Summary

Technical Problem

Existing technologies struggle to efficiently generate minute-long videos, and the generated video content is disjointed and inconsistent, resulting in low generation efficiency and poor quality.

Method used

By expanding the basic text content input by the user, key and transitional content segments are extracted, and key video frames, key video segments, and transitional video segments are generated using a video generation model. These segments are then spliced ​​together to form a long video.

Benefits of technology

It improves the efficiency of video generation, ensures the consistency of video content, and enhances the quality of generated videos.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122093636A_ABST
    Figure CN122093636A_ABST
Patent Text Reader

Abstract

This invention provides a video generation method, apparatus, device, medium, and product, relating to the field of artificial intelligence technology. It addresses the shortcomings of low efficiency and poor quality in text-based video generation, achieving efficient video generation from text content and improving the quality of the generated video content. The method includes: expanding user-inputted basic text content to obtain expanded text content; extracting content from the expanded text content to obtain multiple key content segments and multiple transitional content segments, wherein the number of text features included in the key content segments is greater than a preset number; generating multiple key video frames, multiple key video segments, and multiple transitional video segments using a video generation model based on the expanded text content, the multiple key content segments, and the multiple transitional content segments; and generating a target video corresponding to the basic text content based on the multiple key video frames, multiple key video segments, and multiple transitional video segments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a video generation method, apparatus, device, medium, and product. Background Technology

[0002] Text-generated video is an AI-based video generation method that generates videos based on user-provided text content. Most current text-generated video algorithms are based on a diffusion framework, which progressively denoises random noise and generates semantically correct video content guided by text prompts. However, this method produces short videos (e.g., 5 or 10 seconds) ranging from 80 to 160 frames per second, which is insufficient to meet users' demand for longer videos (lasting several minutes).

[0003] This is because existing methods rely on serial generation logic. As video length increases, the computational cost of generating the video grows linearly or exponentially. Therefore, the generation efficiency of long videos contradicts the video length, making it difficult to meet the demand for rapid generation of long videos. Furthermore, using static keyframes or isolated video blocks as constraints fails to convey dynamic information such as motion rate and trajectory, leading to stuttering and semantic drift in the resulting long videos, resulting in issues like video discontinuity or inconsistencies. Therefore, current methods for generating videos based on text content are inefficient and produce poor-quality video content. Summary of the Invention

[0004] This invention provides a video generation method, apparatus, device, medium, and product to address the shortcomings of existing technologies in generating videos based on text content, such as low efficiency and poor quality of the generated video content. It achieves efficient video generation based on text content, improves the quality of the generated video content, and ensures that the video content is coherent and consistent.

[0005] This invention provides a video generation method, comprising the following steps.

[0006] Expand the basic text content input by the user to obtain expanded text content; Content extraction is performed on the extended text content to obtain multiple key content fragments and multiple transitional content fragments. The number of text features included in the key content fragments is greater than the preset number, and the transitional content fragments are the content fragments between two adjacent key content fragments. Based on extended text content, multiple key content fragments, and multiple transitional content fragments, multiple key video frames, multiple key video fragments, and multiple transitional video fragments are generated through a video generation model. Based on multiple key video frames, multiple key video segments, and multiple transition video segments, the target video corresponding to the basic text content is generated.

[0007] According to a video generation method provided by the present invention, the video generation model includes: a key video frame generation model, a key video segment generation model, and a transition video segment generation model; Based on extended text content, multiple key content fragments, and multiple transitional content fragments, a video generation model generates multiple key video frames, multiple key video fragments, and multiple transitional video fragments, including: The extended text content and multiple key content fragments are input into the key video frame generation model to generate multiple key video frames. Each key video frame corresponds to a key content fragment, and the content included in multiple key video frames satisfies spatial consistency. Multiple key video frames and multiple key content segments are input into the key video segment generation model to generate multiple key video segments. One key video frame corresponds to one key video segment, and the content included in multiple key video segments satisfies spatial consistency and dynamic consistency. Multiple key video clips and multiple transition content clips are input into the transition video clip generation model to generate multiple transition video clips.

[0008] According to a video generation method provided by the present invention, the video generation model is trained in the following manner: Construct a training dataset, which includes multiple sets of training data. Each set of training data includes: training videos, complete video description text of the training videos, multiple training key video clips, key video description text of each training key video clip, multiple training transition video clips, and transition video description text of each training transition video clip. The multiple training key video clips and multiple training transition video clips are video clips extracted from the training videos. Based on the diffusion process of the preset diffusion model and the training dataset, the preset model is trained to obtain the video generation model.

[0009] According to a video generation method provided by the present invention, the method further includes: Identify multiple key training video frames from the training video; For each training key video frame in a plurality of training key video frames, determine N video frames that are adjacent to each training key video frame from the training video. Each training key video frame and its N adjacent video frames form a video segment, which is then defined as a training key video segment. One training key video frame corresponds to one training key video segment. The video segment consisting of video frames between two adjacent key training video segments is identified as the training transition video segment.

[0010] According to a video generation method provided by the present invention, the preset model includes a preset key video frame generation model; Based on the diffusion process of the pre-defined diffusion model and the training dataset, the pre-defined model is trained to obtain a video generation model, including: For each set of training data in multiple sets of training data, determine the training key video frame corresponding to each training key video segment in multiple training key video segments, and obtain multiple training key video frames. Video encoding is performed on multiple key training video frames to extract video features from the multiple key training video frames; Noise is added to the video features of multiple key training video frames to obtain Gaussian noise; Text encoding is performed on the complete video description text and the key video description text to extract the complete text features of the complete video description text and the key text features of the key video description text. The complete text features and key text features are fused to obtain the fused text features; Based on Gaussian noise and fused text features, a preset key video frame generation model is trained to obtain the key video frame generation model.

[0011] According to a video generation method provided by the present invention, a key video frame generation model is trained based on Gaussian noise and fused text features to obtain the key video frame generation model, including: When the number of multiple training key video segments exceeds the preset number, image encoding is performed on the preset reference image to extract the image features of the preset reference image; Based on Gaussian noise, fused text features and image features, a preset key video frame generation model is trained to obtain the key video frame generation model.

[0012] The present invention also provides a video generation apparatus, comprising the following modules: an extension module, an extraction module, and a processing module; The extension module is used to expand the basic text content input by the user to obtain extended text content; The extraction module is used to extract content from the extended text content, resulting in multiple key content fragments and multiple transitional content fragments. The number of text features included in the key content fragments is greater than the preset number, and the transitional content fragments are the content fragments between two adjacent key content fragments. The processing module is used to generate multiple key video frames, multiple key video segments, and multiple transitional video segments based on extended text content, multiple key content segments, and multiple transitional content segments, using a video generation model. The processing module is also used to generate target videos corresponding to basic text content based on multiple key video frames, multiple key video segments, and multiple transition video segments.

[0013] According to a video generation apparatus provided by the present invention, the video generation model includes: a key video frame generation model, a key video segment generation model, and a transition video segment generation model; the processing module is specifically used for: The extended text content and multiple key content fragments are input into the key video frame generation model to generate multiple key video frames. Each key video frame corresponds to a key content fragment, and the content included in multiple key video frames satisfies spatial consistency. Multiple key video frames and multiple key content segments are input into the key video segment generation model to generate multiple key video segments. One key video frame corresponds to one key video segment, and the content included in multiple key video segments satisfies spatial consistency and dynamic consistency. Multiple key video clips and multiple transition content clips are input into the transition video clip generation model to generate multiple transition video clips.

[0014] According to the video generation apparatus provided by the present invention, the processing module is further used for: Construct a training dataset, which includes multiple sets of training data. Each set of training data includes: training videos, complete video description text of the training videos, multiple training key video clips, key video description text of each training key video clip, multiple training transition video clips, and transition video description text of each training transition video clip. The multiple training key video clips and multiple training transition video clips are video clips extracted from the training videos. Based on the diffusion process of the preset diffusion model and the training dataset, the preset model is trained to obtain the video generation model.

[0015] According to the video generation apparatus provided by the present invention, the processing module is further used for: Identify multiple key training video frames from the training video; For each training key video frame in a plurality of training key video frames, determine N video frames that are adjacent to each training key video frame from the training video. Each training key video frame and its N adjacent video frames form a video segment, which is then defined as a training key video segment. One training key video frame corresponds to one training key video segment. The video segment consisting of video frames between two adjacent key training video segments is identified as the training transition video segment.

[0016] According to a video generation apparatus provided by the present invention, the preset model includes a preset key video frame generation model; the processing module is specifically used for: For each set of training data in multiple sets of training data, determine the training key video frame corresponding to each training key video segment in multiple training key video segments, and obtain multiple training key video frames. Video encoding is performed on multiple key training video frames to extract video features from the multiple key training video frames; Noise is added to the video features of multiple key training video frames to obtain Gaussian noise; Text encoding is performed on the complete video description text and the key video description text to extract the complete text features of the complete video description text and the key text features of the key video description text. The complete text features and key text features are fused to obtain the fused text features; Based on Gaussian noise and fused text features, a preset key video frame generation model is trained to obtain the key video frame generation model.

[0017] According to the present invention, a video generation apparatus, including a processing module, is specifically used for: When the number of multiple training key video segments exceeds the preset number, image encoding is performed on the preset reference image to extract the image features of the preset reference image; Based on Gaussian noise, fused text features and image features, a preset key video frame generation model is trained to obtain the key video frame generation model.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video generation method as described above.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video generation method as described above.

[0020] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the video generation method as described above.

[0021] This invention provides a video generation method, apparatus, device, medium, and product. By expanding the basic text content input by the user, more detailed extended text content can be obtained. Then, based on the number of text features included in the text content, content extraction is performed on the extended text content to obtain multiple key content segments and multiple transitional content segments. Thus, based on the extended text content, multiple key content segments, and multiple transitional content segments, multiple key video frames, multiple key video segments, and multiple transitional video segments are generated using a video generation model. Based on these multiple key video frames, multiple key video segments, and multiple transitional video segments, a target video corresponding to the basic text content can be generated. Based on this, this application extracts content from the extended text content to obtain multiple key content segments and multiple transitional content segments, thereby generating corresponding short-duration key video segments and transitional video segments based on short text-form key content segments and transitional content segments. Finally, the multiple key video segments and multiple transitional video segments are spliced ​​together to generate a longer target video corresponding to the basic text content. This improves the efficiency of video generation based on text content and enhances the quality of the generated video content, making the video content coherent and consistent. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is one of the flowcharts illustrating the video generation method provided by this invention.

[0024] Figure 2 This is the second flowchart illustrating the video generation method provided by this invention.

[0025] Figure 3 This is the third flowchart of the video generation method provided by the present invention.

[0026] Figure 4 This is the fourth flowchart of the video generation method provided by the present invention.

[0027] Figure 5 This is the fifth flowchart illustrating the video generation method provided by the present invention.

[0028] Figure 6 This is the sixth flowchart of the video generation method provided by the present invention.

[0029] Figure 7This is a schematic diagram of the video generation device provided by the present invention.

[0030] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] Currently, long video generation methods can be mainly divided into three categories: autoregressive generation, keyframe-driven generation, and video continuation generation. Autoregressive generation is based on a general video generation model, using the output frames of the previous video block as input conditions for the next, generating and stitching together a long video block by block; essentially, it extends the video length through serial continuation. Keyframe-driven generation first generates key video frames (single static images) based on text descriptions, then uses adjacent key video frames as conditions to generate intermediate transitional video frames through image-generated video, first and last video frames, or inter-frame interpolation methods, ultimately stitching them together to form a long video. Video continuation generation initializes the first video segment and then continuously extends the video content by using the latest generated last few frames as constraints; the generation process proceeds linearly and serially.

[0033] However, these methods have drawbacks. Autoregressive generation and video continuation are limited by sequential dependencies, resulting in extremely long generation times for minute-long videos. Furthermore, the cumulative effect of errors leads to a significant decrease in content consistency and video quality in the latter half of the video. While keyframe-driven generation can improve efficiency by generating segments in parallel, it relies solely on the constraints of static keyframes and cannot convey dynamic information such as motion rate and trajectory. This results in differences in motion frequency and amplitude between adjacent segments, making it prone to stuttering and abrupt transitions after splicing.

[0034] Therefore, current long video generation technology faces two major technical bottlenecks: First, the contradiction between video generation efficiency and video length. Existing methods mostly rely on serial generation logic, and the computational load increases linearly or exponentially with video length, making it difficult to meet the rapid generation requirements of minute-long videos. Second, the contradiction between dynamic coherence and content consistency. Most solutions use static keyframes or isolated video blocks as constraint units, which cannot convey dynamic information such as action rate and motion trajectory, leading to problems such as stuttering and semantic drift in the spliced ​​video.

[0035] To address the aforementioned shortcomings, from a technical perspective, it breaks through the bottlenecks in efficiency and coherence of long video generation, and can expand the capabilities of generative AI such as diffusion models in the temporal dimension; from an application perspective, it can provide core technical support for scenarios such as film and television creation, education and training, and digital human live streaming, and promote the upgrade of video production from "fragmented short videos" to "scenario-based long videos".

[0036] To address this, this application proposes a long video generation method based on key video segments. By pre-generating key video segments and using the first and last video segments as conditions to generate intermediate videos, the long videos are spliced ​​together, aiming to simultaneously solve the problems of efficiency and coherence in long video generation.

[0037] The following is combined Figures 1 to 8 This application describes the video generation method, apparatus, equipment, medium, and product provided.

[0038] Figure 1 This is one of the flowcharts illustrating the video generation method provided by the present invention, such as... Figure 1 As shown, the method includes the following: Step 101: Expand the basic text content input by the user to obtain expanded text content.

[0039] Step 102: Extract content from the extended text to obtain multiple key content fragments and multiple transition content fragments.

[0040] Among them, the number of text features included in the key content fragments is greater than the preset number, and the transition content fragments are the content fragments between two adjacent key content fragments.

[0041] Step 103: Based on the extended text content, multiple key content fragments, and multiple transition content fragments, generate multiple key video frames, multiple key video fragments, and multiple transition video fragments using a video generation model.

[0042] Step 104: Generate the target video corresponding to the basic text content based on multiple key video frames, multiple key video segments, and multiple transition video segments.

[0043] In this embodiment, when a user needs to obtain a video, the user can independently write descriptive text (i.e., basic text content) describing the video content to be obtained. The written descriptive text is then input into an electronic device, which can generate a corresponding video (i.e., the target video) based on the user-input descriptive text. The generated video content is consistent with the user-input descriptive text content, achieving the effect of text-based video generation (i.e., text-generated video).

[0044] In one possible implementation, after the user inputs basic text content, during the process of generating a video based on this input, the electronic device can infer from the basic text content to expand it using the Qwen3-32B large language model, resulting in three fine-grained captions. These three fine-grained captions are structurally aligned with the captions included in the training data used to train the video generation model.

[0045] Among them, there are three fine-grained explanatory texts: extended text content, multiple key content segments, and multiple transition content segments.

[0046] Specifically, the basic text input by the user is first expanded to obtain expanded text content. Then, based on the number of text features included in each sentence of the expanded text content, key content fragments with a number of text features greater than a preset number are extracted from the expanded text content.

[0047] Furthermore, content fragments not extracted from the extended text content are identified as transitional content fragments.

[0048] Alternatively, based on the number of text features included in each statement of the extended text content, the extended text content is split, and the number of text features included in each of the split content segments is determined. Content segments with a number of text features greater than a preset number are identified as key content segments, while content segments with a number of text features less than or equal to the preset number are identified as transitional content segments.

[0049] It should be noted that the number of text features included in a sentence or content fragment represents its importance within the overall extended text content. The more text features included in a sentence or content fragment, the higher its importance within the overall extended text content. In other words, key content fragments are used to generate the main segments of the target video.

[0050] Thus, by extracting content from the extended text to obtain multiple key content segments and multiple transitional content segments, corresponding video segments can be generated based on each key content segment and each transitional content segment. Key content segments can generate key video segments, and transitional content segments can generate transitional video segments.

[0051] Finally, by splicing or merging the generated key video frames, key video segments, and transition video segments, the final complete video, which is the target video corresponding to the basic text content, can be obtained.

[0052] In this embodiment, by expanding the basic text content input by the user, more detailed expanded text content can be obtained. Then, based on the number of text features included in the text content, content extraction is performed on the expanded text content to obtain multiple key content segments and multiple transitional content segments. Thus, based on the expanded text content, multiple key content segments, and multiple transitional content segments, a video generation model generates multiple key video frames, multiple key video segments, and multiple transitional video segments respectively. Based on these multiple key video frames, multiple key video segments, and multiple transitional video segments, a target video corresponding to the basic text content can be generated. Therefore, this application extracts content from the expanded text content to obtain multiple key content segments and multiple transitional content segments, thereby generating corresponding short-duration key video segments and transitional video segments based on short text-form key content segments and transitional content segments. Finally, by splicing the multiple key video segments and multiple transitional video segments, a longer target video corresponding to the basic text content can be generated. This improves the efficiency of generating videos based on text content and enhances the quality of the generated video content, making the video content coherent and consistent.

[0053] In one possible implementation, the video generation model includes: a key video frame generation model, a key video segment generation model, and a transition video segment generation model.

[0054] Figure 2 This is the second flowchart illustrating the video generation method provided by the present invention, as shown below. Figure 2 As shown, the method includes the following: Step 201: Expand the basic text content input by the user to obtain expanded text content.

[0055] Step 202: Extract content from the extended text to obtain multiple key content fragments and multiple transition content fragments.

[0056] Step 203: Input the extended text content and multiple key content fragments into the key video frame generation model to generate multiple key video frames.

[0057] One key video frame corresponds to one key content segment, and the content included in multiple key video frames satisfies spatial consistency.

[0058] Step 204: Input multiple key video frames and multiple key content segments into the key video segment generation model to generate multiple key video segments.

[0059] One key video frame corresponds to one key video segment, and the content of multiple key video segments satisfies spatial consistency and dynamic consistency.

[0060] Step 205: Input multiple key video clips and multiple transition content clips into the transition video clip generation model to generate multiple transition video clips.

[0061] Step 206: Based on multiple key video frames, multiple key video segments, and multiple transition video segments, generate the target video corresponding to the basic text content.

[0062] In one possible implementation, after extracting content from the extended text to obtain multiple key content segments and multiple transitional content segments, the extended text content and multiple key content segments can be first input into the key video frame generation model, thereby generating a corresponding key video frame based on each key content segment.

[0063] It should be noted that, since the key video frame generation model also inputs extended text content when generating corresponding key video frames based on each key content segment, the spatial consistency among the generated key video frames can be guaranteed by using the extended text content.

[0064] It should be noted that spatial consistency among multiple key video frames can be understood as the attributes, relationships, or structures between different locations or representations in space conforming to the geometric and semantic constraints of the real world. Essentially, it is the coordination and verifiability of spatial relationships, meaning that the spatial position, shape, and semantics of the same object or feature remain consistent across different viewpoints and scales, satisfying physical or geometric laws. For example, multiple key video frames may have consistent image backgrounds or depict the same scene, or the objects included in multiple key video frames may have consistent appearances and sizes.

[0065] In one possible implementation, after generating multiple key video frames based on multiple key content segments, the multiple key video frames and multiple key content segments can be further input into the key video segment generation model in parallel, thereby generating the corresponding key video segment based on a key video frame and the corresponding key content segment.

[0066] It is understandable that when the key video segment generation model generates a corresponding key video segment based on a key video frame and a corresponding key content segment, it can expand the key video frame based on the content expressed by the key content segment to generate video frames whose content is consistent with the key video frame, thereby obtaining the corresponding key video segment.

[0067] It should be noted that since the content of multiple key video frames satisfies spatial consistency, the content of the generated multiple key video segments also satisfies spatial consistency. Furthermore, since the content of multiple key content segments is continuous during the generation of key video segments, the content of the generated multiple key video segments also satisfies dynamic consistency.

[0068] Dynamic consistency can be understood as the consistency between the state, behavior, or output characteristics and the preset rules, physical constraints, or goals during the temporal evolution process. Its core is to describe the stability and coordination of the actions of the same object across multiple key video segments over time. For example, the movement of the same object in a key video segment should be coherent and natural, without any abrupt changes in position.

[0069] In one possible implementation, after generating multiple key video segments, the multiple key video segments and multiple transition content segments can be further input into the transition video segment generation model. Thus, for two adjacent key video segments, a transition video segment between the two adjacent key video segments can be generated based on the corresponding transition content segments between the two adjacent key video segments.

[0070] Specifically, based on the last frame of the previous key video segment and the first frame of the next key video segment, and combined with the corresponding transition content segment, video frames can be added between the last frame of the previous key video segment and the first frame of the next key video segment, thereby generating a transition video segment that makes the previous key video segment and the next key video segment continuous.

[0071] Finally, the generated key video frames, key video segments, and transition video segments are spliced ​​together in chronological order to generate the target video corresponding to the basic text content.

[0072] In this embodiment, a key video frame generation model is used to generate multiple key video frames using extended text content and multiple key content fragments. Then, using these generated key video frames and corresponding key content fragments, a key video fragment generation model can be used to generate multiple key video fragments. Next, using these generated key video fragments and multiple transition content fragments, a transition video fragment generation model can be used to generate transition video fragments between any two adjacent key video fragments. Finally, by integrating the generated key video frames, key video fragments, and transition video fragments, a complete target video corresponding to the basic text content can be generated. Thus, by splitting the extended text content into text fragments, and then generating and integrating these text fragments to create a complete video, the efficiency of generating videos based on text content can be improved. Furthermore, the constraint of the complete extended text content ensures that the video content is coherent and consistent, improving the quality of the generated video content.

[0073] Figure 3 This is the third flowchart of the video generation method provided by the present invention, as shown below. Figure 3As shown, the video generation model is trained in the following way: Step 301: Construct the training dataset.

[0074] The training dataset includes multiple sets of training data. Each set of training data includes: training video, complete video description text of the training video, multiple key training video segments, key video description text of each key training video segment, multiple transitional training video segments, and transitional video description text of each transitional training video segment. The multiple key training video segments and multiple transitional training video segments are video segments extracted from the training video.

[0075] Step 302: Based on the diffusion process of the preset diffusion model and the training dataset, train the preset model to obtain the video generation model.

[0076] In one possible implementation, it is necessary to pre-establish a diffusion process based on Denoising Diffusion Probabilistic Models (DDPM) and construct a pre-defined model.

[0077] The preset models include: a preset key video frame generation model, a preset key video segment generation model, and a preset transition video segment generation model. The preset transition video segment generation model is used to generate video between the first and last video segments based on the constraints of the first and last video segments.

[0078] In one possible implementation, the constructed training dataset can be... .in, It is the total number of training videos included in the training dataset. The sequence number of each training video, It is the first The complete video description text corresponding to each training video is used to describe the complete plot, main features and scene background of the training video; Indicates the first One training video; It is the first The total number of multiple key training video segments extracted from a training video. The sequence number of each key training video segment; Indicates the first The first training video Each training key video segment can include 9 frames of images; Indicates the first The first training video Key training video clips The corresponding key video description text is used to describe the key video segments used in training. The main actions, postures, and local scenes within the content; Indicates the first The first training video Key training video clips With the +1 key training video clip The transition video description text corresponding to the training transition video segments describes the action connection logic and scene transition relationship between the two key training video segments.

[0079] Thus, based on the diffusion process of DDPM and the training dataset, the preset key video frame generation model, the preset key video segment generation model, and the preset transition video segment generation model are trained to obtain the trained key video frame generation model, key video segment generation model, and transition video segment generation model, thereby constituting the video generation model.

[0080] In this embodiment, a training dataset is constructed. This dataset includes training videos, complete video description text for the training videos, multiple training key video segments, key video description text for each training key video segment, multiple training transition video segments, and transition video description text for each training transition video segment. A preset model is then trained to generate models for key video frames, key video segments, and transition video segments. Using these models, when generating videos based on text content, a complete video corresponding to the text content can be generated by generating key video frames, key video segments, and transition video segments. This improves the efficiency of generating videos based on text content and ensures coherence and consistency in the video content, thus enhancing the quality of the generated video.

[0081] Figure 4 This is the fourth flowchart of the video generation method provided by the present invention, as shown below. Figure 4 As shown, the video generation model is trained in the following way: Step 401: Construct the training dataset.

[0082] Step 402: Identify multiple key training video frames from the training videos.

[0083] Step 403: For each training key video frame in the multiple training key video frames, determine the N video frames adjacent to each training key video frame from the training video.

[0084] Step 404: Determine the video segment consisting of each training key video frame and its N adjacent video frames as the training key video segment.

[0085] One training key video frame corresponds to one training key video segment.

[0086] Step 405: Determine the video segment consisting of video frames between two adjacent training key video segments as the training transition video segment.

[0087] Step 406: Based on the diffusion process of the preset diffusion model and the training dataset, train the preset model to obtain the video generation model.

[0088] In one possible implementation, for each set of training data in multiple sets of training data, it is necessary to first analyze the training videos included in each set of training data in order to determine multiple key training video frames.

[0089] Furthermore, based on each training key video frame, N video frames adjacent to each training key video frame are determined from the training video, and the video segment formed by each training key video frame and the N adjacent video frames is determined as the training key video segment.

[0090] Specifically, each training key video segment can extract multiple training key video frames from the training video using the Keye-VL-1.5 multimodal model. Then, based on each training key video frame, N video frames adjacent to each training key video frame are determined to constitute the training key video segment.

[0091] For example, based on each training key video frame, the N-1 adjacent video frames after (or before) each training key video frame can be determined, and the training key video frame and the N-1 video frames after (or before) it can be combined to form a training key video segment.

[0092] Alternatively, based on each training key video frame, determine the M adjacent video frames after each training key video frame and the P adjacent video frames before each training key video frame. The training key video frame, together with the M video frames after it and the P video frames before it, constitutes a training key video segment, where M+P=N-1.

[0093] In one possible implementation, if there are fewer than N-1 video frames remaining after (before) training the key video frames, video frames can be determined from before (after) training the key video frames to make up N-1 video frames to form the training key video segment.

[0094] Furthermore, based on the determined key training video segments, a UniVL model is used to determine the key video description text for each key training video segment from the complete video description text of the training videos. The video segments consisting of video frames between two adjacent key training video segments are then identified as transitional training video segments. Additionally, transitional video description text for each transitional training video segment is determined from the complete video description text of the training videos.

[0095] Finally, based on this data and the diffusion process of the preset diffusion model, the preset model is trained to obtain the final trained video generation model.

[0096] In this embodiment, a training dataset is constructed by identifying multiple key training video frames from the training video, then progressively determining key training video segments and transitional training video segments, and extracting complete video description text from these video segments. The trained video generation model is then trained using this training dataset.

[0097] In one possible implementation, the preset model includes a preset key video frame generation model.

[0098] Figure 5 This is the fifth flowchart illustrating the video generation method provided by the present invention, as shown below. Figure 5 As shown, the video generation model is trained in the following way: Step 501: Construct the training dataset.

[0099] Step 502: For each set of training data in the multiple sets of training data, determine the training key video frame corresponding to each training key video segment in the multiple training key video segments, and obtain multiple training key video frames.

[0100] Step 503: Perform video encoding on multiple training key video frames and extract video features from multiple training key video frames.

[0101] Step 504: Add noise to the video features of multiple training key video frames to obtain Gaussian noise.

[0102] Step 505: Perform text encoding on the complete video description text and the key video description text, and extract the complete text features of the complete video description text and the key text features of the key video description text.

[0103] Step 506: Perform feature fusion on the complete text features and key text features to obtain fused text features.

[0104] Step 507: Based on Gaussian noise and fused text features, train the preset key video frame generation model to obtain the key video frame generation model.

[0105] In one possible implementation, during the training of the preset key video frame generation model, for each set of training data in multiple sets of training data, the complete video description text of the training video, the key video description text of each training key video segment, and the training key video frame corresponding to each training key video segment can be used as input to the preset key video frame generation model, thereby training the key video frame generation model.

[0106] First, the training key video frames corresponding to each training key video segment need to be input into a 3D VAE for video encoding and compression to extract the video features of each training key video frame. Then, Gaussian noise is added to the video features of each training key video frame. Where B represents the number of training key video frames, N represents the number of noise components (also known as the number of tokens), and C represents the number of channels (also known as the feature dimension of a token). This results in Gaussian noise. The segments are flattened into multiple tokens (Patch blocks) connected in parallel and input into a preset key video frame generation model.

[0107] Furthermore, the complete video description text and key video description text need to be input into the T5 text encoder to obtain the complete text features of the complete video description text. Key text features of key video description text Then, the complete text features and key text features are input into the text feature fusion module to process the key text features of the key video description text. Calculate its complete text features compared to the complete video description text. semantic similarity Then, by using a weighted formula, the complete text features and key text features are fused to obtain the fused text features. The specific calculation formula is shown in Formula 1.

[0108] Formula 1 in, This represents the positional encoding of key training video frames, avoiding confusion between the fusion features of different key training video frames.

[0109] In one possible implementation, Gaussian noise After consistent self-attention calculation, the parallel result of the fused text features is used for cross-attention calculation. After passing through N Transformer modules, the features are input into the decoder to obtain key video frames, which are then used to train the preset key video frame generation model to obtain the key video frame generation model.

[0110] It should be noted that Gaussian noise The difference between consistent self-attention calculation and the original self-attention in existing technologies is that the original self-attention calculates self-attention for the video features of each key video frame separately. However, consistent self-attention, in order to build subject consistency among different key video frames in the same video segment, needs to incorporate the video features of other key video frames into the self-attention calculation of the video features of the current key video frame.

[0111] Therefore, the video features of the target key video frames We need to perform weighted fusion of the video features of the other B-1 key video frames, and then use the key text features of the key video description text corresponding to each of the B-1 key video frames. Complete text features of the complete video description text semantic similarity Calculate the weighted percentage of each key video frame in the B-1 key video frames. Then, the fused text features of B-1 key video frames are obtained by fusing features using a weighted formula. The calculation formula is shown in Formula 2.

[0112] Formula 2 The fused text features of B-1 key video frames were obtained. Then, compare the video features with the target key video frames. Parallel processing yields the final fused text features. Use the final fused text features. After a linear mapping layer, consistent self-attention K and V are obtained, while Q still uses video features from the target key video frames. .

[0113] Finally, it should be noted that the key video frame generation model is derived from a modified and fine-tuned general image-based video model. The key video segment generation model is also based on the general image-based video model, with targeted fine-tuning applied to the multiple video frames included in the video segment. The transition video segment generation model is based on the general first and last frame video generation model, modifying the input conditions from first and last frames to first and last video segments, and correspondingly modifying the conditional mask to multiple frames, followed by targeted fine-tuning. In this way, all three models only require fine-tuning to meet the requirements, significantly reducing the training difficulty and the demand for training resources. The total data volume is also only in the millions.

[0114] Figure 6 This is the sixth flowchart of the video generation method provided by the present invention, as shown below. Figure 6 As shown, the above "Step 507, training the preset key video frame generation model based on Gaussian noise and fused text features to obtain the key video frame generation model" specifically includes the following: Step 601: When the number of multiple training key video segments is greater than the preset number, perform image encoding on the preset reference image and extract the image features of the preset reference image.

[0115] Step 602: Based on Gaussian noise, fused text features and image features, train the preset key video frame generation model to obtain the key video frame generation model.

[0116] It is understandable that, in the process of training a preset key video frame generation model based on Gaussian noise and fused text features, a preset reference image (i.e., a reference picture) can also be used as an optional input to the model.

[0117] It's important to note that the preset reference images will influence the appearance of the main subject and scene in the generated keyframes. The purpose of introducing preset reference images is that if generating very long videos (e.g., 10 minutes), the number of key video frames generated in a single inference iteration may not be sufficient. Therefore, a batch of key video frames can be generated first as preset reference images, and then more key video frames can be generated again based on these preset reference images. In this way, by using the previously generated key video frames as reference images for the current key video frames, the number of key video frames can be expanded, thereby expanding the number of key video segments.

[0118] Thus, depending on the requirements, a reference image is selected as the model input. The image features of the reference image, after being encoded by VAE, will be fused with the text features. The input model is processed in parallel to perform cross-attention model computation. It's important to note that an attention mask is introduced during cross-attention computation. That is, the image features corresponding to the k-th reference image are fused only with the features corresponding to the k-th reference image. Perform attention calculations.

[0119] This application's embodiments break down the long video generation task into a three-level process: "key video frame generation → key video segment generation → transition video segment generation." It innovatively proposes key video segments as the beginning and end constraints for both global consistency and local coherence. Each segment includes spatial features such as the main character's appearance and scene style, as well as temporal features such as action rate and motion trajectory. It also supports parallel generation, solving efficiency issues. By breaking down the long video generation into smaller parts, it overcomes the technical challenge of controlling traditional overall generation. This allows for faster generation through parallel acceleration, improving efficiency while ensuring dynamic consistency before and after splicing.

[0120] Furthermore, the proposed key video frame generation model ensures the consistency of the generated key video frames from both text and video feature dimensions, providing a solid foundation for subsequent video generation. This application significantly improves the generation effect and efficiency of long videos, theoretically enabling the generation of infinitely long videos. It solves the problems of dynamic stuttering and slow generation speed in the current field of long video generation. In practical use, long video generation is divided into three stages, facilitating timely correction of problems detected during generation and reducing rework costs.

[0121] The video generation apparatus provided by the present invention is described below. The video generation apparatus described below and the video generation method described above can be referred to in correspondence.

[0122] Figure 7 This is a schematic diagram of the video generation device provided by the present invention, as shown below. Figure 7 As shown, the video generation device includes the following modules: an expansion module 701, an extraction module 702, and a processing module 703; Extension module 701 is used to expand the basic text content input by the user to obtain extended text content; The extraction module 702 is used to extract content from the extended text content to obtain multiple key content fragments and multiple transition content fragments. The number of text features included in the key content fragments is greater than the preset number, and the transition content fragments are the content fragments between two adjacent key content fragments. Processing module 703 is used to generate multiple key video frames, multiple key video segments, and multiple transitional video segments based on extended text content, multiple key content segments, and multiple transitional content segments through a video generation model. The processing module 703 is also used to generate a target video corresponding to the basic text content based on multiple key video frames, multiple key video segments and multiple transition video segments.

[0123] According to a video generation apparatus provided by the present invention, the video generation model includes: a key video frame generation model, a key video segment generation model, and a transition video segment generation model; the processing module 703 is specifically used for: The extended text content and multiple key content fragments are input into the key video frame generation model to generate multiple key video frames. Each key video frame corresponds to a key content fragment, and the content included in multiple key video frames satisfies spatial consistency. Multiple key video frames and multiple key content segments are input into the key video segment generation model to generate multiple key video segments. One key video frame corresponds to one key video segment, and the content included in multiple key video segments satisfies spatial consistency and dynamic consistency. Multiple key video clips and multiple transition content clips are input into the transition video clip generation model to generate multiple transition video clips.

[0124] According to a video generation apparatus provided by the present invention, the processing module 703 is further used for: Construct a training dataset, which includes multiple sets of training data. Each set of training data includes: training videos, complete video description text of the training videos, multiple training key video clips, key video description text of each training key video clip, multiple training transition video clips, and transition video description text of each training transition video clip. The multiple training key video clips and multiple training transition video clips are video clips extracted from the training videos. Based on the diffusion process of the preset diffusion model and the training dataset, the preset model is trained to obtain the video generation model.

[0125] According to a video generation apparatus provided by the present invention, the processing module 703 is further used for: Identify multiple key training video frames from the training video; For each training key video frame in a plurality of training key video frames, determine N video frames that are adjacent to each training key video frame from the training video. Each training key video frame and its N adjacent video frames form a video segment, which is then defined as a training key video segment. One training key video frame corresponds to one training key video segment. The video segment consisting of video frames between two adjacent key training video segments is identified as the training transition video segment.

[0126] According to a video generation apparatus provided by the present invention, the preset model includes a preset key video frame generation model; the processing module 703 is specifically used for: For each set of training data in multiple sets of training data, determine the training key video frame corresponding to each training key video segment in multiple training key video segments, and obtain multiple training key video frames. Video encoding is performed on multiple key training video frames to extract video features from the multiple key training video frames; Noise is added to the video features of multiple key training video frames to obtain Gaussian noise; Text encoding is performed on the complete video description text and the key video description text to extract the complete text features of the complete video description text and the key text features of the key video description text. The complete text features and key text features are fused to obtain the fused text features; Based on Gaussian noise and fused text features, a preset key video frame generation model is trained to obtain the key video frame generation model.

[0127] According to the video generation apparatus provided by the present invention, the processing module 703 is specifically used for: When the number of multiple training key video segments exceeds the preset number, image encoding is performed on the preset reference image to extract the image features of the preset reference image; Based on Gaussian noise, fused text features and image features, a preset key video frame generation model is trained to obtain the key video frame generation model.

[0128] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a video generation method, which includes: expanding the basic text content input by the user to obtain expanded text content; extracting content from the expanded text content to obtain multiple key content segments and multiple transitional content segments, wherein the number of text features included in the key content segments is greater than a preset number, and the transitional content segments are content segments between two adjacent key content segments; generating multiple key video frames, multiple key video segments, and multiple transitional video segments through a video generation model based on the expanded text content, the multiple key content segments, and the multiple transitional content segments; and generating a target video corresponding to the basic text content based on the multiple key video frames, the multiple key video segments, and the multiple transitional video segments.

[0129] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0130] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video generation method provided by the above methods. The method includes: expanding the basic text content input by the user to obtain expanded text content; extracting content from the expanded text content to obtain multiple key content segments and multiple transitional content segments, wherein the number of text features included in the key content segments is greater than a preset number, and the transitional content segments are content segments between two adjacent key content segments; generating multiple key video frames, multiple key video segments, and multiple transitional video segments through a video generation model based on the expanded text content, the multiple key content segments, and the multiple transitional content segments; and generating a target video corresponding to the basic text content based on the multiple key video frames, the multiple key video segments, and the multiple transitional video segments.

[0131] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the video generation method provided by the above methods. The method includes: expanding basic text content input by a user to obtain expanded text content; extracting content from the expanded text content to obtain multiple key content segments and multiple transitional content segments, wherein the number of text features included in the key content segments is greater than a preset number, and the transitional content segments are content segments between two adjacent key content segments; generating multiple key video frames, multiple key video segments, and multiple transitional video segments through a video generation model based on the expanded text content, the multiple key content segments, and the multiple transitional content segments; and generating a target video corresponding to the basic text content based on the multiple key video frames, the multiple key video segments, and the multiple transitional video segments.

[0132] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video generation method, characterized in that, include: Expand the basic text content input by the user to obtain expanded text content; Content extraction is performed on the extended text content to obtain multiple key content segments and multiple transitional content segments. The number of text features included in the key content segments is greater than a preset number, and the transitional content segments are content segments between two adjacent key content segments. Based on the extended text content, the multiple key content segments, and the multiple transition content segments, multiple key video frames, multiple key video segments, and multiple transition video segments are generated through a video generation model. Based on the multiple key video frames, the multiple key video segments, and the multiple transition video segments, a target video corresponding to the basic text content is generated.

2. The video generation method according to claim 1, characterized in that, The video generation model includes: a key video frame generation model, a key video segment generation model, and a transition video segment generation model; The process of generating multiple key video frames, multiple key video segments, and multiple transitional video segments based on the extended text content, the multiple key content segments, and the multiple transitional content segments using a video generation model includes: The extended text content and the multiple key content segments are input into the key video frame generation model to generate the multiple key video frames. Each key video frame corresponds to a key content segment, and the content included in the multiple key video frames satisfies spatial consistency. The multiple key video frames and the multiple key content segments are input into the key video segment generation model to generate the multiple key video segments. One key video frame corresponds to one key video segment. The content included in the multiple key video segments satisfies the spatial consistency and dynamic consistency. The multiple key video segments and the multiple transition content segments are input into the transition video segment generation model to generate the multiple transition video segments.

3. The video generation method according to claim 1 or 2, characterized in that, The video generation model was trained in the following way: Construct a training dataset, which includes multiple sets of training data. Each set of training data includes: a training video, a complete video description text of the training video, multiple key training video segments, key video description text of each key training video segment, multiple transitional training video segments, and transitional video description text of each transitional training video segment. The multiple key training video segments and the multiple transitional training video segments are video segments extracted from the training video. Based on the diffusion process of the preset diffusion model and the training dataset, the preset model is trained to obtain the video generation model.

4. The video generation method according to claim 3, characterized in that, The method further includes: Multiple key training video frames are identified from the training video; For each of the plurality of training key video frames, determine N video frames that are adjacent to each training key video frame from the training video. Each training key video frame and its N adjacent video frames constitute a video segment, which is then defined as a training key video segment. One training key video frame corresponds to one training key video segment. The video segment consisting of video frames between two adjacent key training video segments is identified as the training transition video segment.

5. The video generation method according to claim 3, characterized in that, The preset model includes a preset key video frame generation model; The diffusion process based on the preset diffusion model and the training dataset, training the preset model to obtain the video generation model, includes: For each set of training data in the plurality of training data, determine the training key video frame corresponding to each training key video segment in the plurality of training key video segments, and obtain a plurality of training key video frames. Video encoding is performed on multiple key training video frames, and video features of the multiple key training video frames are extracted. Noise is added to the video features of the multiple training key video frames to obtain Gaussian noise; The complete video description text and the key video description text are text encoded, and the complete text features of the complete video description text and the key text features of the key video description text are extracted. The complete text features and the key text features are fused to obtain fused text features; Based on the Gaussian noise and the fused text features, the preset key video frame generation model is trained to obtain the key video frame generation model.

6. The video generation method according to claim 5, characterized in that, The step of training the preset key video frame generation model based on the Gaussian noise and the fused text features to obtain the key video frame generation model includes: When the number of multiple training key video segments is greater than a preset number, image encoding is performed on a preset reference image to extract image features from the preset reference image; The preset key video frame generation model is trained based on the Gaussian noise, the fused text features, and the image features to obtain the key video frame generation model.

7. A video generation apparatus, characterized in that, include: Extension module, extraction module, and processing module; The extension module is used to expand the basic text content input by the user to obtain extended text content; The extraction module is used to extract content from the extended text content to obtain multiple key content segments and multiple transition content segments. The number of text features included in the key content segments is greater than a preset number, and the transition content segments are content segments between two adjacent key content segments. The processing module is used to generate multiple key video frames, multiple key video segments, and multiple transition video segments based on the extended text content, the multiple key content segments, and the multiple transition content segments, using a video generation model. The processing module is further configured to generate a target video corresponding to the basic text content based on the plurality of key video frames, the plurality of key video segments, and the plurality of transition video segments.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the video generation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the video generation method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the video generation method as described in any one of claims 1 to 6.