Dynamic video generation method and system based on multi-modal control
By employing multimodal fusion control and temporal dynamic control mechanisms, the problem of video generation deviation under single-modal control was solved, achieving high-quality and controllable dynamic video generation and improving the accuracy and dynamic expressiveness of video generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-14
- Publication Date
- 2026-03-06
AI Technical Summary
In existing dynamic video generation technologies, single-modal control leads to deviations between the generated results and user intentions, insufficient controllability, and difficulty in effectively integrating multimodal information, resulting in insufficient quality and dynamic expressiveness of the generated video.
Employing a multimodal fusion control and refined temporal dynamic control mechanism, high-quality videos are generated by extracting features using Transformer and convolutional neural networks through text descriptions, reference images, and other modal inputs, combined with cross-attention mechanisms and generative adversarial networks.
It achieves deep fusion of multimodal information, and the generated video content highly meets user requirements, improving the controllability and dynamic smoothness of the generated results, and enhancing the quality and flexibility of video generation.
Smart Images
Figure CN121619476A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method and system for generating dynamic video based on multimodal control. Background Technology
[0002] In the field of existing dynamic video generation technology, the main methods often rely on a single modality of control input, such as using only text descriptions or only static reference images. These methods have significant limitations in video generation. Relying solely on text descriptions makes it difficult to accurately control the specific appearance, spatial layout, and fine motion details of objects in the video, resulting in deviations between the generated results and the user's intentions, and insufficient controllability. On the other hand, relying solely on static image input makes it difficult to effectively guide the dynamic evolution of video content over time, and it is impossible to generate complex and expected temporal actions or scene transitions, thus limiting the dynamic expressiveness of the video.
[0003] On the other hand, existing video generation system architectures have shortcomings in handling multimodal input coordination and temporal dynamic control. Many systems fail to effectively integrate control information from different modalities, making it difficult to achieve deep fusion and unified understanding of multi-source information. This results in generated videos that cannot fully reflect the complex intentions expressed by users in various ways. Consequently, it is difficult for systems to generate video content that is both faithful to user multimodal instructions and possesses natural and smooth dynamic effects, posing challenges in both flexibility and generation quality. Summary of the Invention
[0004] The purpose of this invention is to provide a dynamic video generation method and system based on multimodal control. By introducing multimodal fusion control and a refined temporal dynamic control mechanism, it effectively solves the technical problems existing in the prior art, such as the large limitations of single-modal control, insufficient temporal dynamic control, low quality and controllability of generated video, and the difficulty of the system in effectively integrating multi-source information and achieving natural and smooth dynamic effects.
[0005] To achieve the aforementioned technical objectives and effects, this invention discloses a dynamic video generation method based on multimodal control, the method comprising at least the following steps:
[0006] Receive multimodal control input provided by the user, wherein the multimodal control input includes at least text description input and reference image input;
[0007] Using a pre-trained feature extraction network, features are extracted from the text description input and the reference image input respectively to obtain text feature vectors and image feature vectors;
[0008] The text feature vector and the image feature vector are input into the multimodal fusion module for feature fusion to generate fused multimodal control features;
[0009] The fused multimodal control features are input into the timing control module, which generates dynamically changing timing control signals based on the multimodal control features.
[0010] The timing control signal is input to the video frame generation module to generate a dynamic video sequence that meets the multimodal control input requirements.
[0011] The multimodal control input further includes at least one of the following:
[0012] Audio input, sketch input, motion capture data input, and 3D model input.
[0013] The feature extraction network includes:
[0014] A text encoder for extracting text feature vectors, the text encoder being based on the Transformer architecture;
[0015] An image encoder for extracting image feature vectors, the image encoder being based on a convolutional neural network or a visual Transformer architecture.
[0016] The multimodal fusion module uses a cross-attention mechanism or a feature concatenation followed by a fully connected layer to fuse the text feature vector and the image feature vector.
[0017] The timing control module includes a motion trajectory prediction submodule and a dynamic rhythm control submodule, which are used to predict the motion path of objects in the video and the rhythm of changes in the video scene, respectively.
[0018] The video frame generation module is built on a generative adversarial network or diffusion model architecture.
[0019] This invention also discloses a dynamic video generation system based on multimodal control, wherein the dynamic video generation system is based on the above-described dynamic video generation method and includes at least:
[0020] Multimodal input interface module: used to receive multimodal control input provided by the user, wherein the multimodal control input includes at least text description input and reference image input;
[0021] Feature extraction module: connected to the multimodal input interface module, used to extract features from the received text description input and reference image input using a pre-trained feature extraction network, and generate text feature vectors and image feature vectors respectively;
[0022] Multimodal fusion module: connected to the feature extraction module, used to receive the text feature vector and image feature vector, and fuse the two features to generate fused multimodal control features;
[0023] Timing control module: connected to the multimodal fusion module, used to receive the fused multimodal control features and generate dynamically changing timing control signals based on the features;
[0024] Video frame generation module: connected to the timing control module, used to receive the timing control signal and generate a dynamic video sequence that meets the multimodal control input requirements based on the signal.
[0025] The multimodal input interface module is further configured to receive at least one of the following inputs: audio input, sketch input, motion capture data input, and 3D model input.
[0026] Preferably, the feature extraction module, multimodal fusion module, timing control module, and video frame generation module are deployed on cloud servers, edge computing devices, and / or user terminal devices.
[0027] Preferably, the timing control module specifically includes:
[0028] Motion trajectory prediction submodule: used to predict the motion path of a target object in a video in consecutive video frames based on the fused multimodal control features; the motion path includes at least the object's position change sequence, motion direction or posture evolution information;
[0029] Dynamic rhythm control submodule: used to predict and control the rhythm of video scene changes based on the fused multimodal control features; the rhythm includes at least the rate of change of scene elements, the frequency of shot switching, or the duration of a specific visual effect;
[0030] The timing control signal includes the motion path information generated by the motion trajectory prediction submodule and the rhythm information generated by the dynamic rhythm control submodule. The timing control signal is used to guide the video frame generation module to dynamically adjust the position, posture and / or presentation state of the object and / or scene elements when generating each video frame, so as to achieve a coherent dynamic effect that conforms to the predicted path and rhythm.
[0031] The present invention has the following beneficial effects:
[0032] 1. This invention overcomes the inherent limitations of single-modal control by integrating complementary information from text descriptions, reference images, and other optional modalities. Text descriptions provide high-level semantic guidance and action intentions, while reference images precisely constrain the appearance, details, and initial spatial layout of objects. The deep fusion of multimodal information enables the generation system to more comprehensively and accurately understand the user's complex creative intentions, thereby generating video sequences that highly meet the user's requirements in terms of content theme, visual details, and spatial composition, significantly improving the controllability of the generated results and user satisfaction.
[0033] 2. This application innovatively introduces a dedicated timing control module, particularly its motion trajectory prediction submodule and dynamic rhythm control submodule. Based on fused multimodal control features, this module can proactively predict the fine motion path of the target object in consecutive frames and the rhythm of video scene changes. The generated timing control signals provide the video frame generation module with clear spatial motion guidance and temporal and spatial information, enabling it to dynamically adjust the object state and scene presentation during frame-by-frame generation. Attached Figure Description
[0034] Figure 1 This is a flowchart illustrating Embodiment 1 of the present invention. Detailed Implementation
[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments.
[0036] Example 1
[0037] like Figure 1 As shown, this embodiment provides a dynamic video generation method based on multimodal control, the method including the following steps:
[0038] Receive multimodal control input provided by the user, wherein the multimodal control input includes at least text description input and reference image input;
[0039] Using a pre-trained feature extraction network, features are extracted from the text description input and the reference image input respectively to obtain text feature vectors and image feature vectors;
[0040] The text feature vector and the image feature vector are input into the multimodal fusion module for feature fusion to generate fused multimodal control features;
[0041] The fused multimodal control features are input into the timing control module, which generates dynamically changing timing control signals based on the multimodal control features.
[0042] The timing control signal is input to the video frame generation module to generate a dynamic video sequence that meets the multimodal control input requirements.
[0043] To facilitate understanding of the design concept of this embodiment, the principle design of this embodiment is explained below:
[0044] To facilitate user input through various methods in guiding the generation of dynamic video, this implementation employs a multimodal control input method for basic data acquisition, including:
[0045] Text description input: Users describe the content, style, scene, and other requirements of the video using natural language. For example, a user can input "On a sunny beach, the waves are gently lapping the sand, and several sailboats are sailing in the distance."
[0046] Reference image input: An image provided by the user that is relevant to the desired video content, serving as a visual reference. For example, a user can provide a picture of a beach, allowing the generated video to visually resemble that image.
[0047] Other inputs:
[0048] Audio input: Users can provide background music or sound effects, and the video generation system can adjust the dynamic changes of the video according to the rhythm and content of the audio. For example, if the audio is a slow-paced sound of waves crashing on the beach or the cries of seagulls flying overhead, the dynamic changes of the video can be more gentle.
[0049] Sketch Input: Users can draw simple sketches to describe the scene layout or object positions in the video. For example, a user can draw a simple line drawing to represent the movement trajectory of a person in the video.
[0050] Motion capture data input: Users can record the motion data of people or objects using motion capture devices. This data can be used to generate precise movements of objects in videos. For example, recording a dancer's dance moves can then generate a video in which a virtual character dances the same moves.
[0051] 3D Model Input: Users can provide 3D models to generate 3D scenes or objects in the video. For example, providing a 3D model of a city can generate a flight-view video showcasing the city's entirety.
[0052] In this embodiment, useful feature vectors are extracted from multimodal inputs for subsequent fusion and processing. Different extraction methods are used for different input sources.
[0053] Text Encoder: A text encoder based on the Transformer architecture can effectively process text sequence data, capturing long-distance dependencies and semantic information within the text. The core of the Transformer architecture is the self-attention mechanism, which allows the model to consider information from other words in the text sequence while processing each word. The output of the text encoder is a text feature vector representing the semantic information described in the text.
[0054] For example: Suppose the input text is:
[0055] x = (x1, x2, ..., xn),
[0056] The output of the text encoder is: Text_Feature = Transformer(x);
[0057] Image encoders: Image encoders can be based on convolutional neural networks (CNNs) or visual Transformers architectures. CNNs extract local features and spatial hierarchical structures of images through convolutional and pooling layers, while visual Transformers capture global dependencies in images through self-attention mechanisms. The output of an image encoder is an image feature vector representing the visual content of the image.
[0058] For example: Assuming the input image is I, the output of the image encoder is:
[0059] Based on Convolutional Neural Networks (CNN):
[0060] Image_Feature = CNN(I);
[0061] Based on Vision Transformer:
[0062] Image_Feature=Vision_Transformer(I).
[0063] It should be noted that for other inputs, such as audio input, the audio input needs to be preprocessed first. For example, semantic voice (Asemantic) and non-semantic voice (Anon-semantic) need to be separated. For the semantically separated voice, it needs to be further converted into corresponding text, which is then used as an import item for text description input for further supplementary calculations.
[0064] Assuming the speech recognition system is ASR, the converted text is: T = ASR(Asemantic), then the input text is adjusted to x = (x1, x2, ..., xn, T); the text output is adjusted to: Text_Feature = Transformer(x1, x2, ..., xn, T);
[0065] For human voices that do not contain semantics, an audio encoder based on a convolutional neural network or Transformer architecture is used to extract audio feature vectors.
[0066] Based on Convolutional Neural Networks (CNN):
[0067] Audio_Feature=CNN(Anon-semantic);
[0068] Based on Transformer architecture:
[0069] Audio_Feature=Transformer(Anon-semantic).
[0070] The function of the multimodal fusion module is to fuse feature vectors from different modalities to generate a comprehensive multimodal control feature.
[0071] Cross-attention mechanism: The cross-attention mechanism allows feature vectors from different modalities to pay attention to each other, thereby better understanding the relationships between them. Text features can focus on relevant information in images and audio, image features can focus on relevant information in text and audio, and audio features can focus on relevant information in text and images. This mechanism can improve the accuracy and richness of fused features.
[0072] Formula expression:
[0073] Fusion_Feature = Cross_Attention(Text_Feature, Image_Feature, Audio_Feature); The concatenated features are followed by a fully connected layer.
[0074] Another fusion method involves directly concatenating text features, image features, and audio features, and then processing them through a fully connected layer, as shown in the formula:
[0075] Fusion_Feature=FC([Text_Feature; Image_Feature; Audio_Feature]);
[0076] Where [;;] represents feature splicing, and FC represents a fully connected layer.
[0077] The role of the timing control module is to generate dynamically changing timing control signals based on the fused multimodal control characteristics in order to control the dynamic changes of the video.
[0078] Motion trajectory prediction submodule: This submodule is responsible for predicting the motion path of objects in the video. For example, if the text description mentions "a person is running", this submodule can predict the person's motion trajectory based on text and image features, enabling the person in the generated video to move along a reasonable path.
[0079] Formula expression:
[0080] The fused feature is Fusion_Feature, and the output of the motion trajectory prediction submodule is:
[0081] Motion_Path=Motion_Predictor(Fusion_Feature);
[0082] Dynamic Rhythm Control Submodule: This submodule is responsible for controlling the rhythm of video scene changes. It adjusts the speed of scene transitions or the pace of object movements in the video based on the rhythm of semantic, image, and / or audio input.
[0083] Formula expression:
[0084] Dynamic_Rhythm=Rhythm_Controller(Fusion_Feature).
[0085] The function of the video frame generation module is to generate dynamic video sequences that meet the requirements of multimodal control input based on the timing control signals.
[0086] Generative Adversarial Network (GAN) architecture: GAN consists of a generator and a discriminator. The generator is responsible for generating video frames, and the discriminator is responsible for evaluating whether the generated video frames are realistic. Through adversarial training, the generator can generate high-quality video frames.
[0087] The formula states that the training objectives for the generator G and discriminator D can be expressed as:
[0088]
[0089] Where x is the real video frame, z is the noise input, and G(z) is the generated video frame.
[0090] Diffusion model architecture: The diffusion model is a generative model based on Markov chains that generates data by progressively removing noise. The diffusion model performs exceptionally well in generating high-quality images and videos.
[0091] The formula states that the generation process of the diffusion model can be represented as follows:
[0092]
[0093] Where ∈t is noise, αt is the diffusion coefficient, and the generated video frame x0 is obtained through a stepwise reverse diffusion process.
[0094] In summary, this embodiment proposes an innovative multimodal cooperative control dynamic video generation method. The core of this method lies in:
[0095] 1) Intelligent layered processing of audio input - using voiceprint separation technology to distinguish semantic human voice from background sound effects / music, and converting human voice into supplementary text commands through speech recognition;
[0096] 2) Use a pre-trained encoder to extract deep features from text, images, and non-semantic audio respectively;
[0097] 3) Cross-modal semantic alignment and feature unification are achieved through a multimodal fusion module;
[0098] 4) Based on the fusion feature-driven timing control module, motion trajectory and dynamic rhythm signals are generated;
[0099] 5) Finally, a high-quality dynamic video is synthesized using a conditional video generation model. Experiments show that this method significantly improves the accuracy of video content generation, dynamic smoothness, and multimodal intent matching.
[0100] It should be specifically noted that the specific formulas shown in this specification, such as attention mechanisms, fusion calculations, and generative model loss functions, are merely illustrative descriptions of implementation schemes. Any approach based on the same inventive concept but employing different feature extraction network structures, such as using graph neural networks to process 3D model inputs, alternative multimodal fusion strategies, such as gated fusion, tensor fusion, and temporal control algorithms, such as motion planning or video generation architectures based on reinforcement learning, such as neural radiation field extension models, as long as their core is to achieve dynamic video generation through multimodal collaborative control, should be considered to fall within the protection scope of this application. The embodiments of this invention are not limited to the specific technical details listed in the above embodiments. All technical solutions formed by equivalent substitutions or equivalent transformations are within the protection scope claimed in this application.
[0101] Example 2
[0102] This embodiment provides a dynamic video generation system based on multimodal control, which is constructed based on the method described in Embodiment 1. The system includes a multimodal input interface module, a feature extraction module, a multimodal fusion module, a timing control module, and a video frame generation module. The multimodal input interface module receives multimodal control input provided by the user. This input includes at least text description input and reference image input, and can be extended to support at least one of audio input, sketch input, motion capture data input, or 3D model input. The feature extraction module is connected to the multimodal input interface module and processes the text description input and reference image input through a pre-trained feature extraction network to generate corresponding text feature vectors and image feature vectors.
[0103] The multimodal fusion module receives text and image feature vectors from the feature extraction module. It achieves feature fusion using a cross-attention mechanism or by concatenating features followed by a fully connected layer, outputting the fused multimodal control features. The temporal control module connects to the multimodal fusion module and generates dynamically changing temporal control signals based on the fused features. This module includes a motion trajectory prediction submodule and a dynamic rhythm control submodule: the motion trajectory prediction submodule predicts the motion path of a target object in consecutive frames of the video, including position change sequences, motion direction, or posture evolution information; the dynamic rhythm control submodule predicts and controls rhythm parameters such as the rate of change of scene elements, shot switching frequency, or duration of specific visual effects. The temporal control signal integrates the above motion path and rhythm information to guide the video frame generation module in dynamically adjusting the object's position, posture, and scene element state during frame-by-frame generation.
[0104] The video frame generation module uses timing control signals to drive the generation process, outputting a dynamic video sequence that meets multimodal control requirements. Furthermore, to meet the needs of different user terminals, the feature extraction module, multimodal fusion module, timing control module, and video frame generation module can be deployed in any combination on cloud servers, edge computing devices, or user terminal devices to adapt to the computing needs of different application scenarios.
[0105] The dynamic video generation system provided in this embodiment achieves collaborative parsing and dynamic generation of multimodal inputs through modular design. The system transforms text, images, and extended inputs into unified temporal control signals, and combines a joint optimization mechanism of motion trajectory prediction and dynamic rhythm control to effectively improve the accurate mapping of cross-modal intentions and the coherence of dynamic effects. Its flexible deployment scheme supports adaptation to cloud, edge, and terminal devices, providing a scalable technical implementation path for controllable video generation.
[0106] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A dynamic video generation method based on multi-modal control, characterized in that, The method comprises at least the following steps: receiving a user-provided multi-modal control input, the multi-modal control input comprising at least a text description input and a reference image input; extracting features from the text description input and the reference image input respectively using a pre-trained feature extraction network to obtain a text feature vector and an image feature vector; inputting the text feature vector and the image feature vector into a multi-modal fusion module for feature fusion to generate a fused multi-modal control feature; inputting the fused multi-modal control feature into a time sequence control module, which generates a dynamically changing time sequence control signal according to the multi-modal control feature; inputting the time sequence control signal into a video frame generation module to generate a dynamic video sequence that meets the requirements of the multi-modal control input.
2. The dynamic video generation method based on multi-modal control of claim 1, wherein, The multi-modal control input further comprises at least one of: audio input, sketch input, motion capture data input, three-dimensional model input.
3. The dynamic video generation method based on multi-modal control of claim 1, wherein, The feature extraction network comprises: a text encoder for extracting a text feature vector, the text encoder being based on a Transformer architecture; an image encoder for extracting an image feature vector, the image encoder being based on a convolutional neural network or a visual Transformer architecture.
4. The dynamic video generation method based on multi-modal control of claim 1, wherein, The multi-modal fusion module uses a cross-attention mechanism or a feature splicing method followed by a fully connected layer to fuse the text feature vector and the image feature vector.
5. The dynamic video generation method based on multi-modal control of claim 1, wherein, The time sequence control module comprises a motion trajectory prediction submodule and a dynamic rhythm control submodule for predicting the motion path of an object in the video and the rhythm of the video scene change, respectively.
6. The dynamic video generation method based on multi-modal control of claim 1, wherein, The video frame generation module is based on a generative adversarial network or a diffusion model architecture.
7. A dynamic video generation system based on multi-modal control, characterized by, The dynamic video generation system is based on the dynamic video generation method of any one of claims 1-6 and comprises at least: a multi-modal input interface module for receiving a user-provided multi-modal control input, the multi-modal control input comprising at least a text description input and a reference image input; a feature extraction module connected to the multi-modal input interface module for extracting features from the received text description input and reference image input respectively using a pre-trained feature extraction network to generate a text feature vector and an image feature vector; a multi-modal fusion module connected to the feature extraction module for receiving the text feature vector and image feature vector and fusing the two features to generate a fused multi-modal control feature; a time sequence control module connected to the multi-modal fusion module for receiving the fused multi-modal control feature and generating a dynamically changing time sequence control signal according to the feature; a video frame generation module connected to the time sequence control module for receiving the time sequence control signal and generating a dynamic video sequence that meets the requirements of the multi-modal control input based on the signal.
8. A dynamic video generation system based on multi-modal control as claimed in claim 7, wherein, The multi-modal input interface module is further configured to receive at least one of the following inputs: audio input, sketch input, motion capture data input, three-dimensional model input.
9. The dynamic video generation system based on multi-modal control of claim 7, wherein, The feature extraction module, the multi-modal fusion module, the timing control module, and the video frame generation module are deployed on a cloud server, an edge computing device, and / or a user terminal device.
10. The dynamic video generation system based on multi-modal control of claim 7, wherein, The timing control module specifically includes: a motion trajectory prediction submodule for predicting a motion path of a target object in a continuous video frame based on the fused multi-modal control feature; the motion path at least includes a position change sequence, a motion direction, or posture evolution information of the object; a dynamic rhythm control submodule for predicting and controlling the rhythm of a video scene change based on the fused multi-modal control feature; the rhythm at least includes a rate of scene element change, a frequency of shot switching, or a duration of a specific visual effect; wherein the timing control signal contains the motion path information generated by the motion trajectory prediction submodule and the rhythm information generated by the dynamic rhythm control submodule; the timing control signal is used to guide the video frame generation module to dynamically adjust the position, posture, and / or presentation state of scene elements of the object when generating each frame of video, so as to achieve a coherent dynamic effect consistent with the predicted path and rhythm.