Method, apparatus and medium for generating motion enhanced video with temporal-aware stream matching

By introducing inter-frame temporal relationship constraints and motion priors into the video generation model, the dynamic correlation between video frames is explicitly learned, which solves the problems of discontinuous inter-frame motion and violation of physical laws in the existing technology, and improves the temporal coherence and physical rationality of the generated video.

CN122340333APending Publication Date: 2026-07-03TSINGHUA UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610490906.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-04-14
Publication Date
2026-07-03

AI Technical Summary

Technical Problem

Existing video generation models suffer from severe problems such as discontinuous motion between frames, jittery objects, and structural deformation when generating videos containing complex motion or fast movements, and the generated results often violate physical laws.

Method used

By introducing inter-frame temporal relationship constraints, the dynamic correlation and motion prior between video frames are learned. A temporal-aware stream matching method is used for video generation. Motion priors are explicitly embedded during model training, and the model parameters are updated using the difference between the predicted velocity vector and the true velocity vector.

Benefits of technology

It improves the inter-frame motion coherence and physical plausibility of the generated video, reduces jitter and structural breakage, and makes the generated video content more consistent with the physical laws of the real world, especially showing stronger stability in complex motion scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122340333A_ABST
    Figure CN122340333A_ABST
Patent Text Reader

Abstract

This application provides a method, apparatus, device, and medium for generating motion-enhanced videos using time-aware flow matching, relating to the field of video generation technology. The method includes: inputting sample videos and their descriptive text into a video generation base model; performing intra-frame modeling and inter-frame modeling with adjacent frames on the sample video frames, aiming to learn the distribution mapping from the initial noise vector to the sample video frames, to determine the noise vector associated with each time step; obtaining a predicted velocity vector through model processing; determining the true velocity vector based on the initial noise vector and the sample video frames; updating model parameters based on the difference between the predicted and true velocity vectors to obtain a trained model; and inputting the target descriptive text into the model to generate the corresponding video. This application enhances the model's ability to learn motion priors by explicitly introducing inter-frame temporal constraints, thereby improving the temporal coherence and physical plausibility of the generated video.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of video generation technology, and in particular to a method, apparatus, device and medium for generating motion-enhanced video using time-aware stream matching. Background Technology

[0002] Text-to-video generation is an important research direction in computer vision and multimedia generation. Current mainstream methods are mostly based on diffusion models or flow matching (FM) frameworks. Flow matching, in particular, demonstrates excellent performance in image generation tasks by constructing deterministic probabilistic paths from noise distributions to target data distributions. Inspired by this, the industry has widely attempted to directly extend flow matching to video generation tasks, treating it as a sequence generation problem of independent image frames, learning the mapping from noise to the target frame for each frame.

[0003] However, the aforementioned extensions have significant limitations in practical applications. Stream matching methods are essentially designed for image generation and lack the ability to specifically model the temporal structure unique to video data. When generating videos containing complex motion or fast movements, existing methods often produce video clips with inconsistent motion between frames, jittery objects or characters, and structural distortions. Furthermore, some generated results violate basic physical laws; for example, object trajectories are unreasonable, and lighting changes are abnormal. Therefore, improving the quality of generated videos output by video generation models remains a pressing technical problem to be solved in this field. Summary of the Invention

[0004] The purpose of this application is to provide a motion-enhanced video generation method, apparatus, device, and medium with time-aware stream matching, which can introduce inter-frame temporal relationship constraints when generating and modeling video data, learn the dynamic correlation and motion prior between video frames, thereby improving the temporal coherence and physical rationality of the generated video.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows: A first aspect of this application discloses a motion-enhanced video generation method using time-aware stream matching, the method comprising: The training samples for video generation are input into the text to be fine-tuned into the video generation base model. The training samples for video generation include: sample videos and descriptive text for the sample videos. The goal is to learn the distribution mapping between the i-th initial noise vector and the i-th sample video frame. Using the text-to-video generation base model to be fine-tuned, intra-frame modeling is performed on the i-th sample video frame, and inter-frame modeling is performed between the i-th sample video frame and its adjacent sample video frames. The noise vector associated with the i-th sample video frame at time step t is determined. Then, using the text-to-video generation base model to be fine-tuned, the noise vector associated with the i-th sample video frame at time step t and time step t are processed to obtain the predicted velocity vector associated with the i-th sample video frame at time step t. The value of time step t is 0 to 1; the value of i is 1 to n, where n represents the total number of frames in the sample video. Based on the vector corresponding to the i-th initial noise vector and the i-th sample video frame, determine the true velocity vector associated with the i-th sample video frame at time step t. Based on the difference between the predicted velocity vector associated with the i-th sample video frame at time step t and the true velocity vector associated with the i-th sample video frame at time step t, the model parameters of the text-to-video generation base model to be fine-tuned are updated to obtain the trained text-to-video generation model. Input the target description text into the trained text and then into the video generation model to obtain the generated video corresponding to the target description text.

[0006] Optionally, inter-frame modeling is performed between the i-th sample video frame and its neighboring sample video frames, including: According to the expression Inter-frame modeling is performed on the i-th sample video frame and the (i-1)-th sample video frame to learn the temporal relationship between the i-th sample video frame and the (i-1)-th sample video frame. According to the expression Inter-frame modeling is performed on the i-th sample video frame and the (i+1)-th sample video frame to learn the temporal relationship between the i-th sample video frame and the (i+1)-th sample video frame. Intra-frame modeling is performed on the i-th sample video frame, including: According to the expression Intra-frame modeling is performed on the i-th sample video frame in the sample video. in, Indicates an indicator function; Represents the Euclidean distance function; This represents the noise vector associated with the i-th sample video frame at time step t. This represents the vector corresponding to the i-th sample video frame; This represents the noise vector associated with the (i-1)th sample video frame at time step t. This represents the vector corresponding to the (i-1)th sample video frame; This represents the noise vector associated with the (i+1)th sample video frame at time step t. This represents the vector corresponding to the (i+1)th sample video frame.

[0007] Optionally, determining the noise vector associated with the i-th sample video frame at time step t includes: Solve the following equation to determine the noise vector associated with the i-th sample video frame at time step t:

[0008] in, A real number greater than or equal to 4; ,and Let represent the Euclidean distance function, and let the direction of the true velocity vector associated with the i-th sample video frame at time step t be: ; This represents the i-th initial noise vector. This represents the (i-1)th initial noise vector. This represents the (i+1)th initial noise vector.

[0009] Optionally, based on the vector corresponding to the i-th initial noise vector and the i-th sample video frame, the true velocity vector associated with the i-th sample video frame at time step t is determined, including at least: Based on the vector corresponding to the i-th sample video frame and the i-th initial noise vector, determine the direction of the true velocity vector associated with the i-th sample video frame at time step t; The noise vector associated with the i-th sample video frame at time step t is processed to obtain the predicted velocity vector associated with the i-th sample video frame at time step t, which includes at least: The noise vector associated with the i-th sample video frame at time step t is processed to obtain the direction of the predicted velocity vector associated with the i-th sample video frame at time step t. Based on the difference between the predicted velocity vector associated with the i-th sample video frame at time step t and the true velocity vector associated with the i-th sample video frame at time step t, the model parameters of the text-to-video generation base model to be fine-tuned are updated, including at least: The direction loss value is obtained by comparing the direction of the predicted velocity vector associated with the i-th sample video frame at time step t with the direction of the true velocity vector associated with the i-th sample video frame at time step t. The model parameters of the text-to-video generation base model to be fine-tuned are updated based at least on the directional loss value.

[0010] Optionally, determining the true velocity vector associated with the i-th sample video frame at time step t based on the vector corresponding to the i-th initial noise vector and the i-th sample video frame further includes: According to the expression Determine the strengths of the first module; among them, A real number greater than or equal to 4; Represents the Euclidean distance function; This represents the i-th initial noise vector. This represents the vector corresponding to the i-th sample video frame; According to the expression Determine the second module's strengths; This represents the (i-1)th initial noise vector. This represents the vector corresponding to the (i-1)th sample video frame; Indicates an indicator function; According to the expression The third modulus term of the modulus of the true velocity vector associated with the i-th sample video frame at time step t is determined. This represents the (i+1)th initial noise vector. This represents the vector corresponding to the (i+1)th sample video frame; Based on the first modulus term, the second modulus term, and the third modulus term, the constant associated with the i-th sample video frame is obtained; The noise vector associated with the i-th sample video frame at time step t is processed to obtain the predicted velocity vector associated with the i-th sample video frame at time step t, which also includes: The noise vector associated with the i-th sample video frame at time step t is processed to obtain the magnitude of the predicted velocity vector associated with the i-th sample video frame at time step t. The model parameters of the text-to-video generation base model to be fine-tuned are updated based on the difference between the predicted velocity vector associated with the i-th sample video frame at time step t and the true velocity vector associated with the i-th sample video frame at time step t. This also includes: The magnitude loss value is obtained by comparing the magnitude of the predicted velocity vector associated with the i-th sample video frame at time step t with the magnitude of the true velocity vector associated with the i-th sample video frame at time step t. Based on the orientation loss value and the magnitude loss value, the model parameters of the text-to-video generation base model to be fine-tuned are updated.

[0011] A second aspect of this application discloses a motion-enhanced video generation apparatus with time-aware stream matching, the apparatus comprising: The sample input module is used to input the training samples of video generation into the text to be fine-tuned into the video generation base model. The training samples of video generation include: sample videos and descriptive text of sample videos. The modeling and prediction module aims to learn the distribution mapping between the i-th initial noise vector and the i-th sample video frame. Using a text-to-video generation base model to be fine-tuned, it performs intra-frame modeling on the i-th sample video frame and inter-frame modeling between the i-th sample video frame and its adjacent sample video frames, determining the noise vector associated with the i-th sample video frame at time step t. Then, using the text-to-video generation base model to be fine-tuned, it processes the noise vector associated with the i-th sample video frame at time step t and time step t to obtain the predicted velocity vector associated with the i-th sample video frame at time step t. The value of time step t ranges from 0 to 1; the value of i ranges from 1 to n, where n represents the total number of frames in the sample video. The velocity determination module is used to determine the real velocity vector associated with the i-th sample video frame at time step t based on the vector corresponding to the i-th initial noise vector and the i-th sample video frame. The parameter update module is used to update the model parameters of the text-to-video generation base model to be fine-tuned based on the difference between the predicted velocity vector associated with the i-th sample video frame at time step t and the real velocity vector associated with the i-th sample video frame at time step t, so as to obtain the trained text-to-video generation model. The video generation module is used to input the target description text into the trained text and then into the video generation model to obtain the generated video corresponding to the target description text.

[0012] Optionally, the modeling and prediction module is used for: According to the expression Inter-frame modeling is performed on the i-th sample video frame and the (i-1)-th sample video frame to learn the temporal relationship between the i-th sample video frame and the (i-1)-th sample video frame. According to the expression Inter-frame modeling is performed on the i-th sample video frame and the (i+1)-th sample video frame to learn the temporal relationship between the i-th sample video frame and the (i+1)-th sample video frame. According to the expression Intra-frame modeling is performed on the i-th sample video frame in the sample video. in, Indicates an indicator function; Represents the Euclidean distance function; This represents the noise vector associated with the i-th sample video frame at time step t. This represents the vector corresponding to the i-th sample video frame; This represents the noise vector associated with the (i-1)th sample video frame at time step t. This represents the vector corresponding to the (i-1)th sample video frame; This represents the noise vector associated with the (i+1)th sample video frame at time step t. This represents the vector corresponding to the (i+1)th sample video frame.

[0013] Optionally, the modeling and prediction module is further configured to: Solve the following equation to determine the noise vector associated with the i-th sample video frame at time step t:

[0014] in, A real number greater than or equal to 4; ,and Let represent the Euclidean distance function, and let the direction of the true velocity vector associated with the i-th sample video frame at time step t be: ; This represents the i-th initial noise vector. This represents the (i-1)th initial noise vector. This represents the (i+1)th initial noise vector.

[0015] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the motion-enhanced video generation method for time-aware stream matching described in the first aspect of this application.

[0016] A fourth aspect of this application discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the motion-enhanced video generation method for time-aware stream matching described in the first aspect of this application.

[0017] The method provided in this application has the following beneficial effects: This method aims to learn the distribution mapping from the initial noise vector to sample video frames, while simultaneously performing intra-frame and inter-frame modeling on the sample video frames. By introducing inter-frame modeling on top of intra-frame modeling, the model can explicitly perceive the temporal dependencies between adjacent frames while learning single-frame generation capabilities. Furthermore, the method determines the true velocity vector based on the initial noise vector and sample video frames, and obtains the predicted velocity vector through model prediction, using the difference between the two to update the model parameters. This training approach enables the model to learn a velocity field that conforms to intra-frame and inter-frame constraints, thereby internalizing the motion priors in the video data.

[0018] The text-to-video generation model obtained based on the above training method can better maintain the continuity and consistency of motion between frames when generating videos, reducing jitter, jumps, or structural breaks caused by missing temporal information. Simultaneously, since the model has learned dynamic inter-frame relationships during training, its generation stability in complex motion scenes (such as fast-paced actions and multi-object interactions) is improved, and the generated video content is more consistent with the physical laws of the real world. Therefore, this method can improve the temporal continuity and physical plausibility of generated videos while maintaining computational efficiency. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the steps of a motion-enhanced video generation method based on time-aware stream matching, as provided in an embodiment of this application. Figure 2 This is a qualitative comparison result of the scheme provided in this application embodiment with other baseline models 1; Figure 3 This is a qualitative comparison result of the scheme provided in this application embodiment with other baseline models 2; Figure 4 This is a qualitative comparison result of the scheme provided in this application embodiment with other baseline models 3; Figure 5 This is a qualitative comparison result of the scheme provided in this application embodiment with other baseline models 4; Figure 6 This is a schematic diagram of the structure of a motion-enhanced video generation device for time-aware stream matching provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In related technologies, stream matching is directly extended to video generation tasks, treating video generation as a sequence generation problem of independent image frames, and learning the mapping from noise to the target frame for each frame. However, stream matching methods are essentially designed for image generation and lack the ability to specifically model the temporal structure unique to video data. When generating videos containing complex motion or fast movements, the generated video clips often exhibit phenomena such as discontinuous motion between frames, jittery objects or characters, and structural deformation. Some generated results even violate basic physical laws, such as unreasonable object trajectories and abnormal changes in lighting and shadow.

[0023] Analysis revealed that the root cause of the aforementioned problems lies in the inability of relevant methods to effectively capture the dynamic relationships and temporal dependencies between video frames, indicating insufficient perceptual ability of the model regarding the temporal dimension. Therefore, how to improve the temporal coherence and physical plausibility of the video generation model while maintaining the quality of single-frame generation is an urgent problem to be solved.

[0024] Based on this, this application proposes a motion-enhanced video generation method using Temporal-aware Flow Matching (TFM). The core inventive concept lies in explicitly embedding motion priors into the generated stream, introducing inter-frame temporal constraints into the flow matching training objective, and using the distance relationship between adjacent frames along the noise evolution path as a supervision signal for model learning. This allows the model to simultaneously satisfy intra-frame generation quality and inter-frame motion consistency during training. In this way, the model can internalize the motion priors in the video data, ultimately generating temporally coherent and physically plausible video content without increasing computational overhead during the inference phase.

[0025] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a motion-enhanced video generation method based on time-aware stream matching, as provided in an embodiment of this application. Figure 1 As shown, the method may include steps S110 to S150: Step S110: Input the video generation training samples into the text to be fine-tuned into the video generation base model. The video generation training samples include: sample videos and descriptive text of the sample videos.

[0026] In this step, the video training samples consist of pairs of sample videos and their corresponding descriptive texts. The sample videos are a continuous sequence of video frames, and the descriptive texts are natural language descriptions of the video content (such as actions, scenes, object interactions, etc.).

[0027] The text-to-video generation base model to be fine-tuned refers to a pre-trained model that already possesses basic video generation capabilities, such as Wan2.1-T2V-14B and CogVideoX. This base model is fine-tuned by inputting training samples, allowing it to learn better motion priors.

[0028] Step S120: With the goal of learning the distribution mapping between the i-th initial noise vector and the i-th sample video frame, the i-th sample video frame in the sample video is modeled intra-frame using the text-to-video generation base model to be fine-tuned, and the i-th sample video frame is modeled inter-frame with its adjacent sample video frames. The noise vector associated with the i-th sample video frame at time step t is determined. The noise vector associated with the i-th sample video frame at time step t and time step t are processed using the text-to-video generation base model to be fine-tuned to obtain the predicted velocity vector associated with the i-th sample video frame at time step t. The value of time step t is 0 to 1; the value of i is 1 to n, where n represents the total number of frames in the sample video.

[0029] In this step, the initial noise vector From the standard Gaussian distribution ( Random sampling is performed on the i-th sample video frame as the starting point for the generation process. This represents the feature representation of the frame in the latent space. The time step t controls the degree of evolution from noise to data; t = 0 corresponds to the pure noise state, and t = 1 corresponds to the real video frame state.

[0030] Intra-frame modeling focuses on a single video frame, learning the mapping relationship from noise to the stream of that frame; inter-frame modeling focuses on adjacent video frames (frame i-1 and frame i, frame i and frame i+1), learning the temporal dependencies between frames, enabling the model to perceive the dynamic changes between frames while generating a single frame.

[0031] The noise vector associated with the i-th sample video frame at time step t This represents the intermediate state during the evolution from the initial noise to the target frame at time step t. The model takes this noise vector and time step t as input and outputs a predicted velocity vector. This is used to describe the trend of noise vector changes over time. Specifically, for each frame, the model simultaneously processes intra-frame constraints (distance between the current frame's intermediate state and its target frame) and inter-frame constraints (distance between the current frame's intermediate state and the adjacent frame's intermediate state). The noise vector is obtained by solving a linear system that satisfies these constraints, and then the velocity vector is predicted.

[0032] Step S130: Based on the vector corresponding to the i-th initial noise vector and the i-th sample video frame, determine the true velocity vector associated with the i-th sample video frame at time step t.

[0033] In this step, the true velocity vector is the theoretical velocity of the noise vector evolving over time under an ideal probabilistic path. In standard stream matching, the true velocity vector is typically defined as the difference between the target frame vector and the initial noise vector. For the i-th frame, calculate The direction of the true velocity vector is obtained; combined with the magnitude constraints of intra-frame and inter-frame data, the complete value of the true velocity vector (including direction and magnitude) can be further determined as a supervision target for model prediction.

[0034] Step S140: Based on the difference between the predicted velocity vector associated with the i-th sample video frame at time step t and the true velocity vector associated with the i-th sample video frame at time step t, update the model parameters of the text-to-video generation base model to be fine-tuned, and obtain the trained text-to-video generation model.

[0035] In this step, the difference between the predicted velocity vector and the true velocity vector is calculated (e.g., using mean squared error loss, cosine similarity loss, or a combination of both) to obtain a loss value. Then, using optimization algorithms such as gradient descent, the loss is backpropagated to the model parameters, and the parameters are gradually adjusted. This allows the model's predictions to continuously approximate the actual velocity field.

[0036] Repeat steps S110 to S140 until the model converges, resulting in a trained text-to-video generation model. This training process ensures that the velocity field learned by the model satisfies both intra-frame and inter-frame constraints, thereby internalizing the motion priors between video frames.

[0037] Step S150: Input the target description text into the trained text to the video generation model to obtain the generated video corresponding to the target description text.

[0038] In this step, the target descriptive text is a user-provided natural language statement describing the desired video content (e.g., "a running cheetah"). During the inference phase, the model starts with a random, noisy video sequence, iteratively applies a learned velocity field for denoising, and ultimately generates a video clip that is semantically consistent with the descriptive text and has temporal coherence. This inference process is completely consistent with standard stream matching models and does not add any additional computational overhead.

[0039] The technical solution adopted in this embodiment, by simultaneously performing intra-frame and inter-frame modeling during the training phase and updating model parameters using the difference between the predicted velocity vector and the true velocity vector, allows the model to explicitly learn the temporal dependencies and motion priors between video frames. The video generation model trained in this way can improve the coherence and consistency of inter-frame motion when generating videos, reducing jitter, jumps, and structural breaks; it also exhibits stronger stability in complex motion scenes (such as fast-paced action and multi-object interaction), and the generated content is more consistent with real physical laws (such as smooth motion trajectories and reasonable changes in lighting).

[0040] In an optional embodiment, step S120 above, "performing inter-frame modeling of the i-th sample video frame and its adjacent sample video frames," may include steps A1 to A2: Step A1: According to the expression Inter-frame modeling is performed on the i-th sample video frame and the (i-1)-th sample video frame to learn the temporal relationship between the i-th sample video frame and the (i-1)-th sample video frame.

[0041] Step A2: According to the expression Inter-frame modeling is performed on the i-th sample video frame and the (i+1)-th sample video frame to learn the temporal relationship between the i-th sample video frame and the (i+1)-th sample video frame.

[0042] Furthermore, the step S120 above, "performing intra-frame modeling of the i-th sample video frame in the sample video," may include: according to the expression... Intra-frame modeling is performed on the i-th sample video frame in the sample video. in, Indicates an indicator function; Represents the Euclidean distance function; This represents the noise vector associated with the i-th sample video frame at time step t. This represents the vector corresponding to the i-th sample video frame; This represents the noise vector associated with the (i-1)th sample video frame at time step t. This represents the vector corresponding to the (i-1)th sample video frame; This represents the noise vector associated with the (i+1)th sample video frame at time step t. This represents the vector corresponding to the (i+1)th sample video frame.

[0043] In this embodiment, during inter-frame modeling, step A1 calculates the noise vector associated with the current frame. Noise vector associated with the previous frame The distance between them constrains the similarity or continuity of adjacent frames along the noise evolution path. Indicator function This eliminates the need to calculate the relationship between the first and previous frames in the first frame.

[0044] Similarly, for the relationship with the next frame (step A2), the noise vector associated with the current frame is calculated. Noise vector associated with the next frame The distance between them is used to implement timing constraints with the next frame. Indicator function This eliminates the need to calculate the relationship between the last frame and the next frame.

[0045] Intra-frame modeling involves computing the noise vector associated with the current frame. With the real target vector of this frame The Euclidean distance between the generated state and the real frame is used to measure the deviation between the generated state and the real frame. This term is a fundamental constraint in standard stream matching, ensuring that the model can learn the mapping from noise to a single frame image.

[0046] By combining the above intra-frame terms with inter-frame terms (e.g., with hyperparameters) The weighted summation constitutes the overall constraint on the video generation process. During training, the model needs to ensure that its predicted velocity fields simultaneously satisfy these constraints in order to learn the dynamic relationships between frames.

[0047] It is understood that the process of determining the noise vector associated with the (i-1)th sample video frame at time step t is the same as the process of determining the noise vector associated with the ith sample video frame at time step t; similarly, the process of determining the noise vector associated with the (i+1)th sample video frame at time step t is the same as the process of determining the noise vector associated with the ith sample video frame at time step t. The specific process of determining the noise vector associated with the ith sample video frame at time step t will be explained in subsequent embodiments.

[0048] The technical solution adopted in this embodiment explicitly models the noise vector distance between the current frame and the previous and next frames, enabling the model to directly perceive the temporal relationship between adjacent frames during training. Compared with implicit attention mechanisms, this is more effective in injecting motion priors and improving inter-frame coherence. The indicator function ensures the rationality of boundary frame processing and is compatible with the standard flow matching framework without changing the main structure of the model.

[0049] In an optional embodiment, the step S120 above, "determining the noise vector associated with the i-th sample video frame at time step t," may include: solving the following equation to determine the noise vector associated with the i-th sample video frame at time step t:

[0050] in, A real number greater than or equal to 4; ,and Let represent the Euclidean distance function, and let the direction of the true velocity vector associated with the i-th sample video frame at time step t be: ; This represents the i-th initial noise vector. This represents the (i-1)th initial noise vector. This represents the (i+1)th initial noise vector.

[0051] In this embodiment, the (i-1)th initial noise vector The process is the same as for the i-th initial noise vector, to determine the noise vector associated with the (i-1)-th sample video frame at time step t; the (i+1)-th initial noise vector The process is the same as that used for the i-th initial noise vector, to determine the noise vector associated with the (i+1)-th sample video frame at time step t.

[0052] hyperparameters Control the relative weights between intra-frame constraints and inter-frame constraints. Guarantee the mathematical validity of the stream (i.e., correctly map noise to real data). Time decay function. Linear decrease indicates that the distance between the noise vector and the target frame gradually decreases with each time step. The left side of the equation represents the distance between the current frame's noise vector and its own target frame (intra-frame term) and the distance to the noise vectors of adjacent frames (inter-frame term); the right side of the equation represents the target value, which is determined by the initial distance between noise vectors, the distance between target frames, and... Weighted decision.

[0053] When solving the above equation, it is transformed into a linear system relating velocity and magnitude. , where C is a constant vector, calculated from the initial noise distance, the target frame distance, and the time decay function; The velocity field magnitude is a diagonally dominant tridiagonal matrix, derived from the intra- and inter-frame constraints ρ and the unit direction vector, used to describe the linear system whose velocity magnitude must satisfy. It is solved using efficient numerical methods such as the pursuit method to obtain the velocity field magnitude at time step t for each frame. Then, combined with the initial noise vector With directional constraints, the noise vector at the current time step t is iteratively calculated using the Euler discretization method. .

[0054] The technical solution of this embodiment couples the temporal relationship between frames in a strict mathematical form through the above equation. The defined probability path also considers the distance between adjacent frames in the noise space, so that the model can learn during the learning process. The system ensures the validity of the paths and the diagonal dominance property, resulting in stable and efficient numerical solutions. Without altering the computational flow in the inference phase, it enhances the temporal coherence and physical plausibility of the generated video.

[0055] In an optional embodiment, step S130 above, "determining the true velocity vector associated with the i-th sample video frame at time step t based on the vector corresponding to the i-th initial noise vector and the i-th sample video frame," includes at least step C1: Step C1: Based on the vector corresponding to the i-th sample video frame and the i-th initial noise vector, determine the direction of the true velocity vector associated with the i-th sample video frame at time step t.

[0056] In this embodiment, for the i-th frame, the direction of the true velocity vector is determined by the difference between the target vector and the initial noise vector of that frame, i.e. This direction represents the optimal evolution path from noise to the target frame, and is the target direction that the model should learn.

[0057] Furthermore, step S120 above, which involves "processing the noise vector associated with the i-th sample video frame at time step t and time step t to obtain the predicted velocity vector associated with the i-th sample video frame at time step t," includes at least step D1: Step D1: Process the noise vector associated with the i-th sample video frame at time step t and time step t to obtain the direction of the predicted velocity vector associated with the i-th sample video frame at time step t.

[0058] In this embodiment, the noise vector at the current time step The text to be fine-tuned is input into the video generation base model at time step t, and the model outputs a predicted velocity vector. The direction component (i.e., the unit vector) is extracted from this predicted vector as the predicted direction.

[0059] Furthermore, the step S140 above, which involves "updating the model parameters of the text-to-video generation base model to be fine-tuned based on the difference between the predicted velocity vector associated with the i-th sample video frame at time step t and the true velocity vector associated with the i-th sample video frame at time step t," includes at least steps E1 and E2: Step E1: Obtain the direction loss value based on the difference between the direction of the predicted velocity vector associated with the i-th sample video frame at time step t and the direction of the true velocity vector associated with the i-th sample video frame at time step t.

[0060] Step E2: Update the model parameters of the text-to-video generation base model to be fine-tuned, at least based on the orientation loss value.

[0061] In this embodiment, the direction loss value is used as a quantitative indicator to measure the degree of difference between the direction of the velocity vector predicted by the model and the direction of the true velocity vector. Cosine similarity loss can be used, which calculates the cosine of the angle between two direction vectors; the larger the angle, the greater the loss.

[0062] Based on the aforementioned directional loss values, the gradient of the loss with respect to the model parameters is calculated using the backpropagation algorithm, and the model parameters are updated using an optimizer (such as Adam). By minimizing the directional loss, the velocity field direction learned by the model will gradually approach the ideal direction, thereby ensuring that the evolution path of the noise vector during the generation process is correct.

[0063] The technical solution adopted in this embodiment explicitly introduces directional loss to constrain the model by separately determining the true velocity direction and the predicted velocity direction, enabling the model to more accurately capture the directional information of the velocity field during the learning process. Constraining the direction separately can avoid interference from modulus error and improve the model's perception accuracy of motion direction. Since the true velocity direction has a clear geometric meaning (pointing from noise to the target frame), this constraint helps the model converge faster and generate videos with motion trajectories that better match expectations.

[0064] In an optional embodiment, step S130 above, "determining the true velocity vector associated with the i-th sample video frame at time step t based on the vector corresponding to the i-th initial noise vector and the i-th sample video frame," further includes steps C2 to C5: Step C2: According to the expression Determine the strengths of the first module; among them, A real number greater than or equal to 4; Represents the Euclidean distance function; This represents the i-th initial noise vector. This represents the vector corresponding to the i-th sample video frame.

[0065] Step C3: According to the expression Determine the second module's strengths; This represents the (i-1)th initial noise vector. This represents the vector corresponding to the (i-1)th sample video frame; This indicates an indicator function.

[0066] Step C4: According to the expression The third modulus term of the modulus of the true velocity vector associated with the i-th sample video frame at time step t is determined. This represents the (i+1)th initial noise vector. This represents the vector corresponding to the (i+1)th sample video frame.

[0067] Step C5: Based on the first modulus term, the second modulus term, and the third modulus term, obtain the constant associated with the i-th sample video frame.

[0068] In this embodiment, the magnitude of the velocity vector determines the speed of evolution from noise to the target frame; wherein, the first magnitude term, The Euclidean distance between the initial noise of the current frame and the target frame, multiplied by the hyperparameters, is a fundamental component of the true velocity magnitude, reflecting the path length of independent evolution within the current frame. The second magnitude term... This represents the distance between the previous frame and the current frame in the noise space minus the distance in the target frame space; this item is only applicable when... It is effective at certain times and is used to capture relative motion information between adjacent frames. The third module length term... This represents the distance between the current frame and the next frame in the noise space minus the distance in the target frame space; this item is only applicable when... It is effective at the time and provides backward timing constraints.

[0069] By adding these three moduli, we obtain the constant C associated with the i-th sample video frame. i For the sample video, , The integrated information ensures that the velocity magnitude reflects both the evolutionary needs of a single frame and the relative motion relationships between adjacent frames. In one implementation, this is achieved by solving a system of equations. The velocity field is obtained. ,in,

[0070]

[0071] In one implementation, using Representing vectors A unit vector in a direction can be defined. The magnitude of the velocity field satisfies a linear system. ,in,

[0072] Furthermore, step S120 above, which involves "processing the noise vector associated with the i-th sample video frame at time step t and time step t to obtain the predicted velocity vector associated with the i-th sample video frame at time step t," also includes step D2: Step D2: Process the noise vector associated with the i-th sample video frame at time step t and time step t to obtain the magnitude of the predicted velocity vector associated with the i-th sample video frame at time step t.

[0073] In this embodiment, the noise vector at the current time step The text to be fine-tuned is input into the video generation base model at time step t, and the model outputs a predicted velocity vector. The magnitude (i.e., L2 norm) of this predicted vector is extracted as the predicted velocity magnitude.

[0074] Furthermore, the step S140 above, which involves "updating the model parameters of the text-to-video generation base model to be fine-tuned based on the difference between the predicted velocity vector associated with the i-th sample video frame at time step t and the true velocity vector associated with the i-th sample video frame at time step t," also includes steps E3 and E4: Step E3: Based on the difference between the magnitude of the predicted velocity vector associated with the i-th sample video frame at time step t and the magnitude of the true velocity vector associated with the i-th sample video frame at time step t, obtain the magnitude loss value.

[0075] Step E4: Update the model parameters of the text-to-video generation base model to be fine-tuned based on the orientation loss value and the magnitude loss value.

[0076] In this embodiment, the magnitude loss value is used as a quantitative indicator to measure the difference between the magnitude of the predicted velocity vector and the magnitude of the true velocity vector. It can be the mean squared error (MSE) or L1 loss. The smaller the loss value, the closer the magnitude of the velocity predicted by the model is to the theoretical value.

[0077] The directional loss value (from step E1 above) is combined with the magnitude loss value (from step E3) (e.g., by weighted summation) to obtain the total loss value. :

[0078] in, The mathematical expectation represents the average of the expression within the parentheses over the distribution of the specified variable. This is real video data; is a diagonally dominant tridiagonal matrix; C is a constant vector; This is a vector composed of the magnitudes of the velocities in each frame of the velocity field predicted by the model. ; The cosine similarity function; The direction unit vector of the velocity vector predicted by the model; The direction of the actual velocity vector is the unit vector.

[0079] Then, the model parameters are updated through backpropagation. By simultaneously optimizing the direction and magnitude, the velocity field learned by the model can more comprehensively approximate the true velocity vector.

[0080] The technical solution adopted in this embodiment calculates the true velocity magnitude contributions within and between frames separately, and combines the magnitude loss and direction loss for parameter updates, achieving complete constraints on the velocity field (with both directional guidance and magnitude specification). The magnitude term incorporates the difference between the noise distance between adjacent frames and the target frame distance, enabling the model to perceive changes in motion amplitude between frames and generate motion rhythms that are more in line with physical laws (such as acceleration and deceleration). Simultaneously optimizing direction and magnitude allows the velocity field learned by the model to more comprehensively approximate the true value, improving the consistency of the generated video quality.

[0081] like Figures 2 to 5 The figure shows the qualitative comparison results of the solution provided in this embodiment with other baseline models (qualitative comparison of CogvideoX, HunyuanVideo and Wan2.1). Figure 2 A sequence of video frames generated based on the target description text "The chef puts a salad into a baking pan, first tops it with chopped mushrooms, then coats them with a layer of smooth butter, and finally puts it back into the baking pan"; Figure 3 A sequence of video frames generated for the target description text "A sprinter crosses the finish line from the orange-marked track, swinging his strong arms under the morning sun, standing on a wet sports field"; Figure 4 The target description text is "A sushi chef cuts a roll of sushi with a precise, fluid motion." Figure 5 This document describes a sequence of video frames generated for the target text "A figure skater takes off into the air, his skates gliding on the ice." It can be seen that the text-to-video generation model provided in this application generates content that more closely matches real-world physical laws.

[0082] This application also provides a motion-enhanced video generation apparatus for time-aware stream matching, referring to... Figure 6 As shown, Figure 6 This is a schematic diagram of a motion-enhanced video generation device for time-aware stream matching provided in an embodiment of this application. The device includes: The sample input module 610 is used to input the training samples of video generation into the text to be fine-tuned to the video generation base model. The training samples of video generation include: sample videos and descriptive text of sample videos. The modeling and prediction module 620 aims to learn the distribution mapping between the i-th initial noise vector and the i-th sample video frame. Using the text-to-video generation base model to be fine-tuned, it performs intra-frame modeling on the i-th sample video frame and inter-frame modeling between the i-th sample video frame and its adjacent sample video frames, determining the noise vector associated with the i-th sample video frame at time step t. Then, using the text-to-video generation base model to be fine-tuned, it processes the noise vector associated with the i-th sample video frame at time step t and time step t to obtain the predicted velocity vector associated with the i-th sample video frame at time step t. The value of time step t ranges from 0 to 1; the value of i ranges from 1 to n, where n represents the total number of frames in the sample video. The velocity determination module 630 is used to determine the real velocity vector associated with the i-th sample video frame at time step t based on the vector corresponding to the i-th initial noise vector and the i-th sample video frame. The parameter update module 640 is used to update the model parameters of the text-to-video generation base model to be fine-tuned based on the difference between the predicted velocity vector associated with the i-th sample video frame at time step t and the real velocity vector associated with the i-th sample video frame at time step t, so as to obtain the trained text-to-video generation model. The video generation module 650 is used to input the target description text into the trained text and then into the video generation model to obtain the generated video corresponding to the target description text.

[0083] In an optional embodiment, the modeling and prediction module 620 is used for: According to the expression Inter-frame modeling is performed on the i-th sample video frame and the (i-1)-th sample video frame to learn the temporal relationship between the i-th sample video frame and the (i-1)-th sample video frame. According to the expression Inter-frame modeling is performed on the i-th sample video frame and the (i+1)-th sample video frame to learn the temporal relationship between the i-th sample video frame and the (i+1)-th sample video frame. According to the expression Intra-frame modeling is performed on the i-th sample video frame in the sample video. in, Indicates an indicator function; Represents the Euclidean distance function; This represents the noise vector associated with the i-th sample video frame at time step t. This represents the vector corresponding to the i-th sample video frame; This represents the noise vector associated with the (i-1)th sample video frame at time step t. This represents the vector corresponding to the (i-1)th sample video frame; This represents the noise vector associated with the (i+1)th sample video frame at time step t. This represents the vector corresponding to the (i+1)th sample video frame.

[0084] In an optional embodiment, the modeling and prediction module 620 is further configured to: Solve the following equation to determine the noise vector associated with the i-th sample video frame at time step t:

[0085] in, A real number greater than or equal to 4; ,and Let represent the Euclidean distance function, and let the direction of the true velocity vector associated with the i-th sample video frame at time step t be: ; This represents the i-th initial noise vector. This represents the (i-1)th initial noise vector. This represents the (i+1)th initial noise vector.

[0086] In an optional embodiment, the velocity determination module 630 is used to: determine the direction of the real velocity vector associated with the i-th sample video frame at time step t based on the vector corresponding to the i-th sample video frame and the i-th initial noise vector; The modeling and prediction module 620 is used to: process the noise vector associated with the i-th sample video frame at time step t and time step t to obtain the direction of the predicted velocity vector associated with the i-th sample video frame at time step t. The parameter update module 640 is used to: obtain a direction loss value based on the difference between the direction of the predicted velocity vector associated with the i-th sample video frame at time step t and the direction of the true velocity vector associated with the i-th sample video frame at time step t; and update the model parameters of the text-to-video generation base model to be fine-tuned based at least on the direction loss value.

[0087] In an optional embodiment, the speed determination module 630 is used to: According to the expression Determine the strengths of the first module; among them, A real number greater than or equal to 4; Represents the Euclidean distance function; This represents the i-th initial noise vector. This represents the vector corresponding to the i-th sample video frame; According to the expression Determine the second module's strengths; This represents the (i-1)th initial noise vector. This represents the vector corresponding to the (i-1)th sample video frame; Indicates an indicator function; According to the expression The third modulus term of the modulus of the true velocity vector associated with the i-th sample video frame at time step t is determined. This represents the (i+1)th initial noise vector. This represents the vector corresponding to the (i+1)th sample video frame; Based on the first modulus term, the second modulus term, and the third modulus term, the constant associated with the i-th sample video frame is obtained; The modeling and prediction module 620 is used to: process the noise vector associated with the i-th sample video frame at time step t and time step t to obtain the magnitude of the predicted velocity vector associated with the i-th sample video frame at time step t. The parameter update module 640 is used to: obtain a magnitude loss value based on the difference between the magnitude of the predicted velocity vector associated with the i-th sample video frame at time step t and the magnitude of the true velocity vector associated with the i-th sample video frame at time step t; and update the model parameters of the text-to-video generation base model to be fine-tuned based on the direction loss value and the magnitude loss value.

[0088] It is understood that the motion-enhanced video generation device with time-aware stream matching in the embodiments of this application can implement the motion-enhanced video generation method with time-aware stream matching in the above embodiments. The motion-enhanced video generation device with time-aware stream matching has the same advantages as the above motion-enhanced video generation method with time-aware stream matching compared to the prior art, and will not be repeated here.

[0089] This application also provides an electronic device, see embodiments thereof. Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 7 As shown, the electronic device 700 includes a memory 710 and a processor 720. The memory 710 and the processor 720 are connected via a bus for communication. The memory 710 stores a computer program that can run on the processor 720 to implement the steps of the motion-enhanced video generation method for time-aware stream matching described in the embodiments of this application.

[0090] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the motion-enhanced video generation method for time-aware stream matching described in this application.

[0091] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0092] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0096] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0097] The foregoing has provided a detailed description of a motion-enhanced video generation method, apparatus, device, and medium for time-aware stream matching provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating motion-enhanced video using time-aware stream matching, characterized in that, include: The training samples for video generation are input into the text to be fine-tuned into the video generation base model. The training samples for video generation include: sample videos and descriptive text for the sample videos. The goal is to learn the distribution mapping between the i-th initial noise vector and the i-th sample video frame. Using the text-to-video generation base model to be fine-tuned, intra-frame modeling is performed on the i-th sample video frame, and inter-frame modeling is performed between the i-th sample video frame and its adjacent sample video frames. The noise vector associated with the i-th sample video frame at time step t is determined. Then, using the text-to-video generation base model to be fine-tuned, the noise vector associated with the i-th sample video frame at time step t and time step t are processed to obtain the predicted velocity vector associated with the i-th sample video frame at time step t. The value of time step t is 0 to 1; the value of i is 1 to n, where n represents the total number of frames in the sample video. Based on the vector corresponding to the i-th initial noise vector and the i-th sample video frame, determine the true velocity vector associated with the i-th sample video frame at time step t. Based on the difference between the predicted velocity vector associated with the i-th sample video frame at time step t and the true velocity vector associated with the i-th sample video frame at time step t, the model parameters of the text-to-video generation base model to be fine-tuned are updated to obtain the trained text-to-video generation model. Input the target description text into the trained text and then into the video generation model to obtain the generated video corresponding to the target description text.

2. The method according to claim 1, characterized in that, Inter-frame modeling is performed between the i-th sample video frame and its neighboring sample video frames, including: According to the expression Inter-frame modeling is performed on the i-th sample video frame and the (i-1)-th sample video frame to learn the temporal relationship between the i-th sample video frame and the (i-1)-th sample video frame. According to the expression Inter-frame modeling is performed on the i-th sample video frame and the (i+1)-th sample video frame to learn the temporal relationship between the i-th sample video frame and the (i+1)-th sample video frame. Intra-frame modeling is performed on the i-th sample video frame, including: According to the expression Intra-frame modeling is performed on the i-th sample video frame in the sample video. in, Indicates an indicator function; Represents the Euclidean distance function; This represents the noise vector associated with the i-th sample video frame at time step t. This represents the vector corresponding to the i-th sample video frame; This represents the noise vector associated with the (i-1)th sample video frame at time step t. This represents the vector corresponding to the (i-1)th sample video frame; This represents the noise vector associated with the (i+1)th sample video frame at time step t. This represents the vector corresponding to the (i+1)th sample video frame.

3. The method according to claim 2, characterized in that, Determine the noise vector associated with the i-th sample video frame at time step t, including: Solve the following equation to determine the noise vector associated with the i-th sample video frame at time step t: in, It is a real number greater than or equal to 4; ,and Let represent the Euclidean distance function, and let the direction of the true velocity vector associated with the i-th sample video frame at time step t be: ; This represents the i-th initial noise vector. This represents the (i-1)th initial noise vector. This represents the (i+1)th initial noise vector.

4. The method according to claim 1, characterized in that, Based on the vector corresponding to the i-th initial noise vector and the i-th sample video frame, determine the true velocity vector associated with the i-th sample video frame at time step t, including at least: Based on the vector corresponding to the i-th sample video frame and the i-th initial noise vector, determine the direction of the true velocity vector associated with the i-th sample video frame at time step t; The noise vector associated with the i-th sample video frame at time step t is processed to obtain the predicted velocity vector associated with the i-th sample video frame at time step t, which includes at least: The noise vector associated with the i-th sample video frame at time step t is processed to obtain the direction of the predicted velocity vector associated with the i-th sample video frame at time step t. Based on the difference between the predicted velocity vector associated with the i-th sample video frame at time step t and the true velocity vector associated with the i-th sample video frame at time step t, the model parameters of the text-to-video generation base model to be fine-tuned are updated, including at least: The direction loss value is obtained by comparing the direction of the predicted velocity vector associated with the i-th sample video frame at time step t with the direction of the true velocity vector associated with the i-th sample video frame at time step t. The model parameters of the text-to-video generation base model to be fine-tuned are updated based at least on the directional loss value.

5. The method according to claim 4, characterized in that, Based on the vector corresponding to the i-th initial noise vector and the i-th sample video frame, the true velocity vector associated with the i-th sample video frame at time step t is determined, which also includes: According to the expression Determine the strengths of the first module; among them, It is a real number greater than or equal to 4; Represents the Euclidean distance function; This represents the i-th initial noise vector. This represents the vector corresponding to the i-th sample video frame; According to the expression Determine the second module's strengths; This represents the (i-1)th initial noise vector. This represents the vector corresponding to the (i-1)th sample video frame; Indicates an indicator function; According to the expression The third modulus term of the modulus of the true velocity vector associated with the i-th sample video frame at time step t is determined. This represents the (i+1)th initial noise vector. This represents the vector corresponding to the (i+1)th sample video frame; Based on the first modulus term, the second modulus term, and the third modulus term, the constant associated with the i-th sample video frame is obtained; The noise vector associated with the i-th sample video frame at time step t is processed to obtain the predicted velocity vector associated with the i-th sample video frame at time step t, which also includes: The noise vector associated with the i-th sample video frame at time step t is processed to obtain the magnitude of the predicted velocity vector associated with the i-th sample video frame at time step t. The model parameters of the text-to-video generation base model to be fine-tuned are updated based on the difference between the predicted velocity vector associated with the i-th sample video frame at time step t and the true velocity vector associated with the i-th sample video frame at time step t. This also includes: The magnitude loss value is obtained by comparing the magnitude of the predicted velocity vector associated with the i-th sample video frame at time step t with the magnitude of the true velocity vector associated with the i-th sample video frame at time step t. Based on the orientation loss value and the magnitude loss value, the model parameters of the text-to-video generation base model to be fine-tuned are updated.

6. A motion-enhanced video generation apparatus for time-aware stream matching, characterized in that, include: The sample input module is used to input the training samples of video generation into the text to be fine-tuned into the video generation base model. The training samples of video generation include: sample videos and descriptive text of sample videos. The modeling and prediction module aims to learn the distribution mapping between the i-th initial noise vector and the i-th sample video frame. Using a text-to-video generation base model to be fine-tuned, it performs intra-frame modeling on the i-th sample video frame and inter-frame modeling between the i-th sample video frame and its adjacent sample video frames, determining the noise vector associated with the i-th sample video frame at time step t. Then, using the text-to-video generation base model to be fine-tuned, it processes the noise vector associated with the i-th sample video frame at time step t and time step t to obtain the predicted velocity vector associated with the i-th sample video frame at time step t. The value of time step t ranges from 0 to 1; the value of i ranges from 1 to n, where n represents the total number of frames in the sample video. The velocity determination module is used to determine the real velocity vector associated with the i-th sample video frame at time step t based on the vector corresponding to the i-th initial noise vector and the i-th sample video frame. The parameter update module is used to update the model parameters of the text-to-video generation base model to be fine-tuned based on the difference between the predicted velocity vector associated with the i-th sample video frame at time step t and the real velocity vector associated with the i-th sample video frame at time step t, so as to obtain the trained text-to-video generation model. The video generation module is used to input the target description text into the trained text and then into the video generation model to obtain the generated video corresponding to the target description text.

7. The apparatus according to claim 6, characterized in that, The modeling and prediction module is used for: According to the expression Inter-frame modeling is performed on the i-th sample video frame and the (i-1)-th sample video frame to learn the temporal relationship between the i-th sample video frame and the (i-1)-th sample video frame. According to the expression Inter-frame modeling is performed on the i-th sample video frame and the (i+1)-th sample video frame to learn the temporal relationship between the i-th sample video frame and the (i+1)-th sample video frame. According to the expression Intra-frame modeling is performed on the i-th sample video frame in the sample video. in, Indicates an indicator function; Represents the Euclidean distance function; This represents the noise vector associated with the i-th sample video frame at time step t. This represents the vector corresponding to the i-th sample video frame; This represents the noise vector associated with the (i-1)th sample video frame at time step t. This represents the vector corresponding to the (i-1)th sample video frame; This represents the noise vector associated with the (i+1)th sample video frame at time step t. This represents the vector corresponding to the (i+1)th sample video frame.

8. The apparatus according to claim 7, characterized in that, The modeling and prediction module is also used for: Solve the following equation to determine the noise vector associated with the i-th sample video frame at time step t: in, It is a real number greater than or equal to 4; ,and Let represent the Euclidean distance function, and let the direction of the true velocity vector associated with the i-th sample video frame at time step t be: ; This represents the i-th initial noise vector. This represents the (i-1)th initial noise vector. This represents the (i+1)th initial noise vector.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the motion-enhanced video generation method for time-aware stream matching as described in any one of claims 1-5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the motion-enhanced video generation method for time-aware stream matching as described in any one of claims 1-5.