Video generation methods, systems, computer devices, and storage media for physical consistency

By selecting reference videos from a preset video set and extracting physical prior features and injecting them into a denoising network, the physical inconsistency problem of existing text-to-video generation models is solved, achieving physical consistency and visual quality improvement in scenarios such as embodied intelligence, robot operation, and autonomous driving.

CN123123675APending Publication Date: 2026-11-17PEKING UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611552270.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-09
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing text-to-video generation models lack effective physical law modeling mechanisms, resulting in physical inconsistencies in the generated videos and limiting their application in scenarios such as embodied intelligence, robot operation, autonomous driving, and interactive world modeling.

Method used

By selecting reference videos that match the video generation instructions from a preset video set, extracting physical prior features, and injecting them into a denoising network, the generation process is guided to follow physical dynamics. Combined with a learnable query mechanism to filter irrelevant information, a high-quality training dataset is constructed.

Benefits of technology

It significantly improves the physical consistency and visual quality of generated videos, ensuring the physical correctness and causal consistency of generated videos in various physical phenomena, and is suitable for scenarios such as embodied intelligence, robot operation, and autonomous driving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123123675A_ABST
    Figure CN123123675A_ABST
Patent Text Reader

Abstract

This application proposes a video generation method, system, computer device, and storage medium oriented towards physical consistency. The method includes: responding to a video generation instruction, selecting reference videos from a preset video set that match the video prompts in the instruction; extracting video features from the reference videos and generating physical prior information from these features using a learnable query mechanism; using the physical prior information to characterize the physical dynamics corresponding to the prompts; injecting the physical prior features into a denoising network of a video generation model to guide the network to follow physical dynamics during video denoising and generation; and generating a target video corresponding to the video generation instruction based on the denoising network with the injected physical prior features. The embodiments of this application can significantly improve the physical consistency and adherence to physical common sense in the generated videos.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of video generation technology, specifically to a video generation method, system, computer device, and storage medium for physical consistency. Background Technology

[0002] With the improvement of computing power and the continuous expansion of training data, text-to-video generation technology can now synthesize high-fidelity, visually realistic dynamic content. However, existing video generation models frequently violate basic physical laws during the generation process. For example, object motion does not conform to the principles of mechanics, fluid dynamics behavior is distorted, and thermal and optical phenomena defy common sense. This lack of physical consistency severely restricts the application of video generation models in downstream scenarios with stringent requirements for physical realism, such as embodied intelligence, robot operation, autonomous driving, and interactive world modeling. Summary of the Invention

[0003] In view of this, this application proposes a video generation method, system, computer device and storage medium for physical consistency, in order to solve the problem that existing text-to-video generation models lack an effective physical law modeling mechanism, resulting in physical inconsistencies in the generated videos.

[0004] A first aspect of this application proposes a video generation method oriented towards physical consistency, the method comprising: In response to a video generation command, a reference video matching the video prompt information in the video generation command is selected from a preset video set; the reference video contains physical dynamic laws corresponding to the prompt information. The video features of the reference video are extracted, and physical prior information is extracted from the video features through a learnable query mechanism to generate physical prior features; the physical prior information is used to characterize the physical dynamic laws corresponding to the prompt information. The physical prior features are injected into the denoising network of the video generation model to guide the denoising network to follow the physical dynamic laws during the video denoising generation process. Based on the denoising network injected with the physical prior features, a target video corresponding to the video generation instruction is generated.

[0005] In this embodiment of the application, before selecting a reference video from a preset video set that matches the video prompt information in the video generation instruction, the method further includes: Obtain the original video dataset, which contains multiple video samples and a corresponding text description for each video sample; The physical relevance and description quality of each video sample are evaluated based on the text description. Video samples that are not related to physical phenomena are eliminated, and the top-ranked video samples are retained as candidate video samples. For any candidate video sample, extract the key frames of the candidate video sample, input the key frames and the corresponding text description into a multimodal alignment model, verify the consistency between the video frames and the text description, eliminate inconsistent candidate video samples, and retain consistent candidate video samples as reference videos. The preset video set is constructed based on several reference videos.

[0006] In this embodiment of the application, physical prior information is extracted from the video features through a learnable query mechanism to generate physical prior features, including: Using a learnable query vector as the query, and the video features as the key vector and value vector, cross-attention calculation is performed to extract physical dynamic information related to the physical dynamic law from the video features, thus obtaining attention features; The attention features are subjected to nonlinear transformation and residual connection processing through a feedforward network to obtain refined features. The refined features are projected onto the physical prior feature space to obtain the physical prior features.

[0007] In this embodiment of the application, the denoising network includes multiple cascaded early denoising layers, and the physical prior features are injected into the denoising network of the video generation model, including: For any of the multiple cascaded early denoising layers, the physical prior features are aligned according to the feature dimension of the early denoising layer to obtain aligned features; the aligned features are consistent with the feature dimension of the early denoising layer. The alignment feature is spliced ​​and fused with the original video feature of the early denoising layer to obtain the fused feature; The fused features are weighted by learnable gating parameters, and the weighted fused features are superimposed on the original video features of the early denoising layer in the form of residual connections to obtain the injected features, which are then input into the next level of denoising layer.

[0008] In this embodiment of the application, the reference videos in the preset video set are classified according to the category of physical phenomena. The reference videos under the same category of physical phenomena have similar physical characteristics, and the video features of the reference videos are pre-extracted by the video encoder and stored in the form of feature vectors.

[0009] In this embodiment of the application, selecting reference videos from a preset video set that match the video prompt information in the video generation instruction includes: The video prompt information is semantically encoded to obtain prompt semantic features; Calculate the similarity between the semantic features of the prompt and the video features of each video sample in the preset video set; Based on the similarity, at least one video sample is selected from the preset video set as the reference video.

[0010] In this embodiment of the application, the method further includes a training step for the video generation model: Obtain training samples, which include video generation instructions, reference videos retrieved according to the video generation instructions, and corresponding real videos; The video generation instructions and reference video are input into the video generation model, and the denoising network and the learnable query mechanism are jointly trained so that the predicted video output by the video generation model approaches the real video.

[0011] An embodiment of the second aspect of this application provides a video generation system oriented towards physical consistency, comprising: The reference video filtering module is used to filter reference videos from a preset video set that match the video prompt information in the video generation instruction in response to the video generation instruction; the reference videos contain physical dynamic laws corresponding to the prompt information; The physical prior feature generation module is used to extract video features from the reference video and extract physical prior information from the video features through a learnable query mechanism to generate physical prior features; the physical prior information is used to characterize the physical dynamic laws corresponding to the prompt information. The feature injection module is used to inject the physical prior features into the denoising network of the video generation model to guide the denoising network to follow the physical dynamic laws during the video denoising generation process. The target video generation module is used to generate a target video corresponding to the video generation instruction based on a denoising network injected with the physical prior features.

[0012] An embodiment of the third aspect of this application provides a computer device including a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the physically consistent video generation method described in the first aspect above.

[0013] An embodiment of the fourth aspect of this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the physically consistent video generation method described in the first aspect above.

[0014] The technical solution provided in this application, by retrieving reference videos that match the video prompt information from a preset video set and extracting physical prior information from them and injecting it into the denoising network of the video generation model, achieves a deep fusion of explicit physical priors and implicit generation processes. Compared with the prior art, this application has the following beneficial effects: First, by introducing a retrieval-enhanced physical prior injection mechanism, the physical dynamics in real-world videos are explicitly guided into the video generation process, significantly improving the physical consistency and adherence to physical common sense in the generated videos. It demonstrates superior physical correctness in the generation of various physical phenomena, including mechanics, optics, thermodynamics, and materials.

[0015] Second, by using a learnable query mechanism as an information bottleneck, physical prior information is selectively extracted from the video features of the reference video, effectively filtering out irrelevant visual noise and redundant information, avoiding the adverse biases caused by directly introducing dense reference features, and ensuring the visual quality of the generated video while improving physical perception capabilities.

[0016] Third, by injecting physical prior features into multiple cascaded early denoising layers of the denoising network, the model establishes correct physical cognition at the initial stage of video generation, realizing multi-level fusion of physical laws from coarse-grained structure to fine-grained dynamics, ensuring the temporal causal consistency and spatial dynamic authenticity of the generated video.

[0017] Fourth, by constructing a pre-defined video set organized according to the categories of physical phenomena and performing a two-stage quality screening on the original video dataset, high-quality training data with strong physical relevance and high consistency between images and text were obtained, providing a reliable data foundation for the training of the physical perception video generation model.

[0018] Fifth, the proposed solution requires only a very small number of learnable parameters and negligible inference latency to significantly improve physical perception capabilities on the basis of existing video generation models, demonstrating good practicality and deployment friendliness. Attached Figure Description

[0019] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a physically consistent video generation method provided in an embodiment of this application is shown. Figure 2This illustration shows an overall framework diagram of a video generation method for physical consistency provided in an embodiment of this application; Figure 3 The illustration shows a flowchart of a process for generating physical prior information from video features using a learnable query mechanism, according to an embodiment of this application. Figure 4 This illustration shows a schematic diagram of the structure of a video generation system for physical consistency provided in an embodiment of this application; Figure 5 This illustration shows a schematic diagram of the structure of a computer device according to an embodiment of this application; Figure 6 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0020] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0021] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0022] The following explains the shortcomings of the existing technology and the technical effects that the technical solution of this application aims to achieve: Although text-to-video generation technology has made significant progress in visual fidelity, existing methods still have significant shortcomings in physical consistency, which can be mainly attributed to the following three types of technical deficiencies: First, explicit physical modeling methods have limited generalization ability. Existing explicit methods rely on deterministic physics simulators or structured mathematical constraints to directly simulate motion. Although they can strictly follow physical laws under predefined rules, they are difficult to generalize to complex and open real-world scenarios. For example, various physical phenomena such as thermodynamic effects, material deformation, and interactions between non-rigid objects are difficult to model accurately using explicit equations, which severely limits the application of such methods in open-domain video generation.

[0023] Second, implicit physics learning methods lack controllability. Existing implicit methods mainly rely on data-driven strategies, using techniques such as direct preference optimization and reinforcement learning to align the model with a reasonable distribution of physics. However, physics priors are usually implicitly encoded in the form of scalar rewards or cue-side reasoning, making it difficult to inject precise physics guidance into the implicit representations already learned by the model. This "black box" physics alignment lacks controllability and cannot ensure strict adherence to specific physical dimensions (such as fluid dynamics and mechanical collision timing) in the generated videos.

[0024] Third, retrieval-enhanced conditional control lacks a physical perception mechanism. Retrieval-enhanced generative techniques have proven their ability to improve fidelity and diversity in the field of visual generation. However, existing retrieval-enhanced methods in video generation (such as cross-attention conditionalization, ControlNet, IP-Adapter, or plug-and-play memory) typically transfer global style or overall dynamic characteristics, and their retrieval databases are general-domain databases, lacking a mechanism specifically for extracting explicit physical attributes. Therefore, the retrieved reference information contains a large amount of visual noise (such as background, texture, and color) unrelated to physical laws. Directly injecting this noise into the generative network introduces undesirable biases and actually reduces the generation quality. The potential denoising process of spatiotemporally injecting explicit physical prior knowledge into the video generation backbone network through a query method remains largely unexplored in research.

[0025] Fourth, high-quality physical perception training data is scarce. While existing publicly available video datasets are vast, they lack high-quality annotations and filtering for physical phenomena. Common problems include inconsistencies between text descriptions and video content, irrelevant physical phenomena, and incorrect physical behaviors. These issues hinder models from learning reliable underlying physical laws, severely limiting the training effectiveness of physical perception video generation models.

[0026] In view of the aforementioned deficiencies in the existing technology, this application aims to provide a video generation method, system, computer device, and storage medium for physical consistency, the technical objectives of which include: First, an explicit physics prior injection mechanism is constructed. By retrieving reference videos that match the video generation instructions and selectively extracting physics prior information from the reference videos using a learnable query mechanism, the physical dynamics of real-world videos are explicitly injected into the denoising network of the video generation model, realizing the transformation of physical knowledge from "implicit memory" to "explicit conditions".

[0027] Second, a physical knowledge distillation path is established to address information bottlenecks. By using cross-attention interaction between learnable query vectors and video features from reference videos, irrelevant visual noise is filtered out from dense reference video features, retaining only the physical dynamics relevant to the current generation task, thus avoiding interference from irrelevant information on the generation quality.

[0028] Third, it enables multi-level early physical cognition guidance. The extracted physical prior features are injected into multiple cascaded early denoising layers of the denoising network, enabling the model to establish correct physical cognition at the initial stage of video generation. This ensures the multi-level fusion of physical laws from coarse-grained structure to fine-grained dynamics, improving the performance of the generated video in terms of temporal causality and spatial dynamic realism.

[0029] Fourth, a high-quality physical video data foundation is constructed. Through a two-stage data screening process (coarse-grained text pre-screening and fine-grained semantic alignment verification), a pre-set video set with strong physical relevance and high consistency between text and images is constructed from the original video dataset, providing reliable data support for the training and inference of the physical perception video generation model.

[0030] Fifth, physical consistency is improved while maintaining visual quality. By fusing physical prior features with the original features of the denoising network through gated residual connections, physical constraints are introduced while preserving the original interface and visual generation capabilities of the generative model, achieving a synergistic improvement in physical correctness and visual fidelity.

[0031] Example 1 Figure 1 This is a flowchart illustrating a video generation method for physical consistency provided in an embodiment of this application. Figure 2 This is a schematic diagram of the overall framework of a video generation method for physical consistency provided in an embodiment of this application.

[0032] like Figure 1 , 2 As shown, this method is based on the Retrieval-Augmented Generation paradigm. It improves the physical consistency of the generated video by explicitly retrieving reference videos that match the video generation instructions and injecting their encoded physical dynamic information into the video generation model.

[0033] like Figure 1 As shown, the method includes the following steps S101-S104: Step S101: In response to the video generation instruction, a reference video that matches the video prompt information in the video generation instruction is selected from a preset video set.

[0034] Specifically, the video generation instruction includes at least video prompts describing the content of the video to be generated, such as "jam poured on bread". The preset video set is a pre-built physical video database containing multiple reference videos. Each reference video is categorized and stored according to the type of physical phenomenon. Reference videos under the same physical phenomenon category have similar physical dynamic characteristics, such as fluid dynamics, mechanical collision, heat conduction, and optical reflection.

[0035] More specifically, the reference video contains physical dynamics corresponding to the prompt information. For example, when the prompt information describes "pouring jam", the retrieved reference video could be a real physical video of "pouring honey" or "pouring syrup", which contains the physical dynamics of a viscous liquid falling under gravity and spreading upon contact with a surface.

[0036] In some specific implementations, the video features of reference videos in a preset video set are pre-extracted by the video encoder and stored in the form of feature vectors to enable efficient vector similarity calculation during the stage of retrieving reference videos that match the video prompt information. The video encoder may employ a pre-trained model capable of extracting video features, such as VideoMAE V2, whose output features can simultaneously encode the dynamic information of the video in both spatial and temporal dimensions.

[0037] In some specific implementations, the process of selecting reference videos from a preset video set includes: semantically encoding the video prompt information to obtain prompt semantic features; calculating the similarity between the prompt semantic features and the video features of each video sample in the preset video set; and selecting at least one video sample from the preset video set as the reference video based on the similarity. The similarity calculation can be performed in a joint embedding space, where the prompt semantic features and video features are mapped to the same semantic space, and then the vector distance (e.g., cosine similarity) is calculated to select the semantically best-matching reference video.

[0038] Step S102: Extract video features from the reference video, and extract physical prior information from the video features through a learnable query mechanism to generate physical prior features.

[0039] Specifically, the retrieved reference video is input into a video encoder to extract its video features. These video features are denoted as... Its dimensions are ,in The length of the token sequence. The feature dimension is denoted by 'token', and the token is the smallest data unit that the model can directly process.

[0040] In some specific embodiments, step S102 above includes steps S1021-S1023: Step S1021: Using the learnable query vector as the query, and the video features as the key vector and value vector, cross-attention calculation is performed to extract physical dynamic information related to the physical dynamic law from the video features, thereby obtaining attention features.

[0041] Specifically, due to video characteristics It contains rich visual information, including useful information related to physical dynamics (such as liquid flow trajectories and object collisions and bounces), as well as irrelevant visual noise (such as background color, irrelevant object appearances, and shooting angles). Directly embedding this into a denoising network may introduce undesirable biases. Therefore, this application uses a learnable query mechanism as an information bottleneck to selectively extract prior physical information, such as... Figure 3 As shown: The query injection module maintains a set of learnable query vectors. Its dimensions are ,in To query the number of tokens (e.g., 128). This is the hidden dimension of the query adapter. The query vector is automatically optimized through backpropagation during model training, rather than being dynamically generated based on text prompts.

[0042] The extraction process includes: First, utilize learnable query vectors. Q As a query, video features Cross-attention is calculated using the key vector and value vector as the key and value vectors.

[0043] This operation adaptively focuses the query vector on the parts of the video features most relevant to the physical dynamics, extracting physical dynamic information related to the physical dynamics from the video features to obtain attention features. .

[0044] Step S1022: The attention features are subjected to nonlinear transformation and residual connection processing through a feedforward network to obtain refined features.

[0045] Specifically, attention features are applied through a feed-forward network (FFN). Perform nonlinear transformation and residual connection processing:

[0046] in, For attention features The refined features are obtained after performing nonlinear transformation and residual connection processing. , , For a learnable projection matrix, Specifically, the first layer is a linear projection used to map the input features to the hidden dimensions. Specifically, the second-layer linear projection is used to reduce the dimensionality of the activated hidden features and map them back to the output dimension (consistent with the input dimension); GELU is the activation function. This step further refines and enhances the extracted physical dynamic information.

[0047] Step S1023: Project the refined features onto the physical prior feature space to obtain the physical prior features.

[0048] Specifically, the refined features will be achieved through the following methods. Projecting onto the physical prior feature space yields the physical prior features. :

[0049] in, The learnable projection matrix can be understood as a linear transformation layer in the model, used to "compress" the refined physical dynamics into a compact feature space specifically designed to represent physical laws. (Physical prior features) The dimension is , Representing physical prior features Length of the included token sequence Representing physical prior features The hidden dimension, i.e., the vector dimension of each token; physical prior features. Used to characterize the physical dynamics corresponding to the prompt information, and irrelevant visual noise has been filtered out.

[0050] Step S103: Inject the physical prior features into the denoising network of the video generation model to guide the denoising network to follow the physical dynamic laws during the video denoising generation process.

[0051] Specifically, the video generation model adopts a diffusion model architecture, and its denoising network consists of multiple cascaded denoising layers (such as...). Figure 2 The denoising layers (from "denoising layer 0 to denoising layer N") are stacked sequentially. The early denoising layers refer to the initial, consecutive layers in the denoising network (e.g., ...). Figure 2 (Refurbishment layers 0 to Refurbishment layers 2 in the text). The denoising network receives noise input and text conditions, and generates the target video by denoising layer by layer.

[0052] More specifically, the process of injecting physical prior features into the denoising network is shown in steps S1031-S1033: Step S1031: For any of the multiple cascaded early denoising layers, align the physical prior features according to the feature dimensions of the early denoising layer to obtain aligned features.

[0053] Specifically, the feature dimension of the early denoising layer DiT is: ,in, This indicates the length of the token sequence contained in the early denoising layer DiT. This represents the hidden dimension of the early denoising layer DiT, i.e., the vector dimension of each token. Physical prior features. The feature dimension is The two are inconsistent, therefore it is necessary to consider the physical prior features. Perform alignment processing.

[0054] In some specific embodiments, physical prior features can be incorporated using lightweight alignment operators. Mapping to the feature space of this denoising layer yields aligned features. As shown below:

[0055] in, The alignment process involves projecting onto the channel dimensions and incorporating physical prior features. The token sequence length is adjusted to match the length of the earlier denoising layer DiT. T This ensures that the alignment features are consistent with the feature dimensions of that layer.

[0056] Step S1032: The alignment feature is spliced ​​and fused with the original video features of the early denoising layer to obtain the fused feature.

[0057] Align features With the original video features of this layer Perform splicing and fusion along the sequence dimension:

[0058] in, It is a lightweight blending projector used to blend stitched features.

[0059] Step S1033: The fused features are weighted by learnable gating parameters, and the weighted fused features are superimposed on the original video features of the early denoising layer in the form of residual connections to obtain the injected features, and the injected features are input to the next level denoising layer.

[0060] Specifically, the fused features are controlled by learnable gating parameters. Weighting is performed, and the weighted fused features are superimposed onto the original video features of this layer using residual connections:

[0061] in, These are learnable scalar gating parameters used to adaptively adjust the injection strength of physical prior information. The final result is... As the output feature of this layer, it is input to the next level of denoising layer.

[0062] This application embodiment injects physical prior features into multiple cascaded early denoising layers, enabling the denoising network to establish correct physical cognition at the initial stage of video generation, thereby following the physical dynamic laws during the denoising process.

[0063] Step S104: Based on the denoising network after injecting the physical prior features, generate the target video corresponding to the video generation instruction.

[0064] Specifically, after multiple denoising layers, the denoising network outputs the video latent representation, which is then decoded by the decoder to obtain the final target video. Because physical prior features are injected in the early denoising layers, the generated target video is more consistent with the physical laws of the real world in terms of physical dynamic behavior, such as the continuity of fluid flow, the causality of object collisions, and the correctness of optical reflection.

[0065] In some specific embodiments, the preset video set is pre-constructed from the original video dataset through a two-stage data filtering process to ensure the quality and physical relevance of the training data. The specific two-stage data filtering process is as follows: Step a1: Obtain the original video dataset.

[0066] Specifically, the original video dataset contains multiple video samples and corresponding text descriptions for each video sample. The physical relevance and description quality of each video sample are evaluated based on these text descriptions. The evaluation can be achieved by scoring the text descriptions using a multimodal large language model (e.g., Qwen3-VL), with scoring dimensions including description quality (such as completeness and accuracy) and physical relevance (whether it involves physical phenomena such as mechanics, optics, thermal, and materials).

[0067] Step a2: Evaluate the physical relevance and description quality of each video sample based on the text description, eliminate video samples that are irrelevant to the physical phenomenon, and retain the top-ranked video samples as candidate video samples.

[0068] Specifically, based on the evaluation scores, video samples that are irrelevant to physical phenomena or have low descriptive quality are eliminated, and the top 10% of video samples are retained as candidate video samples. This stage is based solely on text filtering, has low computational cost, and can quickly eliminate a large number of irrelevant samples.

[0069] Step a3: For any candidate video sample, extract the key frames of the candidate video sample, input the key frames and the corresponding text descriptions into a multimodal alignment model, verify the consistency between the video frames and the text descriptions, eliminate inconsistent candidate video samples, and retain consistent candidate video samples as reference videos.

[0070] Specifically, for any candidate video sample, keyframes (e.g., several uniformly sampled frames) are extracted from the candidate video sample. The keyframes and their corresponding text descriptions are then input into a multimodal alignment model (e.g., Qwen3-VL) to verify the consistency between the video frames and the text descriptions. If the video frame content does not match the text description (e.g., the text description is "an apple is falling" but the video frame shows a still apple), the candidate video sample is discarded; if they match, it is retained.

[0071] Step a4: Construct the preset video set based on several reference videos.

[0072] Specifically, verified candidate video samples are retained as reference videos, and the preset video set is constructed based on several reference videos.

[0073] In one implementation, a pre-defined percentage (e.g., 90%) of candidate video samples that pass validation are retained. The resulting high-quality pre-defined video set can be used to train a physical perception video generation model or as a retrieval database for the inference phase.

[0074] In some specific embodiments, the training steps of the video generation model are as follows: Step b1: Obtain training samples.

[0075] Specifically, the training samples include video generation instructions, reference videos retrieved according to the video generation instructions, and corresponding real videos; the real videos are high-quality videos that are consistent with the content of the video generation instructions and are physically correct.

[0076] Step b2: Input the video generation instructions and the reference video into the video generation model, and jointly train the denoising network and the learnable query mechanism so that the predicted video output by the video generation model approaches the real video.

[0077] Specifically, the video generation instructions and reference video are input into the video generation model. During training, the denoising network and the learnable query mechanism (including query vectors, feedforward network parameters, projection matrix, and gating parameters) are jointly trained. The joint training is end-to-end training, meaning that the backbone parameters of the denoising network and the parameters of the query injection module are updated and optimized simultaneously and collaboratively.

[0078] This application embodiment minimizes the reconstruction loss (e.g., mean squared error loss or perceptual loss) between the predicted video and the real video, making the predicted video output by the video generation model approximate the real video. Compared to a scheme that only trains the query module while freezing the denoising network, joint training enables the model to more effectively internalize the retrieved physical priors, thereby significantly improving the physical consistency of the generated video.

[0079] Example 2: Corresponding to the above implementation of the physically consistent video generation method, this application also provides a physically consistent video generation system for executing the physically consistent video generation method described in the above embodiments. Figure 4 As shown, this video generation system for physical consistency includes: The reference video filtering module is used to filter reference videos from a preset video set that match the video prompt information in the video generation instruction in response to the video generation instruction; the reference videos contain physical dynamic laws corresponding to the prompt information; The physical prior feature generation module is used to extract video features from the reference video and extract physical prior information from the video features through a learnable query mechanism to generate physical prior features; the physical prior information is used to characterize the physical dynamic laws corresponding to the prompt information. The feature injection module is used to inject the physical prior features into the denoising network of the video generation model to guide the denoising network to follow the physical dynamic laws during the video denoising generation process. The target video generation module is used to generate a target video corresponding to the video generation instruction based on a denoising network injected with the physical prior features.

[0080] Optionally, the system further includes: a preset video set construction module, used to acquire an original video dataset, the original video dataset containing multiple video samples and text descriptions corresponding to each video sample; evaluating the physical relevance and description quality of each video sample based on the text descriptions, eliminating video samples unrelated to physical phenomena, and retaining the top-ranked video samples as candidate video samples; for any candidate video sample, extracting keyframes of the candidate video sample, inputting the keyframes and corresponding text descriptions into a multimodal alignment model, verifying the consistency between video frames and text descriptions, eliminating inconsistent candidate video samples, and retaining consistent candidate video samples as reference videos; and constructing the preset video set based on several reference videos.

[0081] Optionally, the physical prior feature generation module is further configured to use a learnable query vector as a query, use the video features as key vectors and value vectors, perform cross-attention calculation, extract physical dynamic information related to the physical dynamic law from the video features, and obtain attention features; perform nonlinear transformation and residual connection processing on the attention features through a feedforward network to obtain refined features; and project the refined features onto the physical prior feature space to obtain the physical prior features.

[0082] Optionally, the feature injection module is further configured to, for any of the multiple cascaded early denoising layers, align the physical prior features according to the feature dimension of the early denoising layer to obtain aligned features; the aligned features have the same feature dimension as the early denoising layer; concatenate and fuse the aligned features with the original video features of the early denoising layer to obtain fused features; weight the fused features using learnable gating parameters, and superimpose the weighted fused features onto the original video features of the early denoising layer using a residual connection to obtain injected features, and input the injected features into the next level denoising layer.

[0083] Optionally, the reference videos in the preset video set are classified according to the category of physical phenomena. Reference videos under the same category of physical phenomena have similar physical characteristics, and the video features of the reference videos are pre-extracted by the video encoder and stored in the form of feature vectors.

[0084] Optionally, the reference video filtering module is further configured to semantically encode the video prompt information to obtain prompt semantic features; calculate the similarity between the prompt semantic features and the video features of each video sample in the preset video set; and select at least one video sample from the preset video set as the reference video based on the similarity.

[0085] Optionally, the system further includes: a video generation model training module, used to acquire training samples, the training samples including video generation instructions, reference videos retrieved according to the video generation instructions, and corresponding real videos; inputting the video generation instructions and reference videos into the video generation model, jointly training the denoising network and the learnable query mechanism, so that the predicted video output by the video generation model approaches the real video.

[0086] The video generation system for physical consistency provided in the above embodiments of this application and the video generation method for physical consistency provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0087] This application also provides a computer device for performing the above-described physically consistent video generation method. Please refer to... Figure 5 This illustrates a schematic diagram of a computer device provided by some embodiments of this application. For example... Figure 5 As shown, the computer device 5 includes: a processor 500, a memory 501, a bus 502, and a communication interface 503. The processor 500, the communication interface 503, and the memory 501 are connected via the bus 502. The memory 501 stores a computer program that can run on the processor 500. When the processor 500 runs the computer program, it executes the video generation method for physical consistency provided in the foregoing embodiments of this application.

[0088] The memory 501 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 503 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0089] Bus 502 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 501 is used to store programs. After receiving an execution instruction, the processor 500 executes the program. The physically consistent video generation method disclosed in the foregoing embodiments can be applied to the processor 500, or implemented by the processor 500.

[0090] The processor 500 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 500 or by instructions in software form. The processor 500 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 501. The processor 500 reads the information in memory 501 and, in conjunction with its hardware, completes the steps of the above method.

[0091] The computer device provided in this application embodiment and the video generation method for physical consistency provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, operate or implement.

[0092] This application also provides a computer-readable storage medium corresponding to the physically consistent video generation method provided in the foregoing embodiments. Please refer to... Figure 6 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the video generation method for physical consistency provided in any of the foregoing embodiments.

[0093] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0094] The computer-readable storage medium provided in the above embodiments of this application and the video generation method for physical consistency provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0095] It should be noted that: Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0096] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0097] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0098] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for physical consistency oriented video generation, characterized in that, The method includes: In response to a video generation command, a reference video matching the video prompt information in the video generation command is selected from a preset video set; the reference video contains physical dynamic laws corresponding to the prompt information. The video features of the reference video are extracted, and physical prior information is extracted from the video features through a learnable query mechanism to generate physical prior features; the physical prior information is used to characterize the physical dynamic laws corresponding to the prompt information. The physical prior features are injected into the denoising network of the video generation model to guide the denoising network to follow the physical dynamic laws during the video denoising generation process. Based on the denoising network injected with the physical prior features, a target video corresponding to the video generation instruction is generated.

2. The method of claim 1, wherein, Before selecting reference videos from a preset video set that match the video prompt information in the video generation instruction, the method further includes: Obtain the original video dataset, which contains multiple video samples and a corresponding text description for each video sample; The physical relevance and description quality of each video sample are evaluated based on the text description. Video samples that are not related to physical phenomena are eliminated, and the top-ranked video samples are retained as candidate video samples. For any candidate video sample, extract the key frames of the candidate video sample, input the key frames and the corresponding text description into a multimodal alignment model, verify the consistency between the video frames and the text description, eliminate inconsistent candidate video samples, and retain consistent candidate video samples as reference videos. The preset video set is constructed based on several reference videos.

3. The method according to claim 1 or 2, characterized in that, Physical prior information is extracted from the video features through a learnable query mechanism to generate physical prior features, including: Using a learnable query vector as the query, and the video features as the key vector and value vector, cross-attention calculation is performed to extract physical dynamic information related to the physical dynamic law from the video features, thus obtaining attention features; The attention features are subjected to nonlinear transformation and residual connection processing through a feedforward network to obtain refined features. The refined features are projected onto the physical prior feature space to obtain the physical prior features.

4. The method according to claim 1 or 2, characterized in that, The denoising network comprises multiple cascaded early denoising layers, and the physical prior features are injected into the denoising network of the video generation model, including: For any of the multiple cascaded early denoising layers, the physical prior features are aligned according to the feature dimension of the early denoising layer to obtain aligned features; the aligned features are consistent with the feature dimension of the early denoising layer. The alignment feature is spliced ​​and fused with the original video feature of the early denoising layer to obtain the fused feature; The fused features are weighted by learnable gating parameters, and the weighted fused features are superimposed on the original video features of the early denoising layer in the form of residual connections to obtain the injected features, which are then input into the next level of denoising layer.

5. The method of claim 1, wherein, The reference videos in the preset video set are classified according to the category of physical phenomena. Reference videos under the same category of physical phenomena have similar physical characteristics, and the video features of the reference videos are pre-extracted by the video encoder and stored in the form of feature vectors.

6. The method of claim 1 or 2, wherein, The process involves selecting reference videos from a preset video set that match the video prompt information in the video generation instruction, including: The video prompt information is semantically encoded to obtain prompt semantic features; Calculate the similarity between the semantic features of the prompt and the video features of each video sample in the preset video set; Based on the similarity, at least one video sample is selected from the preset video set as the reference video.

7. The method according to claim 1 or 2, characterized in that, The method also includes a training step for the video generation model: Obtain training samples, which include video generation instructions, reference videos retrieved according to the video generation instructions, and corresponding real videos; The video generation instructions and reference video are input into the video generation model, and the denoising network and the learnable query mechanism are jointly trained so that the predicted video output by the video generation model approaches the real video.

8. A video generation system oriented towards physical consistency, characterized in that, The system includes: The reference video filtering module is used to filter reference videos from a preset video set that match the video prompt information in the video generation instruction in response to the video generation instruction; the reference videos contain physical dynamic laws corresponding to the prompt information; The physical prior feature generation module is used to extract video features from the reference video and extract physical prior information from the video features through a learnable query mechanism to generate physical prior features; the physical prior information is used to characterize the physical dynamic laws corresponding to the prompt information. The feature injection module is used to inject the physical prior features into the denoising network of the video generation model to guide the denoising network to follow the physical dynamic laws during the video denoising generation process. The target video generation module is used to generate a target video corresponding to the video generation instruction based on a denoising network injected with the physical prior features.

9. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory storing computer instructions, and the processor executing the computer instructions to perform the video generation method for physical consistency as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the video generation method for physical consistency as described in any one of claims 1 to 7.