Model training method and apparatus, device, medium, and product

Abstract

The application discloses a model training method and device, equipment, medium and product, and relates to the technical field of artificial intelligence. The method comprises the following steps: constructing a static dictionary based on a reference video, the static dictionary comprising a mapping relationship between reference video frames in the reference video and physical expert models, wherein the reference video conforms to a target physical law, and the physical expert models are used to provide prior information when verifying whether a video frame conforms to a physical law; generating a video based on prompt information through a video generation model to obtain a generated video, wherein the prompt information is used to prompt the generation of a video conforming to the target physical law; determining a physical expert model corresponding to a to-be-evaluated video frame from the static dictionary based on a corresponding relationship between the reference video frames in the reference video and generated video frames in the generated video, wherein the to-be-evaluated video frame belongs to the generated video frames; and training the video generation model through the physical expert models corresponding to the to-be-evaluated video frames.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, device, medium, and product. Background Technology

[0002] The video generation model can generate video footage that matches the user's description by iteratively denoising the noise distribution in the latent space or fitting the velocity field based on text prompts or initial conditions.

[0003] In related technologies, when training a video generation model, it is necessary to determine the corresponding expert model based on the content of the generated video frames, calculate the error based on the expert model, and perform backpropagation to update the model parameters.

[0004] However, due to the poor quality of video frames generated by the video generation model in the early stages of training, the expert model determined based on the poor-quality generated content is prone to mismatch or unsuitability for the current generation state, resulting in poor training effect or even training failure. Summary of the Invention

[0005] This application provides a model training method, apparatus, device, medium, and product. The technical solution provided by this application includes the following aspects.

[0006] On the one hand, embodiments of this application provide a model training method, the method comprising: A static dictionary is constructed based on a reference video. The static dictionary contains the mapping relationship between reference video frames in the reference video and a physics expert model. The reference video conforms to the target physical laws, and the physics expert model is used to provide prior information when verifying whether the video frames conform to the physical laws. Based on the prompt information, a video is generated using a video generation model to obtain a generative video. The prompt information is used to prompt the generation of a video that conforms to the target physical laws. Based on the correspondence between the reference video frames in the reference video and the generative video frames in the generative video, the physical expert model corresponding to the video frame to be evaluated is determined from the static dictionary, and the video frame to be evaluated belongs to the generative video frame. The video generation model is trained using the physical expert models corresponding to each of the video frames to be evaluated.

[0007] On the other hand, embodiments of this application provide a model training apparatus, the apparatus comprising: The construction module is configured to build a static dictionary based on a reference video. The static dictionary contains the mapping relationship between reference video frames in the reference video and a physics expert model. The reference video conforms to the target physical laws, and the physics expert model is used to provide prior information when verifying whether the video frames conform to the physical laws. The generation module is configured to generate a generative video based on a prompting information and a video generation model. The prompting information is used to prompt the generation of a video that conforms to the target physical laws. The determination module is configured to determine the physical expert model corresponding to the video frame to be evaluated from the static dictionary based on the correspondence between the reference video frame in the reference video and the generative video frame in the generative video, wherein the video frame to be evaluated belongs to the generative video frame. The training module is configured to train the video generation model using the physical expert models corresponding to each of the video frames to be evaluated.

[0008] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described model training method.

[0009] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described model training method.

[0010] On the other hand, embodiments of this application provide a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor reads from the computer-readable storage medium and executes the computer program to implement the above-described model training method.

[0011] The technical solution provided in this application can bring the following beneficial effects: By constructing a static dictionary based on reference videos, expert models are mapped and bound to reference video frames. Then, based on the correspondence between reference video frames and generative video frames, the expert model is determined from the static dictionary. This ensures that the expert model's matching is based on the temporal correspondence between frames, rather than real-time analysis or evaluation of the generative video frame content. Since this matching process is completely independent of the quality of the generative video, it reduces the possibility of incorrect or unsuitable expert model selection due to poor-quality generated content in the early stages of training. This ensures the accuracy of the supervision signal during training, improving training stability and convergence efficiency. Attached Figure Description

[0012] Figure 1This is a flowchart illustrating a model training method provided in an illustrative embodiment of this application; Figure 2 This is a schematic diagram illustrating a static dictionary construction process provided in an illustrative embodiment of this application; Figure 3 This is a flowchart illustrating a static dictionary construction process provided in an illustrative embodiment of this application; Figure 4 This is a schematic diagram of the physical phase division process provided in an illustrative embodiment of this application; Figure 5 This is a flowchart illustrating the physical expert model determination process provided in an illustrative embodiment of this application; Figure 6 This is a schematic diagram of a mask matrix provided in an illustrative embodiment of this application; Figure 7 This is a schematic diagram of the second frame index determination process provided in an illustrative embodiment of this application; Figure 8 This is a schematic diagram of the second frame index determination process provided in another illustrative embodiment of this application; Figure 9 This is a schematic diagram of a model training process provided in an illustrative embodiment of this application; Figure 10 This is a schematic diagram of the structure of a model training device provided in an illustrative embodiment of this application; Figure 11 This is a structural block diagram of a computer device provided in an illustrative embodiment of this application. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0014] Illustrative embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following illustrative embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0015] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0016] It should be understood that although the terms “first,” “second,” etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word “if” as used herein may be interpreted as “when…” or “in response to determination.”

[0017] It should be noted that this application may display a prompt interface, pop-up window, or output voice prompt information before collecting user, processor, computer device, and other related data, and during the process of collecting user-related data. This prompt interface, pop-up window, or voice prompt information is used to inform the user that their related data is being collected. This ensures that the application only begins executing the steps related to collecting user-related data after receiving confirmation from the user regarding the prompt interface or pop-up window; otherwise (i.e., without receiving confirmation from the user), the steps related to collecting user-related data end, meaning no user-related data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the laws, regulations, and standards of relevant countries and regions.

[0018] First, the terms used in the embodiments of this application will be introduced.

[0019] Video generation models: Video generation models are neural network models to be trained, used to generate corresponding video content based on prompts (such as text descriptions or the first frame image). Video generation models can be neural network models based on diffusion models or flow matching. Diffusion models iteratively denoise the noise distribution in the latent space to gradually generate clear video images; flow matching models learn a velocity field to gradually evolve the noise distribution in the latent space to the target video distribution.

[0020] Latent space features: Latent space features refer to the feature representation obtained by mapping the original video frame from a high-dimensional pixel space to a low-dimensional latent space. They are used to characterize the spatiotemporal geometric and semantic features of a reference video frame, i.e., the physical evolution-related attributes such as the position, shape, and motion trends of objects within the video frame. The original video frame resides in a high-dimensional pixel space, containing a large amount of pixel information such as lighting changes, background texture, and object color. Directly processing this information in the high-dimensional pixel space is not only computationally intensive but also makes it difficult to extract essential features related to physical laws. Computer devices extract latent space features to map the original video frame to a low-dimensional latent space, removing redundant information unrelated to physical laws, such as lighting and texture, while retaining key information related to motion state, object shape changes, and position changes, thus obtaining a more compact and semantic feature representation.

[0021] Latent State: The latent state is the intermediate state representation in the latent space of the video generation model during the generative video process. The video generation model does not directly output the final video frame when generating each frame, but rather generates it step by step through multiple iterations. The intermediate result of each iteration is the latent state. The latent state contains the encoded information of the video frame content by the video generation model at the current generation stage and is the basis for subsequent velocity field calculations. In the stream matching framework, the evolution of the latent state over time constitutes a continuous path from the noise distribution to the target video distribution.

[0022] Velocity field: The velocity field is a core concept in the stream matching framework, used to describe the direction and speed of motion of data points in the latent space at different time steps during the generation process of a video frame. In the stream matching framework, the training objective of the video generation model is to minimize the difference between its predicted velocity field and the target velocity field. The closer the predicted velocity field is to the reference velocity field, the smaller the stream matching loss, indicating that the video generated by the model is physically closer to the reference video.

[0023] Physics expert models are pre-trained neural network models, each corresponding to different physical laws. For example, a gravity expert model provides a supervisory signal for the motion of an object under gravity, a collision expert model provides a supervisory signal for the momentum transfer and energy dissipation during a collision, and a bounce expert model provides a supervisory signal for the bounce motion of an object after colliding with a contact surface. During model training, the physics expert model generates a reference velocity field based on the video output of the video generation model. This reference velocity field serves as the fitting target for the video generation model's predicted velocity field, guiding the video generation model's output to converge in the correct physical direction.

[0024] In related technologies, during the training of video generation models, it is necessary to determine the corresponding expert model based on the content of the generated video frames, calculate the error based on the expert model, and perform backpropagation to update the model parameters. However, since the video frame content generated by the video generation model in the early stages of training is of poor quality, the expert model determined based on this inferior content is prone to mismatches or is not suitable for the current generation state, resulting in poor training performance or even training failure.

[0025] To address the aforementioned issues, this application provides a model training method. This method constructs a static dictionary based on reference videos, mapping and binding expert models to reference video frames. Then, based on the correspondence between reference video frames and generative video frames, the expert model is determined from the static dictionary. This ensures that the expert model's matching is based on the temporal correspondence between frames, rather than real-time analysis or evaluation of the generative video frame content. Since this matching process is completely independent of the quality of the generative video, it reduces the possibility of incorrect or unsuitable expert model selection due to poor-quality generated content in the early stages of training. This ensures the accuracy of the supervision signal during training, improving training stability and convergence efficiency.

[0026] The model training method provided in this application can be applied to computer devices with model training capabilities. This computer device can be a server, which can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The cloud server supports, but is not limited to, basic cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), big data, and artificial intelligence platforms.

[0027] In some application scenarios, the computer device can also be a terminal device with model training capabilities, such as a graphics workstation or a high-performance personal computer (PC). These devices are typically equipped with a graphics processing unit (GPU) or a neural processing unit (NPU) for deep learning training, possessing the computing resources required to perform model training. In the following embodiments, for ease of description, the model training method is illustrated by an example of execution by a computer device.

[0028] like Figure 1 As shown, a flowchart of a model training method provided in an illustrative embodiment of this application is illustrated. The method can be executed by the aforementioned computer device and includes at least one of the following steps.

[0029] Step 110: Construct a static dictionary based on the reference video. The static dictionary contains the mapping relationship between reference video frames and physics expert models. The reference video conforms to the target physical laws, and the physics expert models are used to provide prior information when verifying whether the video frames conform to the physical laws.

[0030] In some embodiments, the computer device first acquires a reference video and then constructs a static dictionary based on the reference video.

[0031] The reference video is a video that conforms to the physical laws of the target, such as the laws governing an object's fall due to gravity, its rebound after a collision, its uniform motion, or its accelerated motion. The reference video contains multiple reference video frames arranged along the time axis. Each reference video frame records the object's position, velocity, attitude, and other state information at a specific moment. The changes in this state information over time constitute the evolution of the physical laws.

[0032] A static dictionary is a pre-built data structure that remains frozen during model training. This data structure stores the mapping between reference video frames in the reference video and the physics expert model.

[0033] Physics expert models are pre-trained neural network models. Different physics expert models correspond to different physical laws and are used to provide supervision signals that conform to the target physical laws during the training process of the video generation model. This helps to verify whether the video frames generated by the video generation model conform to the target physical laws and guides the output of the student model to converge in the correct physical direction.

[0034] Because the reference video conforms to the target's physical laws, the changes in motion states presented in each video frame realistically reflect the evolution of these physical laws. By analyzing the motion states of each frame in the reference video, computer equipment can determine the physical laws represented by each frame, and then establish a mapping relationship between each frame and its corresponding physics expert model, storing this mapping relationship in a static dictionary. Since the static dictionary remains frozen during training, its mapping relationship does not change with the training process, thus providing stable and consistent physical prior guidance for the video generation model.

[0035] Once constructed, the static dictionary can be used as a lookup table for subsequent training. When it is necessary to determine the physics expert model corresponding to a given generative video frame, the computer can quickly match the corresponding physics expert model by looking up the static dictionary. This lookup process does not involve real-time evaluation of the current generative video quality of the video generation model; instead, it directly matches based on the pre-built mapping relationship. This ensures accurate adaptation between the expert model and the physics stage while avoiding the computational overhead and instability caused by dynamic decision-making.

[0036] To illustrate, assume the reference video is a sphere freely falling and bouncing, containing 100 frames. A computer analyzes these 100 frames, determining that frames 1-60 represent the falling phase (with continuously increasing speed), frames 61-65 represent the collision phase (with a sudden change in speed within a very short time), and frames 66-100 represent the bounce phase (with speed reversing and gradually decreasing). Based on this analysis, the static dictionary records: frame indices for frames 1-60 are mapped to the gravity expert model, frame indices for frames 61-65 are mapped to the collision expert model, and frame indices for frames 66-100 are mapped to the bounce expert model. Therefore, multiple reference video frames within the same physical phase correspond to the same physical expert model, while different physical phases correspond to different physical expert models.

[0037] Optionally, the target physical law may include the laws of free fall, projectile motion, collision and rebound, uniform linear motion, and uniformly accelerated motion. This application does not limit the specific type of the target physical law.

[0038] Optionally, the physics expert model may include a gravity expert model, a collision expert model, a bounce expert model, a friction expert model, an air resistance expert model, etc. Different physics expert models can be used independently or in combination to adapt to the coupling of multiple physical laws in complex physical scenarios. This application does not limit the specific type and combination of physics expert models in its embodiments.

[0039] Optionally, in the static dictionary, one reference video frame can correspond to one physical expert model or multiple physical expert models. This application does not impose such limitations in its embodiments.

[0040] Step 120: Based on the prompt information, generate a video using a video generation model to obtain a generative video. The prompt information is used to prompt the generation of a video that conforms to the target physical laws.

[0041] In some embodiments, the computer device acquires a prompt message and, based on the prompt message, generates a video using a video generation model to obtain a generative video.

[0042] A video generation model is an object model that needs to be trained to generate videos that conform to the physical laws of the target. Optionally, the video generation model can be a neural network model based on a diffusion model or flow matching; this application embodiment does not limit this.

[0043] The prompts instruct the video generation model to generate videos that conform to the target physical laws. The prompts correspond to the target physical laws conforming to the reference video, creating a content-based correspondence between the generated video and the reference video. This allows the model to look up the corresponding physics expert model from the static dictionary based on the frame correspondence in subsequent steps.

[0044] To illustrate, if the reference video is a video of a ball falling and bouncing, the prompt message can be a text description of "an elastic ball falls from a height and bounces", or an image of the ball in its initial position can be provided as the first frame image. The video generated by the video generation model based on this prompt message is also a motion video of the ball falling and bouncing.

[0045] The computer device generates video based on prompts corresponding to the reference video, so that there is a correspondence between the physical evolution process of the generative video and the physical evolution process of the reference video. This allows the corresponding physical expert model to be accurately found from the static dictionary through the frame correspondence.

[0046] Step 130: Based on the correspondence between reference video frames in the reference video and generative video frames in the generative video, determine the physical expert model corresponding to the video frame to be evaluated from the static dictionary. The video frame to be evaluated belongs to the generative video frame.

[0047] In some embodiments, after acquiring generative video, the computer device determines the physical expert model corresponding to the video frame to be evaluated from a static dictionary based on the correspondence between reference video frames in the reference video and generative video frames in the generative video.

[0048] Among them, the video frames to be evaluated belong to generative video frames, which are generative video frames that participate in evaluation and training in generative videos.

[0049] Optionally, the video frames to be evaluated can be all generated video frames or only some generated video frames. When all generated video frames are evaluated, the video generation model can obtain supervision signals for each frame, which helps improve generation accuracy. When only some generated video frames are evaluated, since only the selected frames participate in loss calculation and gradient backpropagation, the computational load and memory usage are effectively reduced, making training in long video scenarios feasible in engineering.

[0050] The embodiments of this application do not restrict the selection method of the video frames to be evaluated, and the computer equipment can flexibly determine the selection method according to the actual training needs.

[0051] The correspondence refers to the temporal correspondence between reference video frames and generated video frames. Since the prompt information corresponds to the physical laws of the reference video, and the generated video and the reference video have the same or similar physical evolution process, the correspondence between reference video frames and generated video frames can be determined based on frame number or frame ratio, etc.

[0052] In some embodiments, the computer device first determines the video frame to be evaluated from the generative video, then determines the reference video frame corresponding to the video frame to be evaluated based on the video frame to be evaluated and the correspondence between the reference video frame and the generative video frame, and finally looks up the corresponding physical expert model from the static dictionary based on the index of the reference video frame.

[0053] Because the static dictionary records the mapping relationship between reference video frame indices and physics expert models, the corresponding physics expert model can be determined using the reference video frame. This lookup process does not involve online evaluation of the current generation quality of the video generation model; instead, it directly matches based on the pre-built mapping relationship, thus avoiding the problem of incorrect or unsuitable expert model selection due to poor quality generated content in the early stages of training.

[0054] Optionally, the physical expert model corresponding to a single video frame to be evaluated, as determined by the computer device, can be one or more, and this application embodiment does not limit this.

[0055] Step 140: Train the video generation model using the physical expert model corresponding to each video frame to be evaluated.

[0056] In some embodiments, after the computer device determines the physical expert model corresponding to each video frame to be evaluated, it trains the video generation model using each physical expert model.

[0057] Physics expert models can provide video generation models with supervision signals that conform to the physical laws of the target. These supervision signals serve as the fitting target for the output of the video generation model, guiding the model to update its parameters during training so that the subsequently generated video frames gradually approach the reference video in terms of physical laws.

[0058] Indicatively, the computer device uses the physical expert model corresponding to each video frame to be evaluated as a supervisory signal provider to guide the video generation model in generating results on the video frames to be evaluated.

[0059] Optionally, the above training process can be iterated multiple times until the video generated by the video generation model meets the preset requirements in terms of physical laws. In each round of training, the video frames to be evaluated can be randomly selected again to cover different time positions in the generative video, so that the video generation model can obtain effective supervision signals at each physical stage.

[0060] In summary, by constructing a static dictionary based on reference videos and mapping expert models to reference video frames, and then determining the expert model from the static dictionary based on the correspondence between reference video frames and generative video frames, the matching of expert models is based on the temporal correspondence between frames, rather than real-time analysis or evaluation of the content of generative video frames. Since this matching process is completely independent of the quality of the generative video, it reduces the possibility of incorrect or unsuitable expert model selection due to poor quality generated content in the early stages of training. This ensures the accuracy of the supervision signal during training and improves training stability and convergence efficiency.

[0061] In some embodiments, the computer device generates a generative video by using a video generation model based on the text information describing the physical laws of the target in the prompt message, or the first frame image of the video to be generated, or the text information describing the physical laws of the target and the first frame image of the video to be generated.

[0062] The text information describes the physical laws of the target, such as "a sphere falls freely from a height and bounces back" or "an object slides at a constant speed on an inclined plane." The video generation model generates corresponding video content based on the text description. The first frame refers to the starting frame of the video to be generated. The video generation model continues to generate subsequent frame sequences based on the first frame, which provides the initial state information of the generative video.

[0063] Optionally, the prompt information may only include text information describing the target physical laws, and the video generation model generates complete video content based on the text description; it may also only include the first frame image of the video to be generated, and the video generation model continues to generate the subsequent frame sequence based on the first frame image; or it may include both text information and the first frame image, and the video generation model combines both to generate the video. When the prompt information includes both text information and the first frame image, the video generation model can obtain richer generation guidance information, thereby generating a video that better conforms to the target physical laws. This application does not limit the specific method of the prompt information.

[0064] To illustrate, if the target physical law is that a ball falls and bounces, the text information can be "an elastic ball falls from a height and bounces," the first frame image can be an image of the ball at the initial height, and the video generation model generates a motion video of the ball falling and bouncing based on the above prompt information.

[0065] By supporting text information and / or the first frame image as prompts, the video generation model can generate corresponding video content based on text descriptions and / or initial images, ensuring that the generated video and the reference video have a content correspondence, which facilitates the accurate determination of the physics expert model from the static dictionary through frame correspondence.

[0066] Optionally, the prompt information may also include total frame count information, which indicates the total number of frames in the video to be generated, and the indicated total frame count is the same as the total number of frames in the reference video.

[0067] By using the total frame count information, the computer device can control the length of the video generated by the video generation model, so that the total number of frames in the generated video is consistent with the total number of frames in the reference video. This facilitates the establishment of a correspondence between the reference video frames and the generated video frames in subsequent steps based on the frame sequence number or frame ratio.

[0068] To illustrate, if the reference video contains 100 frames, the total frame count information in the prompt indicates that the generative video has a total of 100 frames, and the video generation model generates a generative video containing 100 frames accordingly.

[0069] Optionally, the total frame count information in the prompt message can be used alone or in combination with the aforementioned text information or the first frame image. This application embodiment does not impose any limitations on this.

[0070] By including the total number of frames in the prompt information, and ensuring that this total number of frames is the same as the total number of frames in the reference video, the generated video and the reference video maintain the same length. This facilitates the establishment of a one-to-one correspondence between the reference video frames and the generated video frames based on the frame sequence number in subsequent steps, eliminating the need for additional alignment processing.

[0071] Optionally, the reference video is generated based on the prompt information, or it is simulated in the physics engine.

[0072] In this context, generating reference videos based on prompts involves a computer device inputting prompts describing the physical laws of a target object (e.g., "a ball falls freely from a height and bounces back") into a pre-trained teacher model. The teacher model then generates a reference video that conforms to these physical laws based on the prompts. The teacher model can be a video generation model based on diffusion models or flow matching, possessing strong generation capabilities and able to generate high-quality video content based on text descriptions.

[0073] In physics engines, simulating and generating reference videos refers to computer devices using a physics engine to simulate the motion of objects based on set physical parameters (such as gravitational acceleration, elastic coefficient, friction coefficient, initial velocity, etc.), and rendering and generating reference videos that conform to the target physical laws. Physics engines can accurately simulate the trajectory of objects under the influence of physical laws, such as a ball falling under gravity, colliding with the ground, and bouncing after the collision. Because physics engines perform calculations based on precise physical equations, the reference videos they generate have high physical accuracy and can provide reliable prior information about the target physical laws.

[0074] Optionally, the physics engine may include different types such as rigid body physics engines, soft body physics engines, and fluid physics engines, each suitable for simulating different physical scenarios. This application embodiment does not limit the specific type of physics engine.

[0075] Optionally, when using a physics engine to simulate and generate reference videos, the computer device can also generate reference videos under different conditions by adjusting physical parameters (such as the magnitude of gravitational acceleration, the magnitude of the elastic coefficient, the initial position and velocity of the object, etc.) to cover richer physical scenes and meet different training needs.

[0076] This application does not limit the specific source of the reference video in its embodiments.

[0077] By supporting two methods for obtaining reference videos—generating them based on prompts or simulating them in the physics engine—the acquisition of reference videos becomes more flexible. Generating them based on prompts is efficient and requires no manual intervention, while generating them based on physics engine simulation is accurate in physics and has controllable physical parameters. The two methods can be flexibly selected according to actual needs, providing a reliable source of reference videos for the subsequent construction of static dictionaries.

[0078] Since the reference video frames are data in a high-dimensional pixel space, directly analyzing and identifying their physical laws presents challenges such as high workload and difficulty in feature extraction. To accurately and efficiently determine the physical expert model corresponding to each reference video frame, the computer first extracts features from the reference video frames, and then analyzes and processes these extracted features to construct an accurate static dictionary.

[0079] In some embodiments, the computer device extracts features from each reference video frame in the reference video to obtain the latent space features of each reference video frame. The latent space features are used to characterize the spatiotemporal geometric and semantic features of the reference video frame.

[0080] Among them, latent space features refer to the feature representation obtained by mapping the original video frame from a high-dimensional pixel space to a low-dimensional latent space. These features are used to characterize the spatiotemporal geometric and semantic features of the reference video frame, namely, the physical evolution-related attributes such as the position, shape, and motion trends of objects in the video frame. The original video frame contains a large amount of pixel information, including a significant amount of redundant information (such as lighting changes and background textures). Directly processing this information is not only computationally intensive but also makes it difficult to extract the essential features related to physical laws.

[0081] Computer devices can obtain more compact and semantic feature representations by mapping raw video frames to latent space, while retaining key information related to motion state and changes in object shape in the video frames.

[0082] Schematic, the process of feature extraction of a reference video frame by a computer device can be represented by the following formula (1):

[0083] in, The input reference video frame, For feature extraction encoder, These are the latent space features obtained after extraction.

[0084] Schematic illustration: A computer device can use a pre-trained visual encoder to process each reference video frame, encoding each reference video frame into a low-dimensional feature vector, which is the latent space feature of that frame. The latent space features of each reference video frame are arranged along the time axis to form a latent space feature sequence.

[0085] In some embodiments, the computer device extracts features from each reference video frame using a pre-trained video encoder to obtain the latent space features of each reference video frame.

[0086] In other embodiments, the computer device can perform feature extraction based on the difference information between a reference video frame and its adjacent frames. For example, the computer device can calculate a difference image between adjacent reference video frames, input the difference image into a video encoder for feature extraction, to highlight motion change information between frames, thereby obtaining the latent space features of each reference video frame. This application does not limit this approach.

[0087] In some embodiments, the computer device determines the physical expert model corresponding to each reference video frame based on latent space features and constructs a static dictionary.

[0088] Optionally, the computer device analyzes the latent space features of each reference video frame, determines the physical expert model corresponding to each frame based on the motion state and physical laws reflected by the latent space features, and constructs a static dictionary based on each reference video frame and its corresponding physical expert model. The specific methods by which the computer device analyzes the latent space features can take various forms.

[0089] like Figure 2The figure illustrates a schematic diagram of a static dictionary construction process provided in an illustrative embodiment of this application. In the figure, reference video 201 includes multiple reference video frames arranged along a time axis. The figure exemplarily shows a first reference video frame 202, a second reference video frame 203, a third reference video frame 204, and a fourth reference video frame 205. A computer device performs feature extraction on each reference video frame to determine the latent space features corresponding to each reference video frame, including a first latent space feature 206 corresponding to the first reference video frame 202, a second latent space feature 207 corresponding to the second reference video frame 203, a third latent space feature 208 corresponding to the third reference video frame 204, and a fourth latent space feature 209 corresponding to the fourth reference video frame 205.

[0090] After determining the latent space characteristics, the computer device determines the physical expert model corresponding to each reference video frame based on the latent space characteristics of each reference video frame. Specifically, the first reference video frame 202 and the second reference video frame 203, which have similar latent space characteristics, correspond to the same first physical expert model 210; the third reference video frame 204 corresponds to the second physical expert model 211; and the fourth reference video frame 205 corresponds to the third physical expert model 212. Based on each reference video frame and its corresponding physical expert model, the computer device constructs a static dictionary 213. The static dictionary 213 stores the mapping relationship between reference video frames and physical expert models. As shown in the figure, the first reference video frame 202 and the second reference video frame 203 are both mapped to the first physical expert model 210, the third reference video frame 204 is mapped to the second physical expert model 211, and the fourth reference video frame 205 is mapped to the third physical expert model 212.

[0091] Optionally, the computer device can also extract other features to characterize the physical motion state and determine the corresponding physical expert model based on these features.

[0092] Schematic, the computer device can extract optical flow features from the reference video frame, which characterize the motion direction and velocity of pixels between adjacent frames, and determine the corresponding physical expert model based on the optical flow features; or, the computer device can extract key point trajectory features from the reference video frame, which characterize the positional change trajectory of specific key points on an object between different frames, and determine the corresponding physical expert model based on the key point trajectory features; or, the computer device can extract frequency domain features from the reference video frame, which characterize the energy distribution of the video frame at different frequency components, and determine the corresponding physical expert model based on the frequency domain features.

[0093] This application does not limit the specific types of features or the specific methods for determining physical expert models based on features.

[0094] By mapping reference video frames to the latent space for feature extraction, computational complexity is reduced and redundant information is removed, improving the efficiency and accuracy of expert model matching. This results in the construction of a static dictionary with accurate mapping relationships, providing a stable and reliable physical prior supervision foundation for subsequent training.

[0095] Because the physical laws in reference videos typically persist over a period of time, rather than existing only in a single frame—for example, the process of an object falling from a height might last for dozens of frames, and the collision and bounce process might also last for multiple frames—computer devices can group reference video frames. Frames in the same physical state and representing the same physical laws can be classified into the same physical stage, and then a corresponding physics expert model can be matched to each physical stage. Since all reference video frames within the same physical stage correspond to the same physics expert model, the computer device does not need to determine the expert model frame by frame, but rather in batches on a physical stage basis. This improves the efficiency of static dictionary construction and reduces processing time.

[0096] like Figure 3 The diagram illustrates a flowchart of a static dictionary construction process provided in an illustrative embodiment of this application. This process can be executed by the aforementioned computer device and includes at least one of the following steps: Step 301: Extract features from each reference video frame in the reference video to obtain the latent space features of each reference video frame. The latent space features are used to characterize the spatiotemporal geometric and semantic features of the reference video frame.

[0097] Please refer to the feature extraction methods mentioned in the above embodiments; the embodiments of this application will not be repeated here.

[0098] Step 302: Perform feature clustering on the feature sequence composed of the latent space features of each reference video frame to obtain the feature clustering result.

[0099] In some embodiments, after obtaining the latent space features of each reference video frame, the computer device arranges these latent space features in chronological order to form a latent space feature sequence, and performs feature clustering on the feature sequence to obtain the feature clustering result.

[0100] Feature clustering refers to a classification method that groups feature vectors that are close in distance in the feature space into the same category. Reference video frames corresponding to latent space features belonging to the same category have similar spatiotemporal geometric and semantic features, that is, they are in similar physical motion states. The feature clustering result refers to the category classification information output by the feature clustering algorithm, which is used to indicate the category to which the latent space features of each reference video frame belong.

[0101] Optionally, feature clustering can be implemented using various clustering algorithms, such as K-means clustering, density-based clustering, or hierarchical clustering. This application does not limit the specific type of clustering algorithm.

[0102] Optionally, the feature clustering results may include the category label corresponding to each latent space feature, or the set of latent space features contained in each category, thereby indicating the classification result of the reference video frame.

[0103] Step 303: Based on the feature clustering results, the reference video frame is divided into at least one physical stage, wherein the physical laws represented by the reference video frames in the same physical stage are the same.

[0104] In some embodiments, the computer device divides a reference video frame into at least one physical phase based on feature clustering results. A physical phase refers to a continuous time period during which an object's motion exhibits the same physical laws, such as the falling phase under gravity, the collision phase upon contact with the ground, and the rebound phase after the collision.

[0105] Because reference video frames within the same cluster share similar latent space features—which characterize the spatiotemporal geometric and semantic features of video frames—they are in the same physical motion state and exhibit the same physical laws. Computer devices classify consecutive reference video frames belonging to the same cluster as a single physical stage, thus dividing the entire reference video into multiple physical stages.

[0106] Step 304: For each physical stage obtained from the division, the physical expert model corresponding to the physical law represented by the physical stage is determined as the physical expert model corresponding to each reference video frame in the physical stage, and a static dictionary is constructed.

[0107] In some embodiments, for each divided physical stage, the computer device determines the physical laws represented by that physical stage and determines the physical expert model corresponding to that physical law as the physical expert model corresponding to each reference video frame in that physical stage.

[0108] Since the physical stages are determined by clustering based on latent space features, reference video frames within the same physical stage are in the same physical motion state, and therefore correspond to the same physical expert model. The computer device stores the mapping relationship between each reference video frame and its corresponding physical expert model in a static dictionary, thus completing the construction of the static dictionary.

[0109] By performing cluster analysis on latent space feature sequences, reference video frames with similar latent space features are divided into the same physical stage. This allows reference video frames within the same physical stage to share the same physical expert model, eliminating the need for separate physical law identification and expert model matching for each frame. This significantly reduces the computational load of expert model matching while ensuring the accuracy of the mapping relationship, improving the efficiency of static dictionary construction, and also ensuring the consistency of physical stage division.

[0110] Steps 302 to 304 above describe a physical stage division method based on feature clustering. In practical applications, computer devices can also use other methods to divide physical stages. Steps 305 to 308 describe another physical stage division method based on the changes in candidate physical laws.

[0111] Step 305: Based on the latent space features of each reference video frame, determine the candidate physical laws represented by each reference video frame.

[0112] In some embodiments, for each reference video frame, the computer device determines the candidate physical laws represented by the frame based on the latent space characteristics of the frame.

[0113] Candidate physical laws refer to the physical laws that a frame may correspond to, initially determined based on the latent space characteristics of a single frame. It should be noted that, due to the limited information in a single frame, the physical laws determined based on the latent space characteristics of a single frame may have a certain degree of uncertainty, hence the term "candidate physical laws."

[0114] Optionally, the computer device can classify the latent space features of each reference video frame using a classifier to obtain candidate physical laws corresponding to each reference video frame. This classifier is a pre-trained machine learning model whose input is the latent space features and whose output is the category of the physical law corresponding to that frame.

[0115] Optionally, the computer device can also calculate the similarity between the latent space features of each reference video frame and the reference features corresponding to various preset physical laws, and determine the physical law category with the highest similarity as the candidate physical law of that frame.

[0116] Alternatively, computer devices can combine the above methods, for example, first classifying the data using a classifier and then verifying or correcting the classification results through similarity calculation, in order to improve the accuracy of determining candidate physical laws.

[0117] The embodiments of this application do not limit the specific method for determining candidate physical laws.

[0118] Step 306: Based on the temporal changes of the candidate physical laws represented by the continuous reference video frames, the reference video frames are divided into at least one physical stage.

[0119] In some embodiments, the computer device analyzes the temporal changes of candidate physical laws represented by consecutive reference video frames and divides the reference video frames into at least one physical stage based on these changes. When the candidate physical laws represented by consecutive frames remain stable, it indicates that these frames are in the same physical stage; when the candidate physical laws change, it indicates that the physical stage has switched, and the location of the change is the boundary between physical stages.

[0120] Optionally, the computer device may divide a series of consecutive frames with the same candidate physical laws into the same physical phase. For example, when the candidate physical laws for frames 1 to 20 are all falling, and the candidate physical laws for frames 21 to 35 are all collisions, the computer device may divide frames 1 to 20 into the falling phase and frames 21 to 35 into the collision phase.

[0121] Optionally, the computer device can also detect the frame position where the candidate physical law changes, and use that frame position as the physical stage boundary, thereby dividing the reference video frame into multiple physical stages. For example, when the computer device detects a change in the candidate physical law between frame 20 and frame 21, the area between frame 20 and frame 21 is determined as the stage boundary.

[0122] This application does not limit the specific way of dividing the physical stages in the embodiments, as long as the reference video can be divided into multiple physical stages according to the changes in the candidate physical laws.

[0123] Step 307: For each physical stage, determine the physical law represented by the physical stage based on the candidate physical laws represented by the reference video frames in the physical stage.

[0124] Since a physical phase is a continuous time period, and the multiple reference video frames contained therein are all in the same physical motion state, the computer device can select one of the candidate physical laws represented by each reference video frame in the physical phase as the physical law represented by the physical phase.

[0125] In some embodiments, for each divided physical stage, the computer device integrates the candidate physical laws represented by all reference video frames in that physical stage to determine the final physical law represented by that physical stage.

[0126] Optionally, the computer device may identify the candidate physical law that appears most frequently in the physical stage as the physical law characterized by that physical stage.

[0127] Optionally, the computer device can also perform weighted voting based on the position of each candidate physical law in the physical stage, and determine the candidate physical law with the highest weight as the physical law represented by that physical stage. For example, frames located in the middle of the physical stage are given higher weights, and frames located at the edges are given lower weights.

[0128] Optionally, the computer device can also identify the candidate physical law with the highest confidence level in a physical stage as the physical law represented by that physical stage. For example, when the similarity between the latent space features of a frame and the reference features of a certain physical law is higher than that of other candidate physical laws, the candidate physical law corresponding to that frame has a higher confidence level and can be given priority as the basis for determining that physical stage.

[0129] The embodiments of this application do not limit the method of determining physical laws.

[0130] Step 308: For each physical stage, determine the physical expert model corresponding to the physical law represented by the physical stage as the physical expert model corresponding to each reference video frame in the physical stage, and construct a static dictionary.

[0131] In some embodiments, for each physical stage, the computer device determines the physical expert model corresponding to the physical law represented by that physical stage as the physical expert model corresponding to each reference video frame in that physical stage, and stores the mapping relationship between each reference video frame and its corresponding physical expert model in a static dictionary, thus completing the construction of the static dictionary.

[0132] like Figure 4 The diagram illustrates a schematic embodiment of the physical stage division process provided in this application. The top shows the temporal arrangement of candidate physical laws corresponding to each of the consecutive reference video frames. Consecutive frames with the same candidate physical law constitute a continuous interval segment. The diagram shows the interval segment corresponding to law A, law B, and law C. The computer device divides the reference video frames into multiple physical stages based on the temporal changes in the sequence of candidate physical laws represented by the consecutive reference video frames. When a candidate physical law changes, the location of this change is the boundary between physical stages. The continuous interval segment corresponding to law A is divided into the first physical stage, the continuous interval segment corresponding to law B is divided into the second physical stage, and the continuous interval segment corresponding to law C is divided into the third physical stage. Reference video frames within each physical stage have the same candidate physical laws, representing the same physical motion state.

[0133] By determining candidate physical laws frame by frame and dividing the physical stages according to the temporal changes of the candidate physical laws represented by consecutive frames, the boundary positions where physical laws change can be accurately captured, making the division of physical stages more refined. This allows for the matching of more accurate physical expert models for each physical stage, thereby improving the construction accuracy of the static dictionary.

[0134] Steps 302 to 304 and steps 305 to 308 above describe two different methods for dividing physical stages. In practical applications, computer devices can choose either method to construct a static dictionary based on the specific circumstances of the reference video.

[0135] Alternatively, the computer device can also directly obtain the correspondence between reference video frames and physical expert models based on the results of manual annotation.

[0136] Specifically, the computer equipment acquires the manually labeled results, which are obtained by professionals who annotate each reference video frame in the reference video with physical laws. The annotators label each reference video frame with the corresponding physical law type (such as gravity fall, collision, rebound, etc.) based on the motion state and physical characteristics presented by the frame. The computer equipment then determines the corresponding physics expert model for each reference video frame based on the annotation results and constructs a static dictionary based on the correspondence between the reference video frames and the physics expert models.

[0137] To illustrate, suppose the reference video contains 100 frames. By observing the video content, the annotator determines that frames 1 to 60 represent the gravity-induced descent phase, frames 61 to 65 represent the collision phase, and frames 66 to 100 represent the rebound phase. Based on this annotation result, the computer device determines the physics expert model corresponding to frames 1 to 60 as the gravity expert model, the physics expert model corresponding to frames 61 to 65 as the collision expert model, and the physics expert model corresponding to frames 66 to 100 as the rebound expert model, and constructs a static dictionary based on the above mapping relationship.

[0138] The manual annotation method directly utilizes the annotators' judgments on the physical laws of reference video frames to determine the corresponding physics expert model, eliminating the need for automated processing such as feature extraction and cluster analysis. This approach is suitable for scenarios with complex reference video content or where automatic segmentation results are insufficient. The feature extraction method, on the other hand, can automatically complete physical stage segmentation and expert model matching without manual intervention, making it suitable for scenarios with relatively standardized and regulated reference video content. This application does not limit the specific method for constructing the static dictionary.

[0139] In the steps described above, the computer device has constructed a static dictionary containing the mapping relationships between reference video frames and physics expert models, and generated generative videos using a video generation model. Next, the computer device needs to determine the video frames to be evaluated from the generative videos, and identify the corresponding physics expert models for each of these video frames, so that the video generation model can be trained based on these physics expert models.

[0140] like Figure 5 The diagram illustrates a flowchart of a physical expert model determination process according to an illustrative embodiment of this application. This process can be executed by the aforementioned computer device and includes at least one of the following steps: Step 510: Set a mask matrix based on preset rules. The mask matrix is ​​used to indicate the video frames to be evaluated in the generative video frames.

[0141] The mask matrix is ​​a binary matrix corresponding to a sequence of generative video frames. Each element in the mask matrix corresponds to a frame in the generative video. Using the mask matrix, the computer can identify the video frames from the generative video that need to be evaluated and used in subsequent evaluation and training. Preset rules instruct the computer how to set the mask matrix.

[0142] like Figure 6 The diagram illustrates a schematic embodiment of a mask matrix provided in this application. Reference video 601 includes multiple reference video frames arranged along the time axis, including a first reference video frame 602, a second reference video frame 603, a third reference video frame 604, and a fourth reference video frame 605. The mask matrix 606 is a binary matrix corresponding to the sequence of reference video frames. Each element in the mask matrix corresponds to a reference video frame and indicates whether the frame is a video frame to be evaluated. Specifically, the mask value for the second reference video frame 603 is 1, and the mask value for the fourth reference video frame 605 is 1, indicating that these two reference video frames are video frames to be evaluated; the mask value for the first reference video frame 602 is 2, and the mask value for the third reference video frame 604 is 2, indicating that these two reference video frames are not video frames to be evaluated. Based on this mask matrix, a computer device can determine the frames from the reference video that need to participate in subsequent evaluation and training.

[0143] In some embodiments, the computer device sets the mask of the generative video tail frame and at least one generative video intermediate frame other than the generative video tail frame in the generative video frames in the mask matrix to a first value; wherein, the mask of the generative video frames in the mask matrix is ​​set to a second value by default, and the mask of the first value in the mask matrix is ​​used to indicate that the corresponding generative video frame is a video frame to be evaluated.

[0144] In this context, the last frame of a generative video refers to the last frame in the temporal sequence of the generative video, while the intermediate frames of a generative video refer to all other frames in the generative video except for the first and last frames.

[0145] Schematic, the rules for setting the mask matrix can be expressed as follows: (2)

[0146] in, In generative videos, the first... The mask value corresponding to the frame. This represents the total number of frames in the generated video. and This indicates the frame number randomly selected from the intermediate frames. Formula (2) indicates that the mask corresponding to the last frame in the generated video is forcibly set to the first value (i.e., the mask value is 1), the masks corresponding to the two frames randomly selected from the intermediate frames are also set to the first value, and the masks corresponding to the remaining frames are set to the second value (i.e., the mask value is 0).

[0147] Since the end frame of a generative video contains the final prediction of the target physical state by the video generation model at the end of the generation process, it has significant supervisory value for the final physical state of the video. Therefore, forcing it to be set as the frame to be evaluated ensures that the model's quality of the generated results is effectively supervised. The intermediate frames of a generative video refer to the frames other than the first and last frames. They reflect the video generation model's prediction of the physical evolution process during generation and are an important basis for evaluating whether the model accurately simulates physical laws.

[0148] The mask of the generative video frames in the mask matrix is ​​set to the second value by default, which means that all frames are not video frames to be evaluated by default. The video frames to be evaluated need to be explicitly specified by setting the mask to the first value, so as to avoid incorrect frames being included in the evaluation range due to improper default settings, thus ensuring the reliability of the training process.

[0149] To illustrate, suppose the generative video contains 100 frames. The computer device sets the mask of the 100th frame (the last frame) to a first value, and randomly selects one or more frames from the 2nd to the 99th frame (the middle frames) to set their masks to the first value. The masks of the remaining frames retain the second value. Thus, the frames with the first mask value are the video frames to be evaluated and participate in subsequent evaluation and training; the frames with the second mask value are the video frames not to be evaluated and do not participate in subsequent evaluation and training.

[0150] Optionally, when selecting video frames to be evaluated from intermediate frames, the computer device may use a random selection method, a uniform sampling method (such as selecting one frame every fixed number of frames), or a keyframe selection method (such as selecting frames with significant changes in motion). This application embodiment does not limit the specific method of selecting intermediate frames.

[0151] Optionally, the computer device may also use other methods to determine the video frames to be evaluated. For example, the computer device may not select the last frame at a fixed time, but may randomly select a preset number of frames from the generated video as the video frames to be evaluated; or, the computer device may select multiple frames from the generated video at equal intervals using a uniform sampling method; or, the computer device may select the video frames to be evaluated based on the magnitude of motion state changes in each frame of the generated video, for example, determining frames with larger motion state changes as the video frames to be evaluated, and determining frames with smaller motion state changes as non-evaluation video frames. The specific method for determining the video frames to be evaluated is not limited in the embodiments of this application.

[0152] By setting the mask matrix as described above, computer devices can identify a portion of representative frames from generative videos for subsequent evaluation and training with lower computational and memory overhead. Since only a portion of the frames participate in training, the memory usage during training is significantly reduced, thereby ensuring that the model can obtain effective supervision signals while making training in long video scenarios engineering feasible.

[0153] Step 520: Based on the correspondence between the reference video frames in the reference video and the generative video frames in the generative video, and the first frame index of the video frame to be evaluated indicated by the mask matrix, determine the second frame index of the reference video frame corresponding to the video frame to be evaluated.

[0154] In some embodiments, after determining the video frame to be evaluated, the computer device determines the second frame index of the reference video frame corresponding to the video frame to be evaluated based on the first frame index of the video frame to be evaluated and the correspondence between the reference video frame and the generated video frame.

[0155] Here, the first frame index refers to the position number of the video frame to be evaluated in the generative video, and the second frame index refers to the position number of the reference video frame corresponding to the video frame to be evaluated in the reference video. Since the generative video and the reference video have the same or similar physical evolution process, there is a frame-level temporal correspondence between the two. Therefore, frames in the generative video can be mapped to frames in the reference video through this correspondence.

[0156] The correspondence between reference video frames and generated video frames depends on whether the total number of frames in both is the same. The processing methods for the two cases are explained below.

[0157] In some embodiments, when the number of reference video frames is the same as the number of generated video frames, the computer device determines the index of the reference video frame that has the same index number as the index of the first frame index as the second frame index.

[0158] When the total number of frames in the reference video and the generative video is the same, they have a one-to-one frame-level relationship on the timeline. That is, the Nth frame in the generative video and the Nth frame in the reference video are at the same stage of physical evolution. Therefore, the reference video frame index with the same sequence number can be directly determined as the second frame index.

[0159] like Figure 7 As shown, this diagram illustrates a schematic embodiment of the second frame index determination process provided in this application. The upper part of the diagram shows a generative video frame sequence, including a first generative video frame 701, a second generative video frame 702, a third generative video frame 703, and a fourth generative video frame 704; the lower part shows a reference video frame sequence, including a first reference video frame 705, a second reference video frame 706, a third reference video frame 707, and a fourth reference video frame 708.

[0160] When the number of reference video frames is the same as the number of generated video frames, the computer device determines that the generated video frame with the same frame number corresponds to the reference video frame. In the diagram, the first frame index indicates the second generated video frame 702, whose index number is 2. The computer device determines the index of the reference video frame (i.e., the second reference video frame 706) with the same index number as the second frame index. If the first frame index indicates another generated video frame, the determination of the second frame index is the same, that is, the index number of the second frame index is the same as the index number of the first frame index.

[0161] In some embodiments, when the number of frames in the reference video frame differs from the number of frames in the generated video frame, the computer device determines the second frame index based on the ratio of the number of frames in the reference video frame to the number of frames in the generated video frame, and the first frame index.

[0162] When the total number of frames in the reference video and the generated video are different, there is no one-to-one frame-level relationship between the two, and mapping is required based on the frame ratio.

[0163] To illustrate, assume that the reference video contains 60 frames and the generative video contains 80 frames, with a frame ratio of 3:4. The reference video frame index corresponding to the 40th frame in the generative video (the first frame index is 40) is 40×60 / 80=30, that is, the second frame index is 30, indicating that the generative frame and the 30th frame in the reference video are at the same stage of physical evolution.

[0164] like Figure 8As shown, this diagram illustrates a second frame index determination process provided by another illustrative embodiment of this application. The upper part of the diagram shows a generative video frame sequence, including a first generative video frame 801, a second generative video frame 802, a third generative video frame 803, a fourth generative video frame 804, a fifth generative video frame 805, and a sixth generative video frame 806; the lower part shows a reference video frame sequence, including a first reference video frame 807, a second reference video frame 808, and a third reference video frame 809.

[0165] When the number of frames in the reference video frame differs from the number of frames in the generated video frame, the computer determines the second frame index based on the ratio of the number of frames in the reference video frame to the number of frames in the generated video frame, and the first frame index. In the diagram, the reference video frame has 3 frames, and the generated video frame has 6 frames, resulting in a frame ratio of 1:2. The first frame index in the diagram indicates the third generated video frame 803, with an index number of 3. The computer determines this index as the second reference video frame 808 based on the frame ratio 3 × (3 / 6) = 1.5, after rounding. Therefore, the second frame index is 2. If the first frame index indicates another generated video frame, the determination of the second frame index follows the same method, based on a mapping of the frame ratio.

[0166] Optionally, if the index obtained after scaling is not an integer, the computer device can perform rounding (such as rounding up, rounding up, or rounding down) and determine the rounded index as the index of the second frame.

[0167] By determining the frame correspondence based on whether the number of reference video frames and the number of generative video frames are the same, either by using frame sequence number mapping or frame number ratio mapping, the generative video frames can be accurately mapped to the corresponding reference video frames regardless of whether the total number of frames in the reference video and the generative video are the same. This provides a reliable index for subsequently searching for physics expert models from the static dictionary.

[0168] Step 530: Based on the second frame index, determine the physical expert model corresponding to the video frame to be evaluated from the static dictionary.

[0169] In some embodiments, the computer device searches for the corresponding physical expert model in a static dictionary based on the determined second frame index. Since the static dictionary stores the mapping relationship between reference video frames and physical expert models, the corresponding physical expert model can be determined by the second frame index, and this physical expert model is the physical expert model corresponding to the video frame to be evaluated.

[0170] Schematic, the mapping relationship stored in the static dictionary can be represented by the following formula (3):

[0171] in, It is a static dictionary. For the frame index of the reference video frame, This is a physics expert model. Equation (3) represents the frame index. The corresponding frames are mapped to the gravity expert model. Frame Index The corresponding frames are mapped to the collision expert model. And so on.

[0172] To illustrate, suppose the computer device determines that the second frame index corresponding to the video frame to be evaluated is 30. The computer device then uses frame index 30 as the key to search the static dictionary and obtain the physical expert model (e.g., gravity expert model) corresponding to that frame index. This model is the physical expert model corresponding to the video frame to be evaluated.

[0173] Since the lookup process does not involve online evaluation of the current generation quality of the video generation model, but directly matches based on a pre-built static dictionary, it can quickly and accurately determine the physical expert model corresponding to each video frame to be evaluated, providing a reliable source of supervision signals for subsequent training.

[0174] In summary, by setting a mask matrix based on preset rules, the mask matrix explicitly indicates the video frames to be evaluated in the generative video frames. This allows the computer device to directly determine which generative video frames need to participate in subsequent processing based on the mask matrix, eliminating the need for online judgment or evaluation of whether each frame needs processing in subsequent steps. This ensures the consistency and reliability of the determination of the video frames to be evaluated. At the same time, the setting of the mask matrix is ​​based on preset rules and does not rely on online analysis or evaluation of the content of the generative video frames. This avoids the deviation in the determination of the video frames to be evaluated caused by poor quality of the generated content in the early stage of training, and provides an accurate basis for the subsequent frame index determination and physical expert model search.

[0175] After determining the physical expert model corresponding to each video frame to be evaluated through the aforementioned steps, the computer equipment needs to train the video generation model based on these physical expert models so that the subsequently generated video conforms to the target physical laws.

[0176] In some embodiments, when training a video generation model, the computer device determines the stream matching loss for each video frame to be evaluated based on the physical expert model corresponding to the video frame to be evaluated.

[0177] Stream matching loss is a quantitative metric used to measure the difference between the velocity field predicted by the video generation model and the reference velocity field provided by the physics expert model. Within the stream matching framework, the training objective of the video generation model is to minimize the difference between its predicted velocity field and the target velocity field. The closer the predicted velocity field is to the reference velocity field, the smaller the stream matching loss, indicating that the video generated by the video generation model is physically closer to the reference video.

[0178] In determining the stream matching loss, the computer needs to acquire the intermediate output of the video generation model when generating the video frame to be evaluated, and based on this intermediate output, obtain the predicted velocity field and the reference velocity field through the video generation model and the physics expert model, respectively. The specific process of determining the stream matching loss is explained below.

[0179] In some embodiments, the output of the video generation model also includes the hidden states corresponding to each generative video frame, which are used to characterize the intermediate states of the generative video frames during the generation process.

[0180] In this context, the hidden state is the intermediate state representation in the latent space of the video generation model during the generation of the video frame to be evaluated. When generating each frame, the video generation model does not directly output the final video frame, but rather generates it step by step through multiple iterations. The intermediate results of each iteration are the hidden states, which contain the encoded information of the video frame content by the video generation model at the current generation stage, and form the basis for subsequent velocity field calculations.

[0181] In some embodiments, a computer device acquires the hidden state corresponding to a video frame to be evaluated.

[0182] Optionally, the video generation model outputs corresponding hidden states when generating each generative video frame; that is, the hidden states are intermediate outputs during the video generation process. When obtaining the hidden states corresponding to the video frame to be evaluated, the computer device can extract the hidden states corresponding to that video frame from the output of the video generation model.

[0183] In some embodiments, the computer device outputs the predicted velocity field corresponding to the video frame to be evaluated through a video generation model based on the hidden state of the video frame to be evaluated.

[0184] Among them, the predicted velocity field is the velocity field result predicted by the video generation model based on the hidden state, which reflects the video generation model's estimation of the motion state of the video frame in the current training stage.

[0185] In some embodiments, the computer device inputs the hidden state of the video frame to be evaluated into the velocity field prediction branch of the video generation model, and outputs the predicted velocity field corresponding to the video frame to be evaluated through the branch.

[0186] In some embodiments, the computer device outputs a reference velocity field corresponding to the video frame to be evaluated based on the hidden state of the video frame to be evaluated through the physical expert model corresponding to the video frame to be evaluated.

[0187] Among them, the reference velocity field is the velocity field reference result output by the physics expert model based on the same hidden state. It reflects the state that the velocity field should have in accordance with the target physical laws and is the fitting target for the predicted velocity field.

[0188] In some embodiments, the computer device inputs the hidden state of the video frame to be evaluated into the physical expert model corresponding to the video frame to be evaluated, and outputs the reference velocity field corresponding to the video frame to be evaluated through the physical expert model.

[0189] It should be noted that the predicted velocity field and the reference velocity field are obtained respectively from the video generation model and the physics expert model, based on the same hidden state. They have the same input but different outputs—the predicted velocity field is the result currently predicted by the video generation model, while the reference velocity field is the standard answer provided by the physics expert model. By comparing the differences between the two, we can measure the gap between the current generation result of the video generation model and the target physical law.

[0190] In some embodiments, the computer device determines the stream matching loss corresponding to the video frame to be evaluated based on the predicted velocity field and the reference velocity field.

[0191] Among them, the flow matching loss is used to measure the degree of deviation between the velocity field predicted by the video generation model and the reference velocity field provided by the physics expert model. The closer the predicted velocity field is to the reference velocity field, the smaller the flow matching loss, indicating that the video generated by the video generation model is closer to the reference video in terms of physical laws.

[0192] In some embodiments, the computer device calculates the difference between the predicted velocity field and the reference velocity field, and determines the difference as the stream matching loss for the video frame to be evaluated.

[0193] Schematic, the computer device can use the mean squared error (MSE) as the loss function to calculate the sum of squares of the differences between corresponding elements between the predicted velocity field and the reference velocity field, and use the calculation result as the flow matching loss.

[0194] In practical applications, when calculating the stream matching loss, computer equipment also needs to consider the indicative role of the mask matrix, that is, only the video frames to be evaluated are calculated for loss, and non-evaluated video frames are not included in the loss calculation. Based on the above principle, the total stream matching loss of the entire training process can be expressed as the following formula (4):

[0195] in, The total number of frames in the generated video. For the first The mask value corresponding to the frame ( This indicates that the frame is the video frame to be evaluated. This indicates that the frame is not a video frame to be evaluated. For the first The hidden state of a frame. For the purpose of displaying information, For video generation models based on hidden states and prompt information The output predicted velocity field, To use a static dictionary The physics expert model obtained from the query is based on the same hidden state. and prompt information The output reference velocity field, The square of the L2 norm is used to measure the degree of difference between the predicted velocity field and the reference velocity field. Let be the mathematical expectation, and represent the average over all samples. Formula (4) indicates that the computer device, based on the mask matrix, only includes the velocity field differences corresponding to the video frames to be evaluated in the total loss, while the velocity field differences corresponding to non-video frames to be evaluated are masked. Ignore it; it will not be included in the loss calculation.

[0196] Alternatively, the computer device may also employ other types of loss functions to measure the difference between the predicted velocity field and the reference velocity field, such as Mean Absolute Error (MAE) or smoothed L1 loss. This application does not limit the specific type of loss function, as long as it can reflect the difference between the predicted velocity field and the reference velocity field.

[0197] In this way, the computer device can obtain the predicted velocity field and the reference velocity field based on the hidden state through the video generation model and the physics expert model, respectively, and determine the flow matching loss based on the difference between the two, providing an accurate supervision signal for subsequent video generation model parameter updates.

[0198] In some embodiments, the computer device updates the model parameters of the video generation model based on the stream matching loss.

[0199] Specifically, the computer device summarizes the stream matching loss corresponding to each video frame to be evaluated (such as summation or weighted summation) to obtain the total loss. Then, backpropagation is performed based on the total loss to calculate the gradient of each parameter of the video generation model, and the parameters of the video generation model are updated using optimization algorithms (such as stochastic gradient descent, Adam, etc.).

[0200] The above parameter update process can be expressed as the following formula (5):

[0201] in, This represents the model parameters of the video generation model. This represents the learning rate (i.e., the step size for parameter updates). This represents the total loss with respect to model parameters. The gradient. Formula (5) represents the direction of gradient descent of the computer device, with a learning rate. Update model parameters for step size This gradually reduces the total loss.

[0202] In some embodiments, the computer device performs backpropagation based on stream matching loss and a mask matrix to update the model parameters of the video generation model, the mask matrix being used to indicate the video frames to be evaluated in the generative video frames.

[0203] The mask matrix indicates the video frames to be evaluated in the generative video frames. It corresponds to the generative video frame sequence. Frames with the first mask value are video frames to be evaluated, and frames with the second mask value are video frames not to be evaluated. During backpropagation, the computer device uses the gradients corresponding to the video frames to be evaluated only for parameter updates based on the mask matrix. The gradients corresponding to the video frames not to be evaluated are blocked and do not participate in parameter updates.

[0204] To illustrate, assume the generative video contains 100 frames. The mask matrix indicates that frames 30 and 100 are the video frames to be evaluated, and the remaining frames are not. The computer calculates the stream matching loss corresponding to frames 30 and 100 and sums them to obtain the total loss. During backpropagation, only the gradients corresponding to frames 30 and 100 participate in the parameter update, while the gradients corresponding to the remaining 98 frames are blocked and do not participate in the parameter update.

[0205] In practical applications, there are multiple ways to block the gradient stream of non-evaluated video frames.

[0206] Optionally, the computer device can block the gradient flow of non-evaluated video frames through computational graph pruning (Detach). Computational graph pruning refers to separating the tensors corresponding to non-evaluated video frames from the computational graph before backpropagation, so that they are no longer associated with the computational history during forward propagation, thus preventing the generation of gradients during backpropagation. This method not only blocks gradient propagation but also releases the computational graph cache occupied by the separated tensors, thereby effectively reducing video memory usage.

[0207] Alternatively, the computer device can also block the gradient flow of non-evaluated video frames by stopping the gradient flow. Stopping the gradient flow means setting the gradient corresponding to the non-evaluated video frame to zero during backpropagation, preventing it from continuing to propagate forward. This method also achieves the goal of preventing non-evaluated video frames from participating in parameter updates.

[0208] Optionally, the computer device can also combine the above two methods. For example, when video memory is scarce, computation graph pruning can be prioritized to free up video memory, while gradient flow blocking can be used when the computation graph structure needs to be preserved. This application does not limit the specific implementation method of blocking gradient flow in its embodiments.

[0209] By using a mask matrix to update parameters only for the gradients of the video frames to be evaluated during backpropagation, while the gradients of non-evaluated video frames are blocked from participating in the update, the memory usage during training is directly related to the number of frames used in training rather than the total number of video frames. This effectively reduces memory overhead and makes model training in long video scenarios feasible in engineering.

[0210] Optionally, the above training process can be iterated multiple times until the video generated by the video generation model meets the preset requirements in terms of physical laws. In each training round, the video frames to be evaluated can be redefined to cover different time positions in the generative video, so that the video generation model can obtain effective supervision signals at each physical stage and achieve comprehensive physical law alignment.

[0211] By determining the flow matching loss based on a physics expert model, the training of the video generation model can use the velocity field that conforms to the target physics as the fitting target, ensuring the accuracy of the training direction. At the same time, this loss is calculated only for the video frame to be evaluated, rather than all generated video frames, thereby reducing the amount of computation while ensuring effective supervision.

[0212] like Figure 9The diagram illustrates a schematic embodiment of the model training process provided in this application, with the static dictionary construction stage on the left. Reference video 901 (a standard video conforming to the target physical laws) undergoes physical state annotation or clustering 902 and feature extraction 903 to obtain a deterministic representation sequence 904. Feature extraction maps reference video frames to a latent space to extract key motion features, while clustering or physical state annotation groups frames with similar motion features into the same physical stage, thus forming a deterministic representation sequence. A computer device constructs a static dictionary 905 based on the deterministic representation sequence. The static dictionary 905 stores the mapping relationship between reference video frames and physical expert models. The diagram exemplarily illustrates the correspondence between each reference video frame index and its corresponding expert model (e.g., the mapping between frame index t and the collision model), as well as the specific type of the physical expert model (e.g., the collision model).

[0213] The right side shows the video generation and model training phases. The computer device inputs initial conditions 906 into the video generation model 907, and the video generation model generates video based on these initial conditions, resulting in generative video 908. Simultaneously, the computer device generates a mask matrix 909 according to preset rules. This mask matrix is ​​used to indicate the video frames to be evaluated within the generative video frames.

[0214] Subsequently, the computer device judges each generative video frame in the generative video 908 based on the mask matrix 909 to determine the video frame to be evaluated. For each video frame to be evaluated, the computer device outputs the predicted velocity field 910 corresponding to the frame through the video generation model 907, queries the corresponding expert model 911 from the static dictionary 905 according to the frame index t of the frame, outputs the reference velocity field 912 corresponding to the frame through the expert model, calculates the distillation loss 913 based on the predicted velocity field 910 and the reference velocity field 912, and uses the distillation loss to update the parameters of the video generation model 907.

[0215] In summary, the computer device provides physical prior guidance through a static dictionary, uses a mask matrix to filter video frames to be evaluated, and trains the video generation model with a reference velocity field provided by a physics expert model as a supervision signal.

[0216] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0217] like Figure 10 As shown, it illustrates a schematic diagram of a model training device provided in an illustrative embodiment of this application. The device includes a construction module 1010, a generation module 1020, a determination module 1030, and a training module 1040.

[0218] Module 1010 is configured to build a static dictionary based on the reference video. The static dictionary contains the mapping relationship between the reference video frames and the physics expert model. The reference video conforms to the target physical laws, and the physics expert model is used to provide prior information when verifying whether the video frames conform to the physical laws. The generation module 1020 is configured to generate a video based on the prompt information and through the video generation model to obtain a generative video. The prompt information is used to prompt the generation of a video that conforms to the physical laws of the target. The determination module 1030 is configured to determine the physical expert model corresponding to the video frame to be evaluated from the static dictionary based on the correspondence between the reference video frame in the reference video and the generative video frame in the generative video, wherein the video frame to be evaluated belongs to the generative video frame. Training module 1040 is configured to train the video generation model using the physics expert model corresponding to each video frame to be evaluated.

[0219] The construction module 1010 is also configured to extract features from each reference video frame in the reference video to obtain the latent space features of each reference video frame. The latent space features are used to characterize the spatiotemporal geometric and semantic features of the reference video frame. Based on latent space features, the physical expert model corresponding to each reference video frame is determined, and a static dictionary is constructed.

[0220] The construction module 1010 is also configured to perform feature clustering on the feature sequence composed of the latent space features of each reference video frame to obtain the feature clustering result; Based on the feature clustering results, the reference video frame is divided into at least one physical stage, wherein the physical laws represented by the reference video frames in the same physical stage are the same. For each physical stage obtained from the division, the physical expert model corresponding to the physical law represented by the physical stage is determined as the physical expert model corresponding to each reference video frame in the physical stage, and a static dictionary is constructed.

[0221] The construction module 1010 is also configured to determine the candidate physical laws represented by each reference video frame based on the latent space features of each reference video frame. Based on the temporal changes of candidate physical laws represented by continuous reference video frames, the reference video frames are divided into at least one physical stage. For each physical stage, the physical laws represented by the physical stage are determined based on the candidate physical laws represented by the reference video frames in the physical stage. For each physical stage, the physical expert model corresponding to the physical law represented by the physical stage is determined as the physical expert model corresponding to each reference video frame in the physical stage, and a static dictionary is constructed.

[0222] The determination module 1030 is also configured to set a mask matrix based on preset rules, the mask matrix being used to indicate the video frames to be evaluated in the generative video frames; Based on the correspondence between the reference video frames in the reference video and the generative video frames in the generative video, and the first frame index of the video frame to be evaluated indicated by the mask matrix, the second frame index of the reference video frame corresponding to the video frame to be evaluated is determined. Based on the second frame index, the physical expert model corresponding to the video frame to be evaluated is determined from the static dictionary.

[0223] The determining module 1030 is further configured to set the mask of the generative video tail frame in the generative video frames in the mask matrix, and at least one generative video intermediate frame other than the generative video tail frame, to a first value. In this context, the mask for the generative video frame in the mask matrix is ​​set to the second value by default, while the mask with the first value in the mask matrix is ​​used to indicate that the corresponding generative video frame is the video frame to be evaluated.

[0224] The determining module 1030 is also configured to determine the index of the reference video frame that has the same index number as the index of the first frame as the index of the second frame when the number of frames of the reference video frame is the same as the number of frames of the generated video frame. When the number of frames in the reference video frame differs from the number of frames in the generated video frame, the index of the second frame is determined based on the ratio of the number of frames in the reference video frame to the number of frames in the generated video frame, and the index of the first frame.

[0225] The training module 1040 is also configured to determine the stream matching loss corresponding to each video frame to be evaluated based on the physical expert model corresponding to the video frame to be evaluated. The model parameters of the video generation model are updated based on stream matching loss.

[0226] In some embodiments, the output of the video generation model also includes the hidden states corresponding to each generative video frame, and the training module 1040 is further configured to acquire the hidden states corresponding to the video frames to be evaluated. Based on the hidden state of the video frame to be evaluated, the predicted velocity field corresponding to the video frame to be evaluated is output through the video generation model. Based on the hidden state of the video frame to be evaluated, the reference velocity field corresponding to the video frame to be evaluated is output through the physical expert model corresponding to the video frame to be evaluated. Based on the predicted velocity field and the reference velocity field, the stream matching loss corresponding to the video frame to be evaluated is determined.

[0227] Training module 1040 is also configured to perform backpropagation based on stream matching loss and a mask matrix to update the model parameters of the video generation model, the mask matrix being used to indicate the video frames to be evaluated in the generative video frames.

[0228] The generation module 1020 is also configured to generate a generative video by using a video generation model based on the text information describing the physical laws of the target in the prompt message, or the first frame image of the video to be generated, or the text information describing the physical laws of the target and the first frame image of the video to be generated.

[0229] In some embodiments, the prompt information may further include total frame count information, which indicates the total number of frames in the video to be generated, and the indicated total frame count is the same as the total number of frames in the reference video.

[0230] In some embodiments, the reference video is generated based on prompt information, or simulated in a physics engine.

[0231] In summary, by constructing a static dictionary based on reference videos and mapping expert models to reference video frames, and then determining the expert model from the static dictionary based on the correspondence between reference video frames and generative video frames, the matching of expert models is based on the temporal correspondence between frames, rather than real-time analysis or evaluation of the content of generative video frames. Since this matching process is completely independent of the quality of the generative video, it reduces the possibility of incorrect or unsuitable expert model selection due to poor quality generated content in the early stages of training. This ensures the accuracy of the supervision signal during training and improves training stability and convergence efficiency.

[0232] Please refer to Figure 11 This diagram illustrates a structural block diagram of a computer device 1100 provided in an illustrative embodiment of this application. The computer device 1100 may be a terminal or a server.

[0233] Typically, computer device 1100 includes a processor 1101 and a memory 1102.

[0234] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1101 may also include an AI processor for handling computational operations related to machine learning.

[0235] In some embodiments, the processor 1101 may perform model training tasks using the model training method provided in the above embodiments.

[0236] The memory 1102 may include one or more computer-readable storage media, which may be tangible and non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices.

[0237] In some embodiments, the computer device 1100 may also optionally include a peripheral device interface 1103 and at least one peripheral device.

[0238] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on the computer device 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0239] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0240] On the other hand, embodiments of this application provide a computer device, which includes a processor and a memory. The memory stores at least one instruction, which is loaded and executed by the processor to implement the model training method provided in the embodiments of this application as described above.

[0241] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the model training method provided in the embodiments of this application as described above.

[0242] On the other hand, embodiments of this application provide a computer program product, which includes computer instructions stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, executes the computer instructions, and causes the processor of the computer device to load and execute the model training methods provided in the above-described method embodiments.

[0243] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A model training method, characterized in that, The method includes: A static dictionary is constructed based on a reference video. The static dictionary contains the mapping relationship between reference video frames in the reference video and a physics expert model. The reference video conforms to the target physical laws, and the physics expert model is used to provide prior information when verifying whether the video frames conform to the physical laws. Based on the prompt information, a video is generated using a video generation model to obtain a generative video. The prompt information is used to prompt the generation of a video that conforms to the target physical laws. Based on the correspondence between the reference video frames in the reference video and the generative video frames in the generative video, the physical expert model corresponding to the video frame to be evaluated is determined from the static dictionary, and the video frame to be evaluated belongs to the generative video frame. The video generation model is trained using the physical expert models corresponding to each of the video frames to be evaluated.

2. The method according to claim 1, characterized in that, The construction of a static dictionary based on reference videos includes: Feature extraction is performed on each reference video frame in the reference video to obtain the latent space features of each reference video frame. The latent space features are used to characterize the spatiotemporal geometric and semantic features of the reference video frame. Based on the latent space features, the physical expert model corresponding to each reference video frame is determined, and the static dictionary is constructed.

3. The method according to claim 2, characterized in that, The process of determining the physical expert model corresponding to each reference video frame based on the latent space features and constructing the static dictionary includes: Feature clustering is performed on the feature sequence composed of the latent space features of each reference video frame to obtain the feature clustering results; Based on the clustering results of the features, the reference video frame is divided into at least one physical stage, wherein the physical laws represented by the reference video frames in the same physical stage are the same. For each physical stage obtained from the division, the physical expert model corresponding to the physical law represented by the physical stage is determined as the physical expert model corresponding to each reference video frame in the physical stage, and the static dictionary is constructed.

4. The method according to claim 2, characterized in that, The process of determining the physical expert model corresponding to each reference video frame based on the latent space features and constructing the static dictionary includes: Based on the latent space characteristics of each reference video frame, the candidate physical laws represented by each reference video frame are determined. Based on the temporal variation of candidate physical laws represented by continuous reference video frames, the reference video frames are divided into at least one physical stage. For each physical stage, the physical law represented by the physical stage is determined based on the candidate physical laws represented by the reference video frames in the physical stage. For each physical stage, the physical expert model corresponding to the physical law represented by the physical stage is determined as the physical expert model corresponding to each reference video frame in the physical stage, and the static dictionary is constructed.

5. The method according to any one of claims 1 to 3, characterized in that, The step of determining the physical expert model corresponding to the video frame to be evaluated from the static dictionary based on the correspondence between the reference video frame in the reference video and the generative video frame in the generative video includes: A mask matrix is ​​set based on preset rules, and the mask matrix is ​​used to indicate the video frame to be evaluated in the generative video frame; Based on the correspondence between the reference video frames in the reference video and the generative video frames in the generative video, and the first frame index of the video frame to be evaluated indicated by the mask matrix, the second frame index of the reference video frame corresponding to the video frame to be evaluated is determined. Based on the second frame index, the physical expert model corresponding to the video frame to be evaluated is determined from the static dictionary.

6. The method according to claim 5, characterized in that, The process of setting the mask matrix based on preset rules includes: Set the mask of the generative video tail frame in the generative video frame in the mask matrix, and the mask of at least one generative video intermediate frame other than the generative video tail frame, to a first value. In this context, the mask of the generative video frame in the mask matrix is ​​set to the second value by default, while the mask of the first value in the mask matrix is ​​used to indicate that the corresponding generative video frame is the video frame to be evaluated.

7. The method according to claim 5, characterized in that, The step of determining the second frame index of the reference video frame corresponding to the video frame to be evaluated, based on the correspondence between the reference video frame in the reference video and the generative video frame in the generative video, and the first frame index of the video frame to be evaluated indicated by the mask matrix, includes: If the number of frames in the reference video frame is the same as the number of frames in the generated video frame, the index of the reference video frame that has the same index number as the index of the first frame is determined as the index of the second frame. If the number of frames in the reference video frame is different from the number of frames in the generated video frame, the second frame index is determined based on the ratio of the number of frames in the reference video frame to the number of frames in the generated video frame, and the first frame index.

8. The method according to any one of claims 1 to 3, characterized in that, The step of training the video generation model using the physical expert model corresponding to each of the video frames to be evaluated includes: For each of the video frames to be evaluated, the stream matching loss corresponding to the video frame to be evaluated is determined based on the physical expert model corresponding to the video frame to be evaluated. The model parameters of the video generation model are updated based on the stream matching loss.

9. The method according to claim 8, characterized in that, The output of the video generation model also includes the hidden states corresponding to each generative video frame, which are used to characterize the intermediate states of the generative video frame during the generation process. The step of determining the stream matching loss corresponding to the video frame to be evaluated based on the physical expert model corresponding to the video frame to be evaluated includes: Obtain the hidden state corresponding to the video frame to be evaluated; Based on the hidden state of the video frame to be evaluated, the predicted velocity field corresponding to the video frame to be evaluated is output by the video generation model. Based on the hidden state of the video frame to be evaluated, the reference velocity field corresponding to the video frame to be evaluated is output through the physical expert model corresponding to the video frame to be evaluated. Based on the predicted velocity field and the reference velocity field, the stream matching loss corresponding to the video frame to be evaluated is determined.

10. The method according to claim 8, characterized in that, The step of updating the model parameters of the video generation model based on the stream matching loss includes: Backpropagation is performed based on the stream matching loss and the mask matrix to update the model parameters of the video generation model, wherein the mask matrix is ​​used to indicate the video frame to be evaluated in the generative video frames.

11. The method according to any one of claims 1 to 3, characterized in that, The process of generating a generative video based on the prompt information and using the video generation model includes: Based on the text information describing the physical laws of the target in the prompt message, or the first frame image of the video to be generated, or the text information describing the physical laws of the target and the first frame image of the video to be generated, the video is generated through the video generation model to obtain the generative video.

12. The method according to claim 11, characterized in that, The prompt information also includes total frame count information, which indicates the total number of frames in the video to be generated, and the indicated total frame count is the same as the total number of frames in the reference video frame.

13. The method according to any one of claims 1 to 3, characterized in that, The reference video is generated based on the prompt information, or it is simulated in the physics engine.

14. A model training device, characterized in that, The device includes: The construction module is configured to build a static dictionary based on a reference video. The static dictionary contains the mapping relationship between reference video frames in the reference video and a physics expert model. The reference video conforms to the target physical laws, and the physics expert model is used to provide prior information when verifying whether the video frames conform to the physical laws. The generation module is configured to generate a generative video based on a prompting information and a video generation model. The prompting information is used to prompt the generation of a video that conforms to the target physical laws. The determination module is configured to determine the physical expert model corresponding to the video frame to be evaluated from the static dictionary based on the correspondence between the reference video frame in the reference video and the generative video frame in the generative video, wherein the video frame to be evaluated belongs to the generative video frame. The training module is configured to train the video generation model using the physical expert models corresponding to each of the video frames to be evaluated.

15. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the model training method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the model training method as described in any one of claims 1 to 13.

17. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, and a processor reads from and executes the computer program to implement the model training method as described in any one of claims 1 to 13.

Images