Training method and device of video generation model, video generation method and device

Abstract

The application relates to the technical field of artificial intelligence, and discloses a training method and a video generation method and device of a video generation model, the method comprising the following steps: acquiring a real frame sequence and a corresponding action sequence; determining a frame type corresponding to a current time point according to action information of the current time point in the action sequence; determining an input frame of the current time point, a contribution ratio of a current real frame and a current predicted frame to the current real frame and the current predicted frame, and the frame type of the current time point according to the frame type corresponding to the current time point based on the current real frame and / or the current predicted frame of the current time point; adding the input frame of the current time point to an input frame sequence; and training a video generation model according to predicted frames of multiple time points to obtain a trained video generation model. Each frame in a training sequence is classified into multiple frame types, and a special input strategy is designed for each type, so that exposure bias can be effectively eliminated.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to training methods for video generation models, video generation methods, and apparatus. Background Technology

[0002] Game world models are a type of artificial intelligence (AI) system that has emerged in recent years. Their goal is to enable neural networks to play the role of a game engine: given the player's action input, generate the corresponding next frame of the game screen to form a complete interactive virtual world.

[0003] However, the errors generated by the game world model during reasoning accumulate over time, eventually leading to increasingly distorted scenes and increasingly chaotic physical logic after prolonged gameplay. Summary of the Invention

[0004] In view of this, this application provides a training method for a video generation model, a video generation method, and an apparatus to solve the problem of large accumulated errors in video generation models such as game world models.

[0005] Firstly, this application provides a method for training a video generation model, the method comprising: Obtain a real frame sequence and a corresponding action sequence; the real frame sequence includes real frames at multiple time points, and the action sequence includes action information corresponding to the real frames; For the current time point, the frame type corresponding to the current time point is determined based on the action information of the current time point in the action sequence; Based on the frame type corresponding to the current time point, and based on the current real frame and / or the current predicted frame at the current time point, the input frame at the current time point is determined; wherein, the current predicted frame is predicted by the video generation model to be trained based on the input frame sequence before the current time point, and the input frame sequence includes input frames at multiple time points; when determining the input frame at the current time point, the contribution ratio of the current real frame to the current predicted frame is associated with the frame type at the current time point; The input frame at the current time point is added to the input frame sequence to predict the predicted frames at other time points after the current time point based on the input frame sequence; The video generation model is trained by predicting frames at multiple time points to obtain the trained video generation model.

[0006] Secondly, this application provides a video generation method, including: Obtain a historical frame sequence; the historical frame sequence includes historical frames predicted based on a trained video generation model, the trained video generation model being trained based on the training method of the video generation model of the first aspect above or any corresponding embodiment; The latest frame is determined by performing autoregressive inference on the historical frame sequence based on the trained video generation model. Output the latest frame and add it as a new historical frame to the historical frame sequence.

[0007] Thirdly, this application provides a training apparatus for a video generation model, the apparatus comprising: The acquisition module is used to acquire a real frame sequence and a corresponding action sequence; the real frame sequence includes real frames at multiple time points, and the action sequence includes action information corresponding to the real frames; The classification module is used to determine the frame type corresponding to the current time point based on the action information of the current time point in the action sequence. The processing module is configured to determine the input frame at the current time point based on the frame type corresponding to the current time point, and on the current real frame and / or the current predicted frame at the current time point; wherein the current predicted frame is predicted by the video generation model to be trained based on the input frame sequence before the current time point, and the input frame sequence includes input frames at multiple time points; when determining the input frame at the current time point, the contribution ratio of the current real frame to the current predicted frame is associated with the frame type at the current time point; An adding module is used to add the input frame at the current time point to the input frame sequence, so as to predict the prediction frame at other time points after the current time point based on the input frame sequence; The training module is used to train the video generation model based on predicted frames at multiple time points, resulting in a trained video generation model.

[0008] Fourthly, this application provides a video generation apparatus, the apparatus comprising: A sequence acquisition module is used to acquire a historical frame sequence; the historical frame sequence includes historical frames predicted based on a trained video generation model, wherein the trained video generation model is trained based on the training method of the video generation model described in any one of the first aspects above; The inference module is used to perform autoregressive inference on the historical frame sequence based on the trained video generation model to determine the latest frame; The output module is used to output the latest frame and add the latest frame as a new historical frame to the historical frame sequence.

[0009] Fifthly, this application provides an electronic device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the training method of the video generation model of the first aspect, the second aspect, or any corresponding embodiment described above.

[0010] In a sixth aspect, this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute a training method for a video generation model according to the first aspect, the second aspect, or any corresponding embodiment thereof.

[0011] In a seventh aspect, this application provides a computer program product, including computer instructions for causing a computer to execute a training method for a video generation model according to the first aspect, the second aspect, or any corresponding embodiment thereof.

[0012] The video generation model training method provided in this application leverages the presence of motion signals in game videos. During model training, it acquires motion sequences containing motion information at various time points, determines different frame types based on the motion information at each time point, and then determines the appropriate contribution ratio between the current real frame and the current predicted frame based on different frame types. Finally, based on the contribution ratio of the current real frame and the current predicted frame, it comprehensively determines the input frame as the input context. This approach ensures model accuracy while systematically exposing errors, guaranteeing the stability of training results. This application utilizes the structured characteristics of game motion signals to classify each frame in the training sequence into multiple frame types and designs a dedicated input strategy for each type, achieving a rational and evidence-based exposure and elimination of biases. Furthermore, this method only changes the way input frames are constructed; it does not require modification to the model architecture or loss function form. It can be directly embedded into the training process of any autoregressive game world model, exhibiting strong compatibility and wide applicability. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application; Figure 2 This is a schematic diagram of the first process of training a video generation model according to an embodiment of this application; Figure 3This is a schematic diagram of a second process for training a video generation model according to an embodiment of this application; Figure 4 This is a schematic diagram illustrating the determination of frame type based on action information according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating the distribution of three types of frames in an input frame sequence according to an embodiment of this application; Figure 6 This is a schematic diagram of a motion-aware training model according to an embodiment of this application; Figure 7 This is a schematic flowchart of a video generation method according to an embodiment of this application; Figure 8 This is a structural block diagram of a training device for a video generation model according to an embodiment of this application; Figure 9 This is a structural block diagram of a video generation apparatus according to an embodiment of this application; Figure 10 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] It is understood that before using the technical solutions disclosed in the various embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0017] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0018] Game world models are a type of AI system that has emerged in recent years. Their goal is to enable neural networks to act as game engines: given player input, they generate the corresponding next frame of the game, forming a complete, interactive virtual world. These models are valuable in scenarios such as rapid prototyping of game content and generating training environments for AI agents.

[0019] Autoregressive generation is the core inference paradigm of game world models: the model uses all historical frames as context and predicts future scenes frame by frame. This paradigm has a natural advantage in inference (it can generate in real time and stream), but it has an essential difficulty in training: because "real historical frames" are used during training, while "self-predicted frames" are used during inference (including the prediction results of the previous frame generated by the model based on the current context at the previous time step), the difference in distribution between the two is the root cause of the exposure bias.

[0020] The game world model primarily employs a teacher-mandated strategy during training, where the teacher is always present and each step uses real historical frames as input, rather than the model's own predicted frames. However, during inference, the model is required to act independently. This results in a persistent distribution gap between training and inference. In long-sequence play scenarios (e.g., players continuously interact for more than 30 seconds), accumulated errors lead to a significant decline in generation quality.

[0021] Analogous to a student learning to drive, the learning process is similar to model training: during each practice session, the instructor sits in the passenger seat, constantly correcting the student's steering wheel. If the student deviates even slightly, the instructor immediately steers the wheel back to the correct position. This teacher-mandated training method may go smoothly during the initial training phase, but the student never has the opportunity to experience what happens to the road conditions after they've veered off course.

[0022] In the process of trainees driving independently, there is no instructor present. It is similar to a model reasoning process: if the trainee's improper control causes the vehicle to veer slightly off course (the reasoning is flawed), the trainee is unaware of this error and will continue driving along the wrong path. Over time, this will accumulate and lead to increasingly more deviations, eventually resulting in a crash into the guardrail. There is a gap between the experience gained during training and the actual driving situation; this is known as exposure bias.

[0023] During the training of the game world model, real frames can be randomly replaced with predicted frames to eliminate exposure bias. However, these methods treat all frames the same and cannot perform adaptive training for different frames.

[0024] As one optional application scenario in the embodiments of this application, such as Figure 1 As shown, application 101 is installed in terminal device 110, and user 130 can interact with application 101 through terminal device 110 and / or access device of terminal device 110.

[0025] For example, application 101 can be any application, such as a game application. For instance, application 101 could be a question-and-answer interactive application, such as a text-to-text application, an image-to-text application, etc. Figure 1In the application scenario shown, if application 101 is active, the terminal device 110 can display the interface 102 of application 101. The interface 102 may include various pages that application 101 can provide, such as interactive pages, settings pages, query pages, etc.

[0026] In some embodiments, terminal device 110 is communicatively connected to server 120 to provide services to application 101. Terminal device 110 may be a mobile terminal, fixed terminal, or portable terminal, including but not limited to mobile phones, desktop computers, laptop computers, multimedia tablets, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 may also support any type of interface, and server 120 may be various types of computing systems or servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, and computing devices in cloud environments.

[0027] It should be noted that, Figure 1 This is merely an example of an application scenario and does not limit the scope of protection of this application.

[0028] The embodiments of this application will be described below with reference to the accompanying drawings. It should be understood that the pages shown in the drawings are merely examples, and various page designs are possible in practice. The various graphic elements on the page may have different arrangements and different visual representations, one or more elements may be omitted or replaced, and one or more other elements may also be present; no limitations are imposed on the embodiments of this application. Furthermore, the embodiments are primarily described below with reference to terminal device 110. It should be understood that the actions described relative to terminal device 110 can be performed by application 101 on terminal device 110, or can be performed by application 101 in conjunction with its server (e.g., server 120).

[0029] This application provides a training method for a video generation model. Utilizing the structured characteristics of game action signals, each frame in the training sequence is classified into three semantic types, and a dedicated input strategy is designed for each type to achieve well-reasoned elimination of exposure bias. Furthermore, this method only changes the way the input frames are constructed; it does not require modification to the model architecture or loss function. It can be directly embedded into the training process of any autoregressive game world model, exhibiting strong compatibility and wide applicability.

[0030] According to an embodiment of this application, a training method for a video generation model is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] This embodiment provides a training method for a video generation model, which can be used in the aforementioned server and can also be applied to terminal devices, such as mobile terminals. Figure 2 This is a flowchart of a training method for a video generation model according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps.

[0032] Step S201: Obtain the real frame sequence and the corresponding action sequence; the real frame sequence includes real frames at multiple time points, and the action sequence includes action information corresponding to the real frames.

[0033] When training a video generation model, a certain amount of training data is required, specifically a sequence of multiple frames, i.e., a sequence of ground truth frames. Each frame in this sequence corresponds to a specific time point and represents a real-world situation. Ground truth frames refer to the real targets (GT) used in model training; they are actual, accurate frames that can be used as positive samples for training. In addition to this sequence of ground truth frames, the training of the video generation model also considers the motion information corresponding to each frame to distinguish different frame types; that is, it obtains motion sequences containing motion information at multiple time points.

[0034] The video generation model can be a game world model, and correspondingly, the real frame sequence is a game frame sequence, meaning the real frames are actual game video frames. For example, game recording data can be extracted to obtain the corresponding real frame sequence, where the real frames can specifically be video frames (such as screenshots) from the game recording data. Furthermore, if training data is obtained from a game engine, the game recording data also includes engine metadata, which specifically includes the trigger time of each action (such as jumping, attacking, or releasing skills), and may also include the duration. Based on the engine metadata, the action information at each time point can be determined, forming the corresponding action sequence.

[0035] For example, the real frame sequence is ,in, Let represent the real frame corresponding to the i-th time point, and T be the total number of real frames. Correspondingly, the action sequence is: , This represents the action information corresponding to the i-th time point. For example, this action information could specifically indicate whether a new action is triggered. For ease of distinction, it will be referred to as... Let represent the real frame corresponding to the i-th time point, and correspondingly, the sequence of real frames is . .

[0036] In this embodiment, because video frames in a game scene differ from other videos, each frame change in the game is driven by explicit player action signals (structured data such as key codes, amplitude, and duration). This characteristic provides the foundation for a more intelligent training strategy—by analyzing action signals, the degree of change in the content of a frame can be predicted before the frame is generated. Using action signals, it is possible to intelligently determine when to rely on the player's own actions and when to use the teacher model, thereby customizing the handling of exposure bias and maintaining training stability.

[0037] Step S202: For the current time point, determine the frame type corresponding to the current time point based on the action information of the current time point in the action sequence.

[0038] In this embodiment, predictions need to be made for each relevant time point in order to train the video generation model. For ease of description, the currently processed time point is referred to as the current time point; it can be understood that any time point can be used as the current time point. For example, the real frame sequence is... It involves T time points, so all T time points can be used as the current time point.

[0039] For the current time point t, the action information of the current time point can be obtained from the action sequence. By leveraging the fact that action information may differ at different points in time, frames can be classified based on action information to determine the corresponding frame type. In this embodiment, frame types can be divided into three categories: Type 1, Type 2, and Type 3. Correspondingly, frames can also be divided into three different types, which will be explained in detail later.

[0040] Step S203: Based on the frame type corresponding to the current time point, determine the input frame for the current time point based on the current real frame and / or the current predicted frame. The current predicted frame is predicted by the video generation model to be trained based on the input frame sequence prior to the current time point, and the input frame sequence includes input frames from multiple time points. When determining the input frame for the current time point, the contribution ratio between the current real frame and the current predicted frame is related to the frame type of the current time point.

[0041] In this embodiment, it is necessary to determine the input frames used for video frame prediction at each time point. These input frames can form a corresponding input frame sequence, which serves as the input context for prediction. Similarly, for the current time point t, it is also necessary to determine the input frame for the current time point t. .

[0042] The input frame at the current time point t is determined. Yes, it doesn't use a uniform approach or randomly select the current real frame. Current prediction frame Instead, it adopts appropriate processing strategies for different frame types to determine the input frame suitable for that frame type.

[0043] Specifically, since the frame type corresponding to the current time point is based on the action information of the current time point. Indeed, this frame type can represent action information to some extent, such as whether a new action has been triggered. Therefore, for different frame types, the current real frame can be determined. With the current prediction frame The contribution ratio, and then based on the current real frame Current prediction frame The input frame at the current time point t is determined by comprehensive analysis. Among them, the contribution ratio of a certain frame can be 100%, meaning that the current real frame can be directly included. or the current prediction frame The input frame at the current time t .

[0044] During the training of the video generation model, for any current time point t, prediction can be made based on the video generation model to be trained. Specifically, prediction is made based on an input frame sequence containing input frames from multiple time points to obtain the predicted frame corresponding to the current time point t, i.e., the current predicted frame. If the input frame is determined at the current time point t and the current prediction frame is not needed (e.g., for subsequent trigger frames), prediction can be omitted, depending on the actual needs.

[0045] Specifically, for the current time point t, it is necessary to determine its corresponding current input frame. Only then will the current input frame be... Added to the input frame sequence. Therefore, prediction is performed based on the input frame sequence to obtain the current prediction frame. At this time, the input frame sequence only includes input frames determined before the current time point t. For example, the input frame sequence can be: The video generation model to be trained is based on the sequence of input frames prior to the current time point t. By making a prediction, the current prediction frame can be obtained. .

[0046] Step S204: Add the input frame at the current time point to the input frame sequence to predict the predicted frames at other time points after the current time point based on the input frame sequence.

[0047] For the current time point t, based on step S203, the input frame for the current time point t can be determined. The input frame It is added to the input frame sequence to update the input frame sequence. The updated input frame sequence can be: When it is necessary to determine the prediction frame at a certain time point after the current time point t (e.g., time point t+1), prediction can be performed based on the input frame sequence.

[0048] For example, at the next time point t+1, if it is necessary to predict the prediction frame for that time point... Then the video generation model to be trained can generate the input frame sequence The context sequence is used for prediction, thereby predicting the predicted frame. .

[0049] Specifically, the video generation model can be an autoregressive generation model, which uses an autoregressive prediction method and can use multiple historical frames during prediction, i.e., it can use the input frame sequence. All input frames can be used, or a sliding window can be set to use only the input frames within the sliding window for prediction, thereby reducing the computational load.

[0050] Step S205: Train the video generation model based on the predicted frames at multiple time points to obtain the trained video generation model.

[0051] In this embodiment, the input frame sequence is updated at each time point, so that subsequent predictions can be made based on the updated input frame sequence to obtain the predicted frame output by the video generation model. By comparing the predicted frame with the real frame at the corresponding time point, the loss between the two (e.g., reconstruction loss) can be calculated. Then, the video generation model is trained through forward propagation and other methods to adjust the weight parameters in the video generation model. Finally, the trained video generation model can be obtained, which can then be used for inference to predict game scenes in real time.

[0052] The video generation model training method provided in this application utilizes the characteristic of motion signals in game videos. During model training, it acquires motion sequences containing motion information at various time points, determines different frame types based on the motion information at each time point, and then determines the appropriate contribution ratio between the current real frame and the current predicted frame based on different frame types. Finally, based on the contribution ratio of the current real frame and the current predicted frame, it comprehensively determines the input frame as the input context. This ensures model accuracy while systematically exposing errors and guaranteeing the stability of training results. This application leverages the structured characteristics of game motion signals to classify each frame in the training sequence into multiple frame types and designs a dedicated input strategy for each type, achieving a rational and evidence-based exposure and elimination of biases. Furthermore, this method only changes the way input frames are constructed; it does not require modification to the model architecture or loss function form. It can be directly embedded into the training process of any autoregressive game world model, exhibiting strong compatibility and wide applicability.

[0053] This embodiment provides a training method for a video generation model, which can be used in the aforementioned server and can also be applied to terminal devices, such as mobile terminals. Figure 3 This is a flowchart of a training method for a video generation model according to an embodiment of this application, such as... Figure 3 As shown, the process includes the following steps.

[0054] Step S301: Obtain the real frame sequence and the corresponding action sequence; the real frame sequence includes real frames at multiple time points, and the action sequence includes action information corresponding to the real frames.

[0055] Please see details Figure 2 Step S201 of the illustrated embodiment will not be described again here.

[0056] Step S302: For the current time point, determine the frame type corresponding to the current time point based on the action information of the current time point in the action sequence.

[0057] Please see details Figure 2 Step S202 of the illustrated embodiment will not be described again here.

[0058] In some optional implementations, step S302, "determining the frame type corresponding to the current time point based on the action information at the current time point in the action sequence," may specifically include steps a1 to a3.

[0059] Step a1: Based on the action information at the current time point, determine whether the frame type corresponding to the current time point is the first type.

[0060] In this embodiment, for the current time point t, the action information at time point t is first used to determine whether the corresponding frame type is the first type, that is, whether it is a trigger frame.

[0061] Optionally, the action information may include an action flag indicating whether a new action is triggered. Action marker This is used to determine whether the frame type corresponding to the current time point is of the first type. Specifically, step a1 includes: determining whether the frame type corresponding to the current time point is of the first type based on the action flag bit corresponding to the current time point.

[0062] For example, ,like This indicates that no new discrete action is triggered at time point t. If this means a new action was triggered at time point t, then the frame corresponding to the current time point t is marked as the trigger frame. For example... , This indicates the frame type at the current time point t. Taking the jump action as an example, when the player clicks the jump button, a new jump action is triggered; or, if the player's action only responds when they lift their hand, then after the player clicks the jump button, the new jump action is triggered when their finger leaves the jump button, and the frame when the finger leaves the jump button is the trigger frame.

[0063] It is understandable that if the action flag is... If it only indicates whether an action exists at present, it can be combined with the action flag bit of the previous time point t-1. Make a comprehensive judgment. If ,as well as If , it means that a new action was triggered at the current time point t.

[0064] Step a2: If the frame type corresponding to the current time point is the first type, reset the target time window; the target time window is used to represent the duration corresponding to the latest triggered action.

[0065] If a new discrete action key is triggered at the current time point t (such as firing, jumping, skill release, etc.), the frame corresponding to the current time point t can be determined as the trigger frame; in addition to determining that the frame type at this time is the first type, the target time window after the frame that triggers the new action will also be reset. Initially, the target time window is empty.

[0066] The target time window represents the duration of the most recently triggered action. For example, the end time of the most recently triggered action can be used to represent the target time window. Alternatively, the current time t can be used as the most recently triggered time, and the duration of the most recently triggered action can be determined. The target time window can then be represented by a combination of the most recently triggered time and the duration.

[0067] For example, if a new action is triggered at the current time t, then the current time t can be directly set as the latest trigger time. ,Right now Furthermore, the corresponding duration can be set. This duration is understandable. Specifically, it can be a time value, such as 100ms; or it can be represented by frames, such as duration. The duration is 10 frames, meaning it can be the number of frames the action continues in. These two can be converted based on FPS (Frames Per Second) to accommodate game data with different frame rates. Correspondingly, the current time point t can be a timestamp or an index in the frame sequence, indicating which video frame it is.

[0068] For example, a fixed duration can be set for different action types, and the fixed duration can be converted according to the actual frame rate of the game to determine the corresponding number of frames. This number of frames can then be used to represent the target time window. Furthermore, if the action information also includes animation duration or physical event end markers, the duration can also be determined based on those markers.

[0069] Optionally, the size of the target time window can be a fixed value, such as the duration mentioned above. It can be a fixed value; or, the size of the target time window can be adaptively determined.

[0070] Specifically, action information may also include action type. This indicates which type of action the action is, such as jumping, firing, skill activation, or scene switching. Appropriate window sizes can be set for different action types, allowing the target time window to be reset based on the action type at the current time.

[0071] For example, the window size corresponding to some action types can be shown in Table 1 below. The values ​​in Table 1 are just examples and can be adjusted based on the actual situation.

[0072] Table 1

[0073] Optionally, step a2, "Reset the target time window", may also include steps a21 to a22.

[0074] Step a21: Based on the action information of the latest trigger time and the latent space features of historical time points before the latest trigger time, predict the duration of the new action to be triggered; the latest trigger time is the trigger time of the latest action.

[0075] Step a22: Reset the target time window based on the predicted duration.

[0076] In this embodiment, if the current time point t corresponds to a trigger frame, then the latest action is triggered at this time, i.e., the latest trigger time. Let's consider the current time point t. At this point, the latest trigger time can be determined. Action information That is, the action information at the current time point t. Furthermore, the latent space characteristics of previous historical time points (e.g., time point t-1) can be determined, and the duration of the new action triggered at the present time can be determined by combining the two.

[0077] Specifically, latent space features can be encoded features corresponding to input frames at historical time points. These features can be obtained by encoding through an encoder of an image generation model (such as a game world model) or by low-dimensional projection of the latent state of the input frame. They are used to represent the visual / state context of historical time points.

[0078] For example, a lightweight consequence duration predictor could be introduced. For each triggered action (i.e., trigger frame), the duration of its consequences is dynamically predicted. ,and . This predictor represents the latent spatial features of the previous historical time point. Specifically, it could be a lightweight multilayer perceptron (MLP) with 2 to 3 layers. This can be understood as being based on the input frame from the previous time point t-1. Determine the characteristics of the latent space .

[0079] In this embodiment, the predicted duration can be directly used. It can be used as the length of the target time window. Alternatively, it can be used in combination with the preset duration of different action types.

[0080] For example, if the preset duration corresponding to action type c is (See Table 1 above for details), you can As a priori lower bound, the final duration is obtained based on this. : Ensure that it is not lower than the prior lower bound.

[0081] Among them, in training the predictor When training data comes from a game engine, the engine typically has precise annotations of action duration (such as animation clip duration, skill duration frames, and physical event end markers). Therefore, these annotations can be directly used as predictors. The supervised target does not require visual computation. However, when only offline recorded data is available without engine metadata (i.e., no engine annotation), the duration of inter-frame pixel / feature changes exceeding a threshold can be used as a proxy label for the consequence duration to train the predictor. This solution is suitable for scenarios involving the migration of video data from third parties.

[0082] Step a3: If the frame type corresponding to the current time point is not the first type, and the current time point is within the target time window, then the frame type corresponding to the current time point is determined to be the second type.

[0083] If the frame corresponding to the current time point t is not the trigger frame, for example Then, it is further determined whether it is a consequence frame. Specifically, it can be determined whether the current time point t is within the target time window. If it is within the target time window, it means that the current time point t is still within the influence range of the newly triggered action, so its corresponding frame type is the second type, that is, it is a consequence frame. Accordingly, the frame type can be represented as: .

[0084] For example, if the target time window has a duration. Then it can be determined that the current time point t is equal to the latest trigger time. The difference between them, if it is less than or equal to the duration If the current time point t is within the target time window, then it can be considered that the current time point is within the target time window. Conversely, if the current time point is not within the target time window, then the frame corresponding to the current time point t can be determined as an inertial frame, such as when the player has not triggered an action, or has maintained the same input, such as moving at a constant speed or continuously pressing keys. Accordingly, the frame type can be represented as: .

[0085] In this embodiment, for the current time point t, it is first determined whether it belongs to the trigger frame (high priority), then whether it belongs to the consequence frame (medium priority), and finally whether it belongs to the inertial frame (low priority). Figure 4 This diagram illustrates a method for determining frame type based on motion information. For example... Figure 4 As shown, for the current time point t, its corresponding action information can be obtained. It includes action flags. By judgment Whether the value is 1 determines whether a new action has been triggered.

[0086] like This indicates that a new action was triggered at time point t, which is the trigger frame, and the target time window is reset; for example... And set the corresponding duration. .

[0087] like Then further judgment Whether it is true or not, if If the current time point t is within the target time window, it can be determined that it is a consequence frame; otherwise, it is an inertial frame.

[0088] It's understandable that if a new action is triggered within the target time window, since determining whether it's a trigger frame has the highest priority, it can be accurately identified as such. The target time window is then reset, starting the count from the latest trigger moment, rendering the old time window invalid. This covers overlapping trigger scenarios such as combos and firing while moving. Furthermore, all time points are categorized into one type, ensuring comprehensive coverage and eliminating any uncovered frames, thus guaranteeing accuracy.

[0089] Step S303: If the frame type corresponding to the current time point is the first type, determine the input frame of the current time point based on the current real frame of the current time point; the first type frame is the frame that triggers the new action.

[0090] For the current time point t, if its corresponding action information This indicates that a new action has been triggered, such as a player jumping. Therefore, the frame type corresponding to the current time point t can be determined to be type 1. In other words, the frame corresponding to the current time point t (e.g., the current actual frame at the current time point) is of type 1. The first type of frame is the frame that triggers a new action; for ease of description, the first type of frame is called the "trigger frame".

[0091] If the frame that triggers the new action at the current time point t (i.e., the trigger frame) is the starting point of the new action and the beginning of the game's physics causal chain, then using a predicted frame with errors as the trigger context will cause severe distortion in the physical response of subsequent action frames (the consequence frames mentioned below). Therefore, to ensure the prediction effect of other subsequent frames, the prediction is mainly based on the current real frame at the current time point t. To make a prediction, the frame to be predicted is called the input frame. Therefore, it can be based on the current real frame at the current time point t. Determine the input frame at the current time point t .

[0092] For example, the current real frame at time t can be directly obtained. The input frame at the current time t ,Right now .

[0093] The trigger frame is the anchor point in the causal chain of actions and cannot be replaced by a predicted value; otherwise, the error in each subsequent step would accumulate from here. For example, the current true frame at time t could be used... The input frame at the current time t ,Right now This ensures that the visual state at the moment of action occurs accurately corresponds to the real physical state, providing an error-free starting point for the evolution of subsequent consequence frames. In the autoregressive context sequence (i.e., the input frame sequence), the trigger frame position is written into the latent variables of the real frame, and the conditional inferences of subsequent steps are all based on this accurate starting point.

[0094] Step S304: If the frame type corresponding to the current time point is the second type, determine the input frame of the current time point based on the current real frame and the current predicted frame of the current time point; the second type of frame is the frame within the target time window after the frame that triggers the new action.

[0095] For the current time point t, if it is not the frame that triggered the new action, it can be determined whether it is within the target time window following the frame that triggered the new action. Specifically, for the triggered new action, since it lasts for a certain duration, the time period corresponding to its duration is taken as the target time window for triggering the new action. This target time window can be understood as starting at the time point corresponding to the frame that triggered the new action, and its length being the duration of the new action.

[0096] If the current time point t is within the target time window, then the frame corresponding to the current time point t can be determined to be a frame within the target time window after the frame that triggered the new action. The frame type corresponding to the current time point t is the second type. It can be understood that the frame corresponding to the current time point t (i.e. the second type frame) is a frame that changes due to the triggering of the new action. For ease of description, the second type frame is called the "consequence frame".

[0097] If the current time point t corresponds to a second type of frame (i.e., a consequence frame), in order to eliminate exposure bias, the current true frame at the current time point t... and the current predicted frame at the current time t The components are mixed and synthesized to obtain the input frame at the current time point t. .

[0098] For example, the input frame at the current time t It can be represented as: , This represents a mixed function, such as a weighted summation function.

[0099] In this embodiment, the number of consequence frames is generally multiple. To reflect the differences between different consequence frames, a progressive mixing process is also performed on different consequence frames. In some optional implementations, the above step "when the frame type corresponding to the current time point is the second type, determine the input frame at the current time point based on the current real frame and the current predicted frame at the current time point" may specifically include steps b1 to b2.

[0100] Step b1: If the frame type corresponding to the current time point is the second type, determine the first mixing coefficient based on the current time difference; the current time difference is the difference between the current time point and the trigger time of the latest action, and the first mixing coefficient and the current time difference have a monotonic relationship.

[0101] Step b2: Mix the current real frame and the current predicted frame according to the first mixing coefficient to obtain the input frame at the current time point.

[0102] In this embodiment, if the current time point t corresponds to a consequence frame, it means that a trigger frame existed in a short period of time before. At this time, the trigger time of the latest action (i.e., the latest trigger time) can be determined based on the target time window reset in the trigger frame. The difference between () and (), i.e., the current time difference And it is: Based on the current time difference You can set the time difference with the current time. The mixing coefficients that are monotonic with each other , which is the first mixing coefficient; where the two can be monotonically increasing or monotonically decreasing.

[0103] The first mixing coefficient is used to represent the importance of the current real frame or the current predicted frame when mixing the current real frame and the current predicted frame. Specifically, the closer the consequence frame is to the trigger frame, the more it should be predicted based on the current real frame, that is, the more it should be used to generate the input frame for the current time point.

[0104] For example, if the first mixing coefficient and the current time difference are monotonically increasing (i.e., positively correlated), then the first mixing coefficient can represent the mixing proportion of the current predicted frame when mixing the current real frame and the current predicted frame. If the first mixing coefficient and the current time difference are monotonically decreasing (i.e., negatively correlated), then the first mixing coefficient can represent the mixing proportion of the current real frame when mixing the current real frame and the current predicted frame.

[0105] Optionally, step b2 above, "mixing the current real frame and the current predicted frame according to the first mixing coefficient to obtain the input frame at the current time point", may include steps b21 to b22.

[0106] Step b21: Determine the first weight corresponding to the current real frame at the current time point and the second weight corresponding to the current predicted frame at the current time point based on the first mixing coefficient; the first weight is negatively correlated with the current time difference, and the second weight is positively correlated with the current time difference.

[0107] Step b22: Based on the first weight and the second weight, perform weighted processing on the current real frame and the current predicted frame to determine the input frame at the current time point.

[0108] Based on the current time difference The corresponding first mixing coefficient It can be the current real frame and the current predicted frame Set appropriate first and second weights respectively, and the current time difference The larger the value, the smaller the first weight, and the larger the second weight.

[0109] After determining the first and second weights, the current real frame can be... and the current predicted frame After weighted processing, the input frame corresponding to the resulting frame is finally obtained. .

[0110] For example, if the current time difference With the first mixing coefficient If the values ​​are monotonically increasing, then the first weight can be set to... Set the second weight to Then the input frame It can be represented as: This ensures that in the initial stages of the consequence frame, real frames dominate (with a low proportion of predictions), guaranteeing the correct initial direction of the physical consequences; as the current time difference... The proportion of predicted frames increases gradually for later consequence frames, allowing the trained model to still stably predict subsequent consequences even with its own error accumulation.

[0111] Among them, the monotonically increasing coefficient Specifically, it can be a linear function, an exponential function, a sigmoid function, or an adaptive first mixing coefficient predicted by a lightweight neural network based on the action context, etc., but this embodiment does not limit it.

[0112] For example, ,but .

[0113] Figure 5 The diagram illustrates the distribution of three types of frames in the input frame sequence. For example, taking a jumping action as an example, its duration is 8 frames (i.e., ... );like Figure 5 As shown, no new action is triggered in frames 1 to 3; these are all inertial frames. In frame 4, a new jump action is triggered (e.g., the instant the jump button is pressed). At this point, frame 4 is the trigger frame, and its corresponding input frame is a 100% real frame to anchor the physical state of the jump. The eight frames following this trigger frame are all consequence frames.

[0114] Figure 5 In this context, 'i' represents the index difference between the consequence frame and the trigger frame, which is also the relative index of the consequence frame. For example, frame 5 is the first consequence frame after the trigger frame, with i=1; frame 6 is the second consequence frame after the trigger frame, with i=2, and so on. Furthermore, using a linear function to determine the first and second weights, the input frames corresponding to each consequence frame can be represented as: This allows for a gradual transition from using primarily real frames to primarily using primarily predicted frames, that is, a linear transition from 87.5% real frames (12.5% ​​predicted frames) to 100% predicted frames, progressively exposing the model's error accumulation effect. In the subsequent 13th and 14th frames, which are the stable segments after the triggered action, the corresponding inertial frames are still fed back using predicted frames.

[0115] Weighting can be performed in the latent variable space or the embedding space, i.e., on the current real frame. and the current predicted frame Each component is mapped by the same encoder before being weighted and fused to ensure that the mixture is completed within the same representation domain.

[0116] In this embodiment, different weights are used to represent the current real frame. and the current predicted frame The input ratio is adjusted, and each consequence frame is progressively mixed. This allows the dominance of predicted frame information in the input to gradually increase as the local action consequence evolves. The ratio of predicted frames to real frames changes more smoothly, the gradient variance is smaller, and the training process is more stable, balancing accuracy and generalization. Furthermore, each action triggering event independently initiates a progressive transition within its local target time window, completely decoupled from the training process of other time steps, and does not affect each other.

[0117] Alternatively, step b2 above, "mixing the current real frame and the current predicted frame according to the first mixing coefficient to obtain the input frame at the current time point", may include the following steps c1 to c2.

[0118] Step c1: Determine either a first sampling probability (selecting the current real frame as the input frame at the current time point) or a second sampling probability (selecting the current predicted frame as the input frame at the current time point) based on the first mixing coefficient. The first sampling probability is negatively correlated with the current time difference, while the second sampling probability is positively correlated with the current time difference.

[0119] Step c2: Randomly select the current real frame or the current predicted frame as the input frame for the current time point based on the first sampling probability or the second sampling probability.

[0120] In this embodiment, when determining the first mixing coefficient Then, based on the determined sampling probability, from the current real frame Current prediction frame Choose one as the input frame for the current time point. .

[0121] Among them, it can be determined that the current real frame is selected. Input frame at the current time point The first sampling probability is used, and sampling is performed based on the first sampling probability. If the current real frame is selected... Input frame at the current time point ,but If the current prediction frame is selected... Input frame at the current time point ,but Among them, the first sampling probability and the current time difference There is a negative correlation between them; that is, the farther the consequence frame is from the trigger frame, the smaller its first sampling probability, and the current real frame is selected. Input frame at the current time point The less likely it is, the more likely it is to be. .

[0122] Similarly, the frame used to select the current prediction frame can also be determined. Input frame at the current time point The second sampling probability is used, and sampling is performed based on the second sampling probability. If the current real frame is selected... Input frame at the current time point ,but If the current prediction frame is selected... Input frame at the current time point ,but Among them, the second sampling probability and the current time difference There is a positive correlation between them; that is, the farther the consequence frame is from the trigger frame, the greater its second sampling probability. The current prediction frame is then selected. Input frame at the current time point The greater the probability, that is, the more likely it is to be .

[0123] For example, Then you can As the second sampling probability, or, As the first sampling probability. It can be understood that selecting based on the first sampling probability or the second sampling probability has the same effect: retaining a large amount of real frame information immediately after the action is triggered, and as the consequences of the action gradually unfold... (Increase), and the prediction frame ratio will increase accordingly until... The frame uses the predicted frame entirely.

[0124] The input frame at the current time point is selected based on the sampling probability. It can achieve a mixing ratio that is equivalent to the input ratio defined based on the first weight and the second weight in the desired sense, and can be applied to discrete input representation scenarios that do not support continuous interpolation.

[0125] Step S305: If the frame type corresponding to the current time point is the third type, determine the input frame of the current time point based on the current predicted frame of the current time point; the third type of frame is any frame other than the first type of frame and the second type of frame.

[0126] For the current time point t, if its corresponding frame type is neither the first type nor the second type, then its corresponding frame type is determined to be the third type. It can be understood that for the third type of frame, the content of the game screen is highly predictable, such as the player maintaining a constant speed of movement, etc. For ease of description, the third type of frame is called "inertial frame" or "normal frame", and will be mainly referred to as the third type of frame thereafter.

[0127] Furthermore, since inertial frames are predictable, even if the predicted frame is used as the context for subsequent predictions, it will not introduce large errors. Therefore, the current predicted frame at the current time point can be used directly. Determine the input frame at the current time point .

[0128] For example, the current predicted frame at the current time point can be... Input frame at the current time point ,Right now .

[0129] In summary, for the three types of frames, the input frame is determined... The specific method can be expressed as follows:

[0130] In some optional implementations, the motion information includes the amplitude of the motion. Specifically, the motion information may include: motion type. (e.g., jumping, firing, skill release, scene switching, etc.) Is it a newly triggered flag? ( and range of motion Among them, the range of motion Specifically, this could refer to the speed and intensity of the action, or the joystick offset, mouse movement, etc.

[0131] Furthermore, the aforementioned step S305, "when the frame type corresponding to the current time point is the third type, determine the input frame of the current time point based on the current predicted frame of the current time point," can specifically include steps d1 to d3.

[0132] Step d1: If the frame type corresponding to the current time point is the third type, determine the change in motion amplitude based on the motion amplitude at the current time point.

[0133] Step d2: If the change in motion amplitude is less than a preset threshold, the current predicted frame at the current time point is used as the input frame at the current time point.

[0134] Step d3: If the change in motion amplitude is greater than a preset threshold, determine the second mixing coefficient based on the change in motion amplitude, and mix the current real frame and the current predicted frame based on the second mixing coefficient to obtain the input frame at the current time point; or, if the change in motion amplitude is greater than a preset threshold, use the current predicted frame at the current time point as the input frame at the current time point, and increase the loss weight corresponding to the input frame at the current time point.

[0135] In this embodiment, if the frame type corresponding to the current time point t is the third type, that is, its corresponding inertial frame, then the low-change inertial frame and the high-change inertial frame can be further distinguished based on the change in motion amplitude, and a differentiated strategy can be adopted.

[0136] Specifically, it can be based on the magnitude of the action at the current time point t. Determine the change in the amplitude of the movement. This can represent the amount of change between frames. Specifically, it represents the change in the amplitude of the motion. Specifically, this could include the rate of change of the amplitude of movement, the L2 norm, etc. For example, Alternatively, any distance metric that satisfies monotonicity can be used. L1 norm, weighted norm, etc., to adapt to different input device types and vectorization methods.

[0137] If the change in the amplitude of the movement Less than the preset threshold This indicates that the current frame is a low-change inertial frame, such as when the player moves at a constant speed or continuously presses a key. In this case, the inertial frame is highly predictable, so the current predicted frame at the current time point can be directly used. Input frame at the current time point ,Right now .

[0138] If the change in the amplitude of the movement Greater than the preset threshold This indicates that the current frame is a high-change inertial frame, such as a sharp turn or acceleration. In this case, it can be treated similarly to a consequence frame, based on the magnitude of the change in the action. Determine the second mixing coefficient, and apply it to the current real frame. and the current predicted frame The components are mixed to obtain the input frame at the current time point. The principle is similar to the mixing method in step b2 above, and will not be elaborated further. Alternatively, the current prediction frame can be limited. The mixing ratio should be set, for example, it should not be less than a preset ratio (e.g., 50%, 80%), to ensure the prediction effect during the model inference stage.

[0139] For example, the more drastic the changes in motion, the less appropriate it is to rely 100% on predicted frames. In such cases, the real frame component can be added to improve stability and physical consistency.

[0140] Or, if the change in the amplitude of the movement Greater than the preset threshold Alternatively, the current predicted frame at the current time point can be directly used. Input frame at the current time point ,Right now At the same time, it is necessary to increase the input frame at the current time point. The corresponding loss weight; that is, for highly variable inertial frames, the established input strategy can still be used, but when calculating the loss of this frame, it can be multiplied by a larger loss weight, so that the model can pay more attention to this type of frame during training, and the model can focus its learning on frames with high changes such as sharp turns and accelerations, but which are not clearly triggered events, thus ensuring the model's performance.

[0141] It is understandable that if the change in the amplitude of the movement... equal to the preset threshold Then, it can be processed according to either step d2 or step d3; this embodiment does not limit this. Furthermore, the preset threshold... It can be a preset fixed value, for example Alternatively, it can be an engineering parameter that depends on the game type, input device, and motion vector normalization method.

[0142] In this embodiment, the predicted frame backfeed distribution during the training phase is actively simulated during inference, allowing the model to fully expose itself to its own errors during training, fundamentally reducing exposure bias and helping to eliminate the training-inference distribution gap. By mixing highly variable inertial frames or increasing the weight coefficients, the processing of inertial frames can be upgraded to a hierarchical control law driven by motion intensity, which can balance generalization and physical consistency.

[0143] Optionally, the method may further include performing a gradient truncation operation on the current prediction frame when the current prediction frame is used to generate the input frame at the current time point.

[0144] In this embodiment, to prevent gradients from backpropagating through multiple steps via the prediction frame, causing training instability, the current prediction frame... The input frame used to construct the current time point Previously, for example before step S304 or step S305, the gradient could be truncated using a gradient truncation operation, i.e.: .

[0145] in, This represents the identity transformation that stops gradient propagation; that is, for inertial frames, the predicted frame is still used. As a backfeeding input, the values ​​of the input frames remain unchanged, but the gradients are prevented from being fed back into subsequent time steps along the prediction frames during backpropagation. The prediction frame components in the consequence frames can be handled in the same way, which will not be elaborated further. Among them, inertial frames account for the majority of the total number of training frames (usually about 50%~70%), and are the main source of training for the model to learn stable prediction capabilities under its own error conditions. By isolating the gradient truncation operation from the gradient calculation of model parameters, training instability caused by gradients being backpropagated through multiple prediction frames can be prevented.

[0146] Step S306: Add the input frame at the current time point to the input frame sequence to predict the predicted frames at other time points after the current time point based on the input frame sequence.

[0147] Please see details Figure 2 Step S204 of the illustrated embodiment will not be described again here.

[0148] Figure 6 A schematic diagram of a motion-aware training model is shown. Figure 6As shown, for game training sequences containing real frame sequences and action sequences, a frame type classifier can be used for action signal analysis. The action signal analyzer reads the action signal of each frame and uniquely classifies the frame into three categories based on the priority of trigger event, consequence window (i.e., target time window), and default classification: trigger frame, consequence frame, and inertial frame, thus achieving action semantic classification. Then, differentiated input is constructed: trigger frames are taken from real frames to anchor the physical causal starting point; consequence frames are progressively mixed according to sequence number; and inertial frames are entirely fed back from predicted frames. The input frames determined by different frame types are written into the input frame sequence and used as context for the next prediction step. This allows the video generation model to obtain the next predicted frame through autoregressive inference. This process is repeated until training is complete.

[0149] Step S307: Train the video generation model based on the predicted frames at multiple time points to obtain the trained video generation model.

[0150] Please see details Figure 2 Step S205 of the illustrated embodiment will not be described again here.

[0151] The existing solutions primarily employ teacher-mandated training, using real frames as context for all frames, but lacking real frames during inference, leading to a large discrepancy between training and inference distributions and the accumulation of long-sequence errors. Random replacement training, on the other hand, randomly uses either real or predicted frames during training, potentially causing trigger frames to be trained with predicted frames. This disrupts the starting point of the action-consequence causal chain, introducing toxic supervision signals and causing the model to completely collapse in subsequent predictions at key action points. None of these solutions distinguish between the temporal structure of action triggers and consequences, making it impossible to precisely control how much real and predicted information should be used for each type of frame during training.

[0152] This application employs a hybrid training approach, differentiating between different types of frames (trigger frames, consequence frames, and inertial frames). Trigger frames are forced to use real frames for training to protect the causal starting point; consequence frames are progressively mixed to orderly expose errors; and inertial frames are fully predicted and fed back to eliminate distribution gaps. The training strategies for the three types of frames work synergistically to simultaneously achieve the dual goals of eliminating exposure biases and preserving physical logic. A comparison of the three training methods is shown in Table 2 below.

[0153] Table 2

[0154] The training method for the video generation model provided in this application determines different frame types based on action information at corresponding time points. For trigger frames, real frames are forcibly used for prediction, ensuring the accuracy of the physical causal starting point. For highly predictable inertial frames, predicted frames are used to eliminate the distribution gap between training and inference. For intermediate consequence frames, both real and predicted frames are combined for prediction, treating consequence frames as a gradual transition stage to systematically expose errors and ensure the stability of training results. This application utilizes the structured characteristics of game action signals to classify each frame in the training sequence into three semantic types and designs a dedicated input strategy for each type, achieving a rational and evidence-based elimination of exposure biases.

[0155] Optionally, the video generation model can be trained based on knowledge distillation. Specifically, the capabilities of a high-quality diffusion model teacher (undistilled version) can be transferred to a lightweight student model (few-step diffusion), while eliminating the exposure bias of the student model without affecting the accuracy of the distilled reference signal. This student model can then serve as the video generation model.

[0156] During training, a teacher-student separation strategy is employed. Specifically, all frames of the teacher model consistently use real frames to ensure the accuracy of the teacher-student comparison signal (distillation reference signal); while on the student model side, the three-frame mixing strategy provided in this embodiment is executed, and distillation supervision signals from the teacher model are received simultaneously. Through this training method, the accumulated error of the trained student model (i.e., the trained video generation model) during inference is significantly less than that of the comparison scheme using only teacher-forced training, while the distillation quality (the difference in image quality compared to the teacher model) remains unaffected.

[0157] Optionally, the loss function of the video generation model includes: an action response consistency constraint loss term; the action response consistency constraint loss term is used to represent the difference between the predicted frame and the real frame in the action response region at the first type of frame and at least part of the second type of frame.

[0158] In this embodiment, in addition to the necessary contrastive losses (such as reconstruction loss, perceptual loss, etc.), the loss function of the video generation model also introduces a loss term to constrain the consistency of action responses. This additional action response alignment loss is applied only to the differences in teacher and student outputs between the trigger frame and the consequence frame to ensure the consistency of action responses.

[0159] For example, the loss function of the video generation model (i.e., the student model) can be expressed as: .

[0160] in, Indicates the total loss. This represents the output loss between teachers and students, used to constrain student output to be close to teacher output. This represents the reconstruction loss between the predicted frame and the true frame, such as the reconstruction error in the pixel domain or latent space. This represents the perceptual loss between the predicted frame and the real frame, comparing the difference between the predicted frame and the real frame in the feature space; This represents the difference in action response calculated at the trigger frame and consequence frame positions, i.e., the action response consistency constraint loss term, to further enhance action response consistency at the trigger frame and consequence frame positions. , , These are the weighting coefficients for the corresponding loss terms.

[0161] The action response region is the key response area corresponding to the most recently triggered action. Differences in this region can include, for example, differences in the optical flow field, pixel reconstruction errors in key areas of the character / object, or differences in physical state variables obtained from the game engine. By using these differences in the action response region to constrain the model's accuracy in modeling the action-consequence causal chain, a supervision objective specific to the action causal chain is provided to the model, independent of the supervision signal provided by the reconstruction loss, in addition to the mixed input strategy. By adding action response alignment constraints to the trigger / consequence segments, distillation supervision can be made more focused on the action causal chain.

[0162] Optionally, when training the image generation model, only the teacher-mandated strategy can be implemented in the early stage of training (the first training phase), that is, the corresponding input frame is determined only based on the real frame; as the training progresses, in subsequent training phases (e.g., the second training phase after the first training phase), the inertial frame prediction backfeeding and the progressive mixing of the consequence frame are gradually enabled; the enabling progress is related to the number of training steps, but the functional form of the above mixing coefficients themselves does not change with the training progress.

[0163] In addition, a replica model with the same structure as the student model can be set up, such as an exponential moving average (EMA) version model. This replica model serves as an exponential moving average copy of the parameters of the current student model, resulting in smoother parameter updates. Subsequently, this replica model generates each prediction frame. That is, the current prediction frame is predicted based on the input frame sequence by a replica model with the same structure as the video generation model, and the parameters of this replica model are obtained by smoothing the parameters of the video generation model.

[0164] For example, ,in, This represents the parameters after the replica model is updated. This represents the parameters before the replica model was updated (i.e., the parameters determined in the last update). The parameters of the student model (which can be obtained through training) are represented. This represents the weighting coefficients. Using this replica model to determine the predicted frames and as a source of backfeeding can reduce jitter and noise in the student model's real-time output, lower the noise level of the predicted frames, and make training more stable.

[0165] The training method for the video generation model provided in this embodiment implements a differentiated strategy for trigger frames, consequence frames, and prediction frames. By introducing prediction frames as input during training, the model learns to generate high-quality next frames even when its predictions have slight errors, thereby significantly reducing the cumulative effect of errors during inference. Especially in long sequence scenes where players continuously operate for more than 30 seconds, compared with traditional teacher-forced training, the rate of quality decay is significantly reduced, and the generation quality is significantly improved. By using real frames for trigger frames, it ensures that the physical response chain after each player action has an accurate starting point, significantly reducing the probability of the action-cause chain being disordered during inference due to prediction errors during training, and improving the generation consistency of key frames related to actions. Progressively mixing of each consequence frame makes the ratio change of prediction frames and real frames smoother, the gradient variance smaller, and the training process more stable, balancing accuracy and generalization.

[0166] In some specific instances, the video generation model can be a first-person game world model based on a diffusion model, in which the game runs at 30 FPS in the scenario where it is trained, and the training data consists of game videos recorded by players (including complete keyboard and mouse operation records).

[0167] First, the training data is preprocessed, and the action type of each frame is labeled. Skill-type actions (Q / W / E / R keys) are marked as large skills, and settings are applied. (That is, the frame rate of the consequence window is 45 frames); a normal attack is marked as Threshold settings for displacement-type actions (W, A, S, D) , with normalized movement speed.

[0168] During training, for inertial frames (approximately 60% of all frames), the model's prediction from the previous frame is used as input, significantly increasing the model's robustness to its own prediction errors; threshold-based methods can be used. Distinguish between frames that are highly variable and those that are inertial. For skill trigger frames (approximately 5%), always use real frames to ensure the physical accuracy of the skill's visual effects. For skill consequence frames (approximately 35%), mix them in a progressively larger proportion. After training, generate 120 frames (4 seconds) of game footage continuously during inference. The rate of quality decay is reduced compared to traditional teacher-forced training methods, and the consistency of physical logic is significantly improved.

[0169] This method can be applied to various scenarios. For example, in the training environment of AI Agents in highly dynamic competitive games, such as action games where skill combos are the core gameplay, training the AI ​​Agent requires a virtual training environment that can accurately respond to high-frequency skill triggers. In this embodiment, the physical response fidelity of skill trigger frames is guaranteed, and the physical feedback signals received by the AI ​​Agent during training are accurate and reliable, which helps to train game AI with more precise action responses.

[0170] Alternatively, in AI-assisted generation scenarios for cinematic cutscenes and plot content, such as in the production of story-driven games, plot events (boss appearance, scene transitions, plot triggers) have extremely high requirements for visual quality and are often the nodes most prone to quality degradation during long-term sequence generation. In this embodiment, the progressive blending strategy of consequence frames ensures that subsequent frames after a plot event maintain visual consistency with the trigger frame, providing a guarantee for the AI-assisted generation of high-quality plot animations.

[0171] This embodiment also provides a video generation method, which can be used in the aforementioned terminal devices, such as mobile terminals. Figure 7 This is a flowchart of a video generation method according to an embodiment of this application, such as... Figure 7 As shown, the process includes the following steps.

[0172] Step S701: Obtain the historical frame sequence; the historical frame sequence includes historical frames predicted based on the trained video generation model, which is trained based on the training method of the video generation model provided in any of the above method embodiments.

[0173] Step S702: Perform autoregressive inference on the historical frame sequence based on the trained video generation model to determine the latest frame.

[0174] Step S703: Output the latest frame and add it as a new historical frame to the historical frame sequence.

[0175] In this embodiment, after obtaining the trained video generation model, inference can be performed using the model. During the inference phase, the historical frame sequence is used as the context sequence (functionally the same as the input frame sequence in the training phase). The historical frame sequence is input into the trained video generation model, enabling it to perform autoregressive inference and generate the video frame corresponding to the latest time point, i.e., the latest frame. This latest frame can then be output, for example, displayed on the device's interface. Furthermore, this latest frame is also added to the historical frame sequence as a new historical frame to update the historical frame sequence. Based on the updated historical frame sequence, the next video frame is predicted and generated again. This process is repeated to predict and generate a video containing all the video frames.

[0176] This embodiment also provides a training device for a video generation model, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0177] This embodiment provides a training device for a video generation model, such as... Figure 8 As shown, the device includes: The acquisition module 801 is used to acquire a real frame sequence and a corresponding action sequence; the real frame sequence includes real frames at multiple time points, and the action sequence includes action information corresponding to the real frames; The classification module 802 is used to determine the frame type corresponding to the current time point based on the action information of the current time point in the action sequence for the current time point; Processing module 803 is configured to determine the input frame at the current time point based on the frame type corresponding to the current time point, and on the current real frame and / or the current predicted frame at the current time point; wherein the current predicted frame is predicted by the video generation model to be trained based on the input frame sequence before the current time point, and the input frame sequence includes input frames at multiple time points; when determining the input frame at the current time point, the contribution ratio of the current real frame to the current predicted frame is associated with the frame type at the current time point; Adding module 804 is used to add the input frame at the current time point to the input frame sequence, so as to predict the prediction frame at other time points after the current time point based on the input frame sequence; The training module 805 is used to train the video generation model based on the predicted frames at multiple time points to obtain the trained video generation model.

[0178] In some optional implementations, determining the input frame for the current time point based on the frame type corresponding to the current time point, and based on the current real frame and / or the current predicted frame for the current time point, includes: If the frame type corresponding to the current time point is the first type, the input frame for the current time point is determined based on the current real frame at the current time point; the frame of the first type is the frame that triggers the new action. If the frame type corresponding to the current time point is the second type, the input frame of the current time point is determined based on the current real frame and the current predicted frame of the current time point; the frame of the second type is the frame within the target time window after the frame that triggers the new action; If the frame type corresponding to the current time point is the third type, the input frame for the current time point is determined based on the current predicted frame for the current time point; the third type of frame is any frame other than the first type of frame and the second type of frame.

[0179] In some optional implementations, determining the frame type corresponding to the current time point based on the action information at the current time point in the action sequence includes: Based on the action information at the current time point, determine whether the frame type corresponding to the current time point is the first type; If the frame type corresponding to the current time point is of type 1, the target time window is reset; the target time window is used to represent the duration of the latest triggered action. If the frame type corresponding to the current time point is not of the first type, and if the current time point is within the target time window, then the frame type corresponding to the current time point is determined to be of the second type.

[0180] In some optional implementations, the action information includes: action type and / or an action flag bit indicating whether a new action is triggered; the action flag bit is used to determine whether the frame type corresponding to the current time point is a first type; The reset target time window includes: Reset the target time window based on the action type corresponding to the current time point.

[0181] In some optional implementations, the reset target time window includes: Based on the action information at the latest trigger time and the latent space characteristics of historical time points prior to the latest trigger time, the duration corresponding to the triggered new action is predicted; the latest trigger time is the trigger time that triggers the latest action. The target time window is reset based on the predicted duration.

[0182] In some optional implementations, determining the input frame at the current time point based on the current real frame and the current predicted frame at the current time point includes: A first mixing coefficient is determined based on the current time difference; the current time difference is the difference between the current time point and the trigger time of the latest action, and the first mixing coefficient and the current time difference have a monotonic relationship. The current real frame and the current predicted frame are mixed according to the first mixing coefficient to obtain the input frame at the current time point.

[0183] In some optional implementations, the step of mixing the current real frame and the current predicted frame according to the first mixing coefficient to obtain the input frame at the current time point includes: A first sampling probability is determined based on the first mixing coefficient to select the current real frame as the input frame at the current time point, or a second sampling probability is determined to select the current predicted frame as the input frame at the current time point; the current real frame or the current predicted frame is randomly selected as the input frame at the current time point based on the first sampling probability or the second sampling probability; the first sampling probability is negatively correlated with the current time difference, and the second sampling probability is positively correlated with the current time difference; or, The first weight corresponding to the current real frame at the current time point and the second weight corresponding to the current predicted frame at the current time point are determined based on the first mixing coefficient; the first weight is negatively correlated with the current time difference, and the second weight is positively correlated with the current time difference; the current real frame and the current predicted frame are weighted based on the first weight and the second weight to determine the input frame at the current time point.

[0184] In some optional implementations, the motion information includes the amplitude of the motion; When the frame type corresponding to the current time point is the third type, determining the input frame for the current time point based on the current predicted frame includes: If the frame type corresponding to the current time point is the third type, the change in motion amplitude is determined based on the motion amplitude at the current time point; If the change in the magnitude of the action is less than a preset threshold, the current predicted frame at the current time point is used as the input frame at the current time point. If the change in motion amplitude is greater than a preset threshold, a second mixing coefficient is determined based on the change in motion amplitude, and the current real frame and the current predicted frame are mixed based on the second mixing coefficient to obtain the input frame at the current time point; or, if the change in motion amplitude is greater than a preset threshold, the current predicted frame at the current time point is used as the input frame at the current time point, and the loss weight corresponding to the input frame at the current time point is increased.

[0185] In some optional implementations, the loss function of the video generation model includes: an action-response consistency constraint loss term; The motion response consistency constraint loss term is used to represent the difference between the predicted frame and the real frame in the motion response region at the first type of frame and at least part of the second type of frame.

[0186] In some optional implementations, the processing module is further configured to: perform a gradient truncation operation on the current prediction frame when the current prediction frame is used to generate the input frame at the current time point.

[0187] This embodiment also provides a video generation device, such as... Figure 9 As shown, the device includes: The sequence acquisition module 901 is used to acquire a historical frame sequence; the historical frame sequence includes historical frames predicted based on a trained video generation model, and the trained video generation model is trained based on the training method of the video generation model described above. Inference module 902 is used to perform autoregressive inference on the historical frame sequence based on the trained video generation model to determine the latest frame; The output module 903 is used to output the latest frame and add the latest frame as a new historical frame to the historical frame sequence.

[0188] The training apparatus for the video generation model provided in this disclosure can execute the training method for the video generation model provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0189] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0190] The following is a detailed reference. Figure 10 The diagram illustrates a structural schematic suitable for implementing the electronic device described in the embodiments of this application. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from memory 1008 into random access memory (RAM) 1003. The RAM 1003 also stores various programs and data required for the operation of the electronic device. The processor 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0191] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; memory devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 10 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0192] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from memory 1008, or installed from ROM 1002. When the computer program is executed by processor 1001, it performs the functions defined in the training method of the video generation model or the video generation method of embodiments of this application.

[0193] Figure 10 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0194] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the training method or video generation method of the video generation model shown in the above embodiments is implemented.

[0195] A portion of this application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to this application through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0196] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method for training a video generation model, the method comprising: The method includes: Obtain a real frame sequence and a corresponding action sequence; the real frame sequence includes real frames at multiple time points, and the action sequence includes action information corresponding to the real frames; For the current time point, the frame type corresponding to the current time point is determined based on the action information of the current time point in the action sequence; Based on the frame type corresponding to the current time point, and based on the current real frame and / or the current predicted frame at the current time point, the input frame at the current time point is determined; wherein, the current predicted frame is predicted by the video generation model to be trained based on the input frame sequence before the current time point, and the input frame sequence includes input frames at multiple time points; when determining the input frame at the current time point, the contribution ratio of the current real frame to the current predicted frame is associated with the frame type at the current time point; The input frame at the current time point is added to the input frame sequence to predict the predicted frames at other time points after the current time point based on the input frame sequence; The video generation model is trained by predicting frames at multiple time points to obtain the trained video generation model.

2. The method of claim 1, wherein, The step of determining the input frame for the current time point based on the frame type corresponding to the current time point, and based on the current real frame and / or the current predicted frame for the current time point, includes: If the frame type corresponding to the current time point is the first type, the input frame for the current time point is determined based on the current real frame at the current time point; the frame of the first type is the frame that triggers the new action. If the frame type corresponding to the current time point is the second type, the input frame of the current time point is determined based on the current real frame and the current predicted frame of the current time point; the frame of the second type is the frame within the target time window after the frame that triggers the new action; If the frame type corresponding to the current time point is the third type, the input frame for the current time point is determined based on the current predicted frame for the current time point; the third type of frame is any frame other than the first type of frame and the second type of frame.

3. The method of claim 2, wherein, Determining the frame type corresponding to the current time point based on the action information at the current time point in the action sequence includes: Based on the action information at the current time point, determine whether the frame type corresponding to the current time point is the first type; If the frame type corresponding to the current time point is of type 1, the target time window is reset; the target time window is used to represent the duration of the latest triggered action. If the frame type corresponding to the current time point is not of the first type, and if the current time point is within the target time window, then the frame type corresponding to the current time point is determined to be of the second type.

4. The method of claim 3, wherein, The action information includes: action type and / or an action flag bit indicating whether a new action is triggered; the action flag bit is used to determine whether the frame type corresponding to the current time point is the first type. The reset target time window includes: Reset the target time window based on the action type corresponding to the current time point.

5. The method of claim 3, wherein, The reset target time window includes: Based on the action information at the latest trigger time and the latent space characteristics of historical time points prior to the latest trigger time, the duration corresponding to the triggered new action is predicted; the latest trigger time is the trigger time that triggers the latest action. The target time window is reset based on the predicted duration.

6. The method of claim 2, wherein, The step of determining the input frame at the current time point based on the current real frame and the current predicted frame at the current time point includes: A first mixing coefficient is determined based on the current time difference; the current time difference is the difference between the current time point and the trigger time of the latest action, and the first mixing coefficient and the current time difference have a monotonic relationship. The current real frame and the current predicted frame are mixed according to the first mixing coefficient to obtain the input frame at the current time point.

7. The method of claim 6, wherein, The step of mixing the current real frame and the current predicted frame according to the first mixing coefficient to obtain the input frame at the current time point includes: A first sampling probability is determined based on the first mixing coefficient to select the current real frame as the input frame at the current time point, or a second sampling probability is determined to select the current predicted frame as the input frame at the current time point; the current real frame or the current predicted frame is randomly selected as the input frame at the current time point based on the first sampling probability or the second sampling probability; the first sampling probability is negatively correlated with the current time difference, and the second sampling probability is positively correlated with the current time difference; or, The first weight corresponding to the current real frame at the current time point and the second weight corresponding to the current predicted frame at the current time point are determined based on the first mixing coefficient; the first weight is negatively correlated with the current time difference, and the second weight is positively correlated with the current time difference; the current real frame and the current predicted frame are weighted based on the first weight and the second weight to determine the input frame at the current time point.

8. The method of claim 2, wherein, The action information includes the range of motion; When the frame type corresponding to the current time point is the third type, determining the input frame for the current time point based on the current predicted frame includes: If the frame type corresponding to the current time point is the third type, the change in motion amplitude is determined based on the motion amplitude at the current time point; If the change in the magnitude of the action is less than a preset threshold, the current predicted frame at the current time point is used as the input frame at the current time point. If the change in motion amplitude is greater than a preset threshold, a second mixing coefficient is determined based on the change in motion amplitude, and the current real frame and the current predicted frame are mixed based on the second mixing coefficient to obtain the input frame at the current time point; or, if the change in motion amplitude is greater than a preset threshold, the current predicted frame at the current time point is used as the input frame at the current time point, and the loss weight corresponding to the input frame at the current time point is increased.

9. The method of claim 1, wherein, The loss function of the video generation model includes: an action-response consistency constraint loss term; The motion response consistency constraint loss term is used to represent the difference between the predicted frame and the real frame in the motion response region at the first type of frame and at least part of the second type of frame.

10. A method of video generation, the method comprising: include: Obtain a historical frame sequence; the historical frame sequence includes historical frames predicted based on a trained video generation model, wherein the trained video generation model is trained based on the training method of the video generation model according to any one of claims 1 to 9; The latest frame is determined by performing autoregressive inference on the historical frame sequence based on the trained video generation model. Output the latest frame and add it as a new historical frame to the historical frame sequence.

11. An apparatus for training a video generation model, comprising: The device includes: The acquisition module is used to acquire a real frame sequence and a corresponding action sequence; the real frame sequence includes real frames at multiple time points, and the action sequence includes action information corresponding to the real frames; The classification module is used to determine the frame type corresponding to the current time point based on the action information of the current time point in the action sequence. The processing module is configured to determine the input frame at the current time point based on the frame type corresponding to the current time point, and on the current real frame and / or the current predicted frame at the current time point; wherein the current predicted frame is predicted by the video generation model to be trained based on the input frame sequence before the current time point, and the input frame sequence includes input frames at multiple time points; when determining the input frame at the current time point, the contribution ratio of the current real frame to the current predicted frame is associated with the frame type at the current time point; An adding module is used to add the input frame at the current time point to the input frame sequence, so as to predict the prediction frame at other time points after the current time point based on the input frame sequence; The training module is used to train the video generation model based on predicted frames at multiple time points, resulting in a trained video generation model.

12. A video generating apparatus characterized by comprising: The device includes: A sequence acquisition module is used to acquire a historical frame sequence; the historical frame sequence includes historical frames predicted based on a trained video generation model, wherein the trained video generation model is trained based on the training method of the video generation model according to any one of claims 1 to 9; The inference module is used to perform autoregressive inference on the historical frame sequence based on the trained video generation model to determine the latest frame; The output module is used to output the latest frame and add the latest frame as a new historical frame to the historical frame sequence.

13. An electronic device, comprising: include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1 to 10.

Images