A method and system for generating interactive game videos based on state prediction
By introducing a state prediction module and a feedback mechanism into the game video generation model, the problem that user operations cannot affect the game process in existing technologies is solved, achieving both user playability and efficient video generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GIANT MOBILE TECH CO LTD
- Filing Date
- 2026-03-09
- Publication Date
- 2026-06-02
AI Technical Summary
Existing interactive game video generation models lack explicit modeling of key states during gameplay, resulting in generated videos that fail to respond to user actions and produce reasonable feedback that conforms to the game rules. Users cannot change the game's progress through interaction, leading to a lack of playability.
We introduce an interactive game video generation method based on state prediction. By setting up a state prediction module and a state feedback mechanism during inference, we use learnable space query and deep learning model to predict the game state and use it as a conditional input for video generation, ensuring that user operations can affect the game process in real time.
It enables user actions to realistically influence the game process and obtain timely feedback in accordance with the game rules, upgrading the video generation model from watchable to playable, while the increase in inference latency is negligible, maintaining high-efficiency video generation.
Smart Images
Figure CN122134882A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for generating interactive game videos based on state prediction. Background Technology
[0002] With technological advancements, existing interactive game video generation processes face the following key technical bottlenecks: user input during video generation (such as mouse clicks and button presses) is typically used only as conditional signals to guide changes in visual style or coarse-grained actions. However, because the generation model lacks explicit modeling of key states during gameplay, the generated video, while visually coherent, fails to respond to user actions and provide reasonable feedback consistent with game rules. Users essentially only watch a video influenced by interaction, unable to alter the game's progress or receive immediate, effective status feedback as in a real game, thus lacking true playability.
[0003] Therefore, it is necessary to provide a method and system for generating interactive game videos based on state prediction, so as to alleviate the problem that the original interactive game video generation model can only watch interactively generated videos and realize the ability to actually play the generated games. Summary of the Invention
[0004] The purpose of this invention is to provide a method and system for generating interactive game videos based on state prediction, thereby alleviating the limitations of existing interactive game video generation models that only allow viewing of interactively generated videos, and enabling users to actually play the generated games.
[0005] To address the problems existing in the prior art, this invention provides a method for generating interactive game videos based on state prediction, comprising the following steps:
[0006] S1: Set up a state prediction header based on the learnable space query to complete the state prediction;
[0007] S2: Set up a state feedback mechanism during inference. Based on the given historical frames, user operation sequence, and historical state sequence, output the prediction noise of the frame to be generated and the state of the next frame after the generated frame.
[0008] Optionally, in the interactive game video generation method based on state prediction, state prediction is performed as follows:
[0009] S11: Perform adaptive normalization on the spatiotemporal feature map, consistent with the backbone of the diffusion model.
[0010] S12: Introduces learnable spatial location encoding, which uses a shared learnable query vector and spatial dimension features to perform cross-attention calculation, dynamically aggregating the region information most relevant to the state within each frame;
[0011] S13: Perform time-dimensional average pooling on the aggregated features and output a deep learning model of discrete game states through a linear projection layer.
[0012] S14: Complete state prediction based on deep learning model.
[0013] Optionally, in the state prediction-based interactive game video generation method, the state feedback during inference is as follows:
[0014] During the reasoning process, the first... At frame rate, the model will be in the currently known state. Based on historical video frames and user operation sequences, generate the first... The video frame is used to simultaneously infer the predicted state of the next frame. ,
[0015] Predicted state Will be used to generate the first The conditional input of frames, this process, while strictly maintaining causal dependence, enables users to achieve real control over the game process through user interaction, and is also used to realize playable interactions.
[0016] This invention also provides an interactive game video generation system based on state prediction, comprising:
[0017] The state prediction module is configured to perform state prediction based on the state prediction header queried from the learnable space.
[0018] The inference-time state feedback mechanism is configured to output the prediction noise of the frame to be generated and the state of the next frame of the generated frame based on the given historical frames, user operation sequence, and historical state sequence.
[0019] Optionally, in the state prediction-based interactive game video generation system, state prediction is performed as follows:
[0020] Adaptive normalization, consistent with the backbone of the diffusion model, is applied to the spatiotemporal feature map.
[0021] We introduce learnable spatial location encoding and use a shared learnable query vector and spatial dimension features to perform cross-attention calculation, dynamically aggregating the region information most relevant to the state within each frame.
[0022] The aggregated features are subjected to time-dimensional average pooling, and a deep learning model of discrete game states is output by a linear projection layer.
[0023] State prediction is accomplished using a deep learning model.
[0024] Optionally, in the state prediction-based interactive game video generation system, the state feedback method during inference is as follows:
[0025] During the reasoning process, the first... At frame rate, the model will be in the currently known state. Based on historical video frames and user operation sequences, generate the first... The video frame is used to simultaneously infer the predicted state of the next frame. ,
[0026] Predicted state Will be used to generate the first The conditional input of frames, this process, while strictly maintaining causal dependence, enables users to achieve real control over the game process through user interaction, and is also used to realize playable interactions.
[0027] Compared with the prior art, the present invention has the following advantages:
[0028] (1) Support for interactive feedback based on real game logic: By training the video generation model using this method, the video state of the next frame can be predicted while the video is being generated in real time. This predicted state is then used as the conditional input for subsequent generation steps. User operations can not only guide changes in the game screen, but also realistically affect the game process through state evolution, obtaining timely and reasonable responses that conform to the game rules, thus upgrading the video generation model from something that can be watched to something that can be played.
[0029] (2) High-efficiency video generation: The state prediction module has a lightweight and plug-and-play structure, which can be seamlessly integrated into mainstream spatiotemporal diffusion Transformer architectures such as DiT without modifying the backbone network structure. The predicted state is injected into the generation process as a conditional signal through an adaptive LayerNorm mechanism, which is naturally compatible with the existing conditional control process. Furthermore, since only a lightweight prediction head based on learnable space query is introduced, the increase in overall inference latency is negligible, achieving playability while maintaining high-efficiency video generation. Attached Figure Description
[0030] Figure 1 This is a schematic diagram of the interactive game video generation process provided in an embodiment of the present invention;
[0031] Figure 2 This is a schematic diagram of a DiT Block provided for an embodiment of the present invention. Detailed Implementation
[0032] The specific embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.
[0033] In the following, if the methods described herein include a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.
[0034] To address the problems existing in the prior art, this invention provides a method and system for generating interactive game videos based on state prediction, mainly including a state prediction module and a state feedback mechanism during inference. This invention introduces dual conditional control of actions and states based on the standard Diffusion Transformer (DiT) architecture: on the one hand, user operations are encoded as external condition vectors; on the other hand, discrete game states are mapped to the latent space through a learnable embedding matrix and fused with the time-step embedding as the control signal for adaptive LayerNorm.
[0035] like Figure 1 and 2 As shown, an interactive game video generation method based on state prediction includes the following steps:
[0036] S1: In order to accurately capture the key spatial information that determines the game state, a state prediction head based on learnable space query is set up to complete the state prediction.
[0037] The state prediction is completed as follows:
[0038] S11: Perform adaptive normalization (LayerNorm) on the spatiotemporal feature map in line with the backbone of the diffusion model.
[0039] S12: Introduces learnable spatial location encoding, which uses a shared learnable query vector and spatial dimension features to perform cross-attention calculation, dynamically aggregating the region information most relevant to the state within each frame;
[0040] S13: Perform time-dimensional average pooling on the aggregated features and output a deep learning model (logits) of discrete game states through a linear projection layer.
[0041] S14: Complete state prediction based on deep learning model.
[0042] This design improves the recognition of rare but critical states, such as character collisions.
[0043] S2: Set up a state feedback mechanism during inference. Based on the given historical frames, user operation sequence, and historical state sequence, output the prediction noise of the frame to be generated and the state of the next frame after the generated frame.
[0044] Specifically, the state feedback method during reasoning is as follows:
[0045] During the reasoning process, the first... At frame rate, the model will be in the currently known state. Based on historical video frames and user operation sequences, generate the first... The video frame is used to simultaneously infer the predicted state of the next frame. ,
[0046] Predicted state Will be used to generate the first The conditional input of frames, this process, while strictly maintaining causal dependence, enables users to achieve real control over the game process through user interaction, and is also used to realize playable interactions.
[0047] This invention also provides an interactive game video generation system based on state prediction, comprising:
[0048] The state prediction module is configured to perform state prediction based on the state prediction header queried from the learnable space.
[0049] The inference-time state feedback mechanism is configured to output the prediction noise of the frame to be generated and the state of the next frame of the generated frame based on the given historical frames, user operation sequence, and historical state sequence.
[0050] Furthermore, the method for completing state prediction is as follows:
[0051] Adaptive normalization (LayerNorm) is applied to the spatiotemporal feature map in accordance with the backbone of the diffusion model.
[0052] We introduce learnable spatial location encoding and use a shared learnable query vector and spatial dimension features to perform cross-attention calculation, dynamically aggregating the region information most relevant to the state within each frame.
[0053] The aggregated features are averaged over time, and a deep learning model (logits) that outputs discrete game states is generated by a linear projection layer.
[0054] State prediction is accomplished using a deep learning model.
[0055] This design improves the recognition of rare but critical states, such as character collisions.
[0056] The state feedback method during inference is as follows:
[0057] During the reasoning process, the first... At frame rate, the model will be in the currently known state. Based on historical video frames and user operation sequences, generate the first... The video frame is used to simultaneously infer the predicted state of the next frame. ,
[0058] Predicted state Will be used to generate the first The conditional input of frames, this process, while strictly maintaining causal dependence, enables users to achieve real control over the game process through user interaction, and is also used to realize playable interactions.
[0059] Compared with the prior art, the present invention has the following advantages:
[0060] (1) Support for interactive feedback based on real game logic: By training the video generation model using this method, the video state of the next frame can be predicted while the video is being generated in real time. This predicted state is then used as the conditional input for subsequent generation steps. User operations can not only guide changes in the game screen, but also realistically affect the game process through state evolution, obtaining timely and reasonable responses that conform to the game rules, thus upgrading the video generation model from something that can be watched to something that can be played.
[0061] (2) High-efficiency video generation: The state prediction module has a lightweight and plug-and-play structure, which can be seamlessly integrated into mainstream spatiotemporal diffusion Transformer architectures such as DiT without modifying the backbone network structure. The predicted state is injected into the generation process as a conditional signal through an adaptive LayerNorm mechanism, which is naturally compatible with the existing conditional control process. Furthermore, since only a lightweight prediction head based on learnable space query is introduced, the increase in overall inference latency is negligible, achieving playability while maintaining high-efficiency video generation.
[0062] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.
Claims
1. A method for generating interactive game videos based on state prediction, characterized in that, Includes the following steps: S1: Set up a state prediction header based on the learnable space query to complete the state prediction; S2: Set up a state feedback mechanism during inference. Based on the given historical frames, user operation sequence, and historical state sequence, output the prediction noise of the frame to be generated and the state of the next frame after the generated frame.
2. The interactive game video generation method based on state prediction as described in claim 1, characterized in that, The state prediction is completed as follows: S11: Perform adaptive normalization on the spatiotemporal feature map, consistent with the backbone of the diffusion model. S12: Introduces learnable spatial location encoding, which uses a shared learnable query vector and spatial dimension features to perform cross-attention calculation, dynamically aggregating the region information most relevant to the state within each frame; S13: Perform time-dimensional average pooling on the aggregated features and output a deep learning model of discrete game states through a linear projection layer. S14: Complete state prediction based on deep learning model.
3. The interactive game video generation method based on state prediction as described in claim 2, characterized in that, The state feedback method during inference is as follows: During the reasoning process, the first... At frame rate, the model will be in the currently known state. Based on historical video frames and user operation sequences, generate the first... The video frame is used to simultaneously infer the predicted state of the next frame. , Predicted state Will be used to generate the first The conditional input of frames, this process, while strictly maintaining causal dependence, enables users to achieve real control over the game process through user interaction, and is also used to realize playable interactions.
4. An interactive game video generation system based on state prediction, characterized in that, include: The state prediction module is configured to perform state prediction based on the state prediction header queried from the learnable space. The inference-time state feedback mechanism is configured to output the prediction noise of the frame to be generated and the state of the next frame of the generated frame based on the given historical frames, user operation sequence, and historical state sequence.
5. The interactive game video generation system based on state prediction as described in claim 4, characterized in that, The method for completing state prediction is as follows: Adaptive normalization, consistent with the backbone of the diffusion model, is applied to the spatiotemporal feature map. We introduce learnable spatial location encoding and use a shared learnable query vector and spatial dimension features to perform cross-attention calculation, dynamically aggregating the region information most relevant to the state within each frame. The aggregated features are subjected to time-dimensional average pooling, and a deep learning model of discrete game states is output by a linear projection layer. State prediction is accomplished using a deep learning model.
6. The interactive game video generation system based on state prediction as described in claim 5, characterized in that, The state feedback method during inference is as follows: During the reasoning process, the first... At frame rate, the model will be in the currently known state. Based on historical video frames and user operation sequences, generate the first... The video frame is used to simultaneously infer the predicted state of the next frame. , Predicted state Will be used to generate the first The conditional input of frames, this process, while strictly maintaining causal dependence, enables users to achieve real control over the game process through user interaction, and is also used to realize playable interactions.