A method for real-time generation of interactive pixel-style game videos

Abstract

This invention relates to the field of game video generation technology, and particularly to a method for real-time generation of interactive pixel-style game videos. The method includes the following steps: establishing a real-time interactive driving mechanism based on a diffusion transformer; establishing a memory consistency constraint mechanism based on a Transformer memory mapping module; constructing a pixel-style optimization module using an encoder and decoder; reconstructing pixel-style video frames using an end-to-end diffusion transformer architecture; and generating interactive pixel-style game videos in real-time based on these pixel-style video frames. This supports core scenarios such as real-time development and live interactive streaming of pixel-style games.

Description

Technical Field

[0001] This invention relates to the field of game video generation technology, and in particular to a method for real-time generation of interactive pixel-style game videos. Background Technology

[0002] Existing pixel-style game video generation technology suffers from the following key flaws, severely restricting the requirements for real-time interaction and consistency:

[0003] Lack of real-time interactivity: Mainstream models (such as Diffusion-based video generation tools) use offline batch processing for generation, which cannot respond to keyboard / mouse streaming input. The interaction latency is generally >500ms, causing user operations (such as character movement commands) to be disconnected from video generation, making it impossible to achieve a "input-as-feedback" gamified experience, and only suitable for static preview scenarios.

[0004] Scene consistency collapse: Existing methods rely on simple inter-frame interpolation (such as optical flow), which results in significant drift of pixel elements (such as terrain textures and character clothing) when generating videos longer than 10 minutes. For example, the size of the same pixel tree may change by 50% in consecutive frames, and the lighting style may be inconsistent, violating the visual consistency requirements of pixel-style games.

[0005] Low computational efficiency: To maintain consistency, the model needs to store the entire historical state (such as the feature vector for each frame), resulting in a computational complexity of O(n²). At 1080P resolution, generating a 1-second video requires 12 seconds of computation time, with a frame rate of <10fps, which cannot meet the requirements for real-time interaction (requiring >20fps).

[0006] Style matching distortion: When inputting pixel-style images, the generated video often incorporates realistic rendering elements (such as high-resolution details and soft lighting effects), destroying the unique low-resolution, high-recognition characteristics of pixel art. For example, inputting an 8-bit style image may result in a 16-bit gradient color output video, causing a style break.

[0007] Lack of pixel style adaptation: Existing models (such as Sora) are trained for realistic videos and have not optimized for pixel style constraints. With low-resolution inputs (such as 64×64 pixel images), the generated video details are blurry and cannot reproduce game-level pixel logic (such as collision detection and frame animation), resulting in a significant reduction in practicality.

[0008] Therefore, it is necessary to provide a method for real-time generation of interactive pixel-style game videos to support core scenarios such as real-time development and live interactive streaming of pixel-style games. Summary of the Invention

[0009] The purpose of this invention is to provide a method for real-time generation of interactive pixel-style game videos to support core scenarios such as real-time development and live interactive streaming of pixel-style games.

[0010] To address the problems existing in the prior art, this invention provides a method for real-time generation of interactive pixel-style game videos, comprising the following steps:

[0011] A real-time interactive driving mechanism is established based on the diffusion converter;

[0012] A memory consistency constraint mechanism is established based on the Transformer memory mapping module;

[0013] A pixel-style optimization module was built using an encoder and decoder.

[0014] An end-to-end diffusion converter architecture is adopted to reconstruct pixel-style video frames and generate interactive pixel-style game videos in real time based on the pixel-style video frames.

[0015] Optionally, in the real-time generation method of the interactive pixel-style game video, a real-time interactive driving mechanism is established, as follows:

[0016] By embedding keyboard and mouse streaming input into the diffusion transformer generation process, millisecond-level response is achieved. The keyboard and mouse streaming input is as follows: ,in, Indicates streaming input. Represents a time frame. Represents the input vector space. This indicates the number of types of keyboard and mouse streaming operations;

[0017] Interactive signals are incorporated into the latent space through linear transformation:

[0018] ;

[0019] in, For a moment Latent variables, This is the interaction weight matrix. Indicates streaming input. For a moment -1 is a hidden variable. This is the noise term.

[0020] Optionally, in the real-time generation method of the interactive pixel-style game video, a memory consistency constraint mechanism is established, as follows:

[0021] A dynamic context memory is introduced to store key scene features, including pixel element positions and lighting patterns, and a consistency loss function is designed:

[0022] ;

[0023] in, Key scene features This is a Transformer-based memory mapping module. Here, t represents the weighting coefficient, t is the t-th frame, and T is the total number of frames. For the key scene features of frame t-1, These are the key scene features for frame 0.

[0024] Optionally, in the real-time generation method of interactive pixel-style game videos, a pixel-style optimization module is constructed as follows:

[0025] Embed style constraints in the encoder and / or decoder:

[0026] ;

[0027] in, This is a pixel-style extractor that forces the output video to maintain a low resolution consistency with the input image. For input video timing The video frames, This is the input image.

[0028] Optionally, in the interactive pixel-style game video real-time generation method, the end-to-end diffusion transformer architecture is: VIT-VAE Encoder → DiT → VIT-VAE Decoder.

[0029] Optionally, in the real-time generation method of interactive pixel-style game videos, an end-to-end diffusion transformer architecture is adopted to reconstruct pixel-style video frames, as follows:

[0030] The encoder will input the image Encoded as , for Latent feature encoding wherein is a dimensionality space ;

[0031] Interactive input Plus Generate through multiple self-attention , for Encoding of latent features after encoding;

[0032] The decoder will Reconstructed into pixelated video frames .

[0033] Compared with the prior art, the present invention has the following advantages:

[0034] (1) Real-time interactive response: Through keyboard or mouse streaming input, user operations (such as movement and jump commands) can dynamically affect the video generation process in real time, ensuring that the interaction delay is less than 100ms, supporting a smooth "input-generation" closed loop, and completely solving the problems of static generation and interaction gaps in existing models.

[0035] (2) Supports single-image gamified video generation: Users only need to input a pixel-style image (such as a level sketch or character illustration), and the system will automatically parse the elements and generate a continuous video sequence containing complete game logic (such as character behavior and scene events). No additional scripts or complex configurations are required, which greatly reduces the creation threshold.

[0036] (3) Ensure long-term scene consistency: Innovative design of "memory-consistency constraint mechanism" to dynamically store key scene features (such as pixel element position and lighting style), and verify and correct style drift in real time during video generation to ensure that scene elements (such as terrain and environment) of videos longer than 3 minutes remain highly consistent.

[0037] (4) Optimize computational efficiency: Based on the DiT architecture (VIT-VAE encoder → DiT → VIT-VAE decoder), an efficient model is built, which reduces computational complexity and supports a real-time generation speed of 50ms / frame. At the same time, it retains the low-resolution and high-recognition style unique to pixel art, and solves the computational bottleneck caused by storing historical data in traditional models.

[0038] (5) Pixel Art Style Preservation: The style constraint module ensures that the input pixel style is preserved at a rate of 98.7%. When the input is an 8-bit style image (such as a 32×32 pixel image), the 8-bit texture recognition of the output video is improved by 40% (from 58.2% to 98.7%), completely avoiding interference from realistic rendering. In the Minecraft scene, the character color consistency score reaches 4.8 / 5.0 (compared to only 2.1 / 5.0 for traditional models).

[0039] (6) This invention promotes the upgrade of pixel-style games from "static preview" to "dynamic co-creation", providing real-time, efficient and consistent video generation capabilities for development, live streaming, education and other scenarios. Attached Figure Description

[0040] Figure 1 A flowchart of the generation method provided in an embodiment of the present invention. Detailed Implementation

[0041] The specific embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.

[0042] In the following, if the methods described herein include a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.

[0043] To address the problems existing in the prior art, this invention provides a method for real-time generation of interactive pixel-style game videos. The core innovation lies in integrating real-time interaction, scene consistency, and pixel-style optimization. The algorithm uses a single input pixel image (… Starting with dynamic interactive input, video generation is driven by dynamic interactive input, while long-term consistency is ensured through a memory mechanism.

[0044] like Figure 1 As shown, the method includes the following steps:

[0045] S1: Establish a real-time interactive driving mechanism based on the diffusion converter, as follows:

[0046] By embedding keyboard and mouse streaming input into the Diffusion Transformer (DiT) generation process, millisecond-level response is achieved. The keyboard and mouse streaming input is as follows: ,in, Indicates streaming input. Represents a time frame. Represents the input vector space. This indicates the number of types of keyboard and mouse streaming operations;

[0047] Interactive signals are incorporated into the latent space through linear transformation:

[0048] ;

[0049] in, For a moment Latent variables, This is the interaction weight matrix. Indicates streaming input. For a moment -1 is a hidden variable. This is for noise. This design keeps the interaction latency stable within 80ms, supporting real-time rendering of continuous operation flows (such as character movement paths).

[0050] S2: A memory consistency constraint mechanism is established based on the Transformer's memory mapping module, as follows:

[0051] A dynamic context memory is introduced to store key scene features, including pixel element positions and lighting patterns, and a consistency loss function is designed:

[0052] ;

[0053] in, Key scene features This is a Transformer-based memory mapping module. Here, t represents the weighting coefficient, t is the t-th frame, and T is the total number of frames. For the key scene features of frame t-1, This is a key scene feature for frame 0. This mechanism can reduce the probability of scene drift during video generation and ensure strict consistency in pixel dimensions such as tree size and character appearance color.

[0054] S3: A pixel-style optimization module is built using an encoder and decoder, as follows:

[0055] To address pixelated style characteristics, embed style constraints in the VAE encoder and / or decoder:

[0056] ;

[0057] in, As a pixel-style extractor, it forces the output video to maintain a low resolution consistency with the input image. This design avoids interference from realistic rendering and greatly improves the recognizability of pixel elements. For input video timing The video frames, This is the input image.

[0058] S4: Adopts an end-to-end diffusion converter architecture (VIT-VAE Encoder → DiT → VIT-VAEDecoder) to reconstruct pixel-style video frames and generate interactive pixel-style game videos in real time based on the pixel-style video frames.

[0059] The method for reconstructing pixelated video frames is as follows:

[0060] The encoder will input the image Encoded as , for Latent feature encoding wherein is a dimensionality space ;

[0061] Interactive input Plus Generate through multiple self-attention , for Encoding of latent features after encoding;

[0062] The decoder will Reconstructed into pixelated video frames .

[0063] Application scenarios of this invention:

[0064] This invention focuses on the real-time generation and interaction of pixel-style game videos, and its application scenarios cover multiple fields such as game development, content creation, live entertainment, education and training, and e-sports analysis. Specifically, it manifests in the following core scenarios:

[0065] (1) Rapid Iteration Scenarios in Game Development

[0066] During game development, developers only need to upload a pixel-style concept image (such as a level sketch) to control character behavior (such as movement, jumping, and interaction) in real time via keyboard or mouse. The system instantly generates video clips containing complete game logic. For example, when designing a "Pixel Farm" game, developers input pixel elements such as trees and houses, and input the "plant" command via keyboard. The system renders the character's planting action in real time and generates a 20fps video with latency controlled within 80ms. Traditional development requires repeated exporting and testing; this invention improves level design efficiency, especially suitable for independent game teams and mobile game development, significantly reducing the cost of repetitive art asset production.

[0067] (2) Game live streaming interactive content generation scenario

[0068] Live streaming platforms can integrate this invention to enable dynamic video generation with real-time audience participation. The streamer uses the current live feed as input, and viewers send commands via chat (such as "move left" or "attack the monster"). The system responds instantly through mouse-based streaming interaction, generating a pixel-style game video stream that matches the commands. For example, in a Stardew Valley-style live stream, viewer commands trigger changes in the character's path, and scene elements (such as farm crops and animals) maintain a consistent pixel style, avoiding the interactive gaps caused by latency in traditional live streams. This scenario increases live stream engagement and viewer dwell time, creating new monetization models for the platform.

[0069] (3) Low-threshold game content creation platform scenario

[0070] This content creation platform is designed for non-professional users. Users can upload any pixel-style image (such as character illustrations or scene images), and the system automatically parses the elements and generates interactive videos. For example, educational users can upload a pixel image of "Dinosaur Park," and by simply dragging and dropping the specified character path with the mouse, the system will output a video in real time containing dinosaur movement and interactive events, all without requiring programming. This invention optimizes computational efficiency through a DiT architecture, enabling video generation speeds of up to 80ms / frame, supporting continuous 10-minute video creation, significantly lowering the barrier to UGC (user-generated content), and driving explosive growth in the pixel-style game content ecosystem.

[0071] (4) Immersive application scenarios in education and training

[0072] In programming education, students upload pixel-art code logic diagrams (such as flowcharts) and input commands (such as "if conditional statements") via the keyboard. The system then generates dynamic demonstration videos that visually showcase the code execution process. For example, when learning "pathfinding algorithms," students input a map image, control the character's path using the keyboard, and the video displays the path in real time. The algorithm's execution trajectory and scene elements (such as obstacles and waypoints) maintain a consistent pixel style, avoiding cognitive confusion caused by stylistic differences in traditional animation. In history teaching, a pixel-style interactive video of an "ancient Roman street" is generated, allowing students to trigger events (such as merchant transactions) by clicking with the mouse. The consistency of the video scene ensures the continuity of historical details and enhances the learning immersion.

[0073] In summary, this invention upgrades game development, live streaming, and content creation scenarios from "static presets" to "dynamic co-creation" through real-time interaction, consistent pixel-style generation, and an efficient DiT architecture. It covers the entire chain of needs from professional development to mass entertainment, injecting real-time and personalized innovative power into the pixel-style game ecosystem.

[0074] Compared with the prior art, the present invention has the following advantages:

[0075] (1) Real-time interactive response: Through keyboard or mouse streaming input, user operations (such as movement and jump commands) can dynamically affect the video generation process in real time, ensuring that the interaction delay is less than 100ms, supporting a smooth "input-generation" closed loop, and completely solving the problems of static generation and interaction gaps in existing models.

[0076] (2) Supports single-image gamified video generation: Users only need to input a pixel-style image (such as a level sketch or character illustration), and the system will automatically parse the elements and generate a continuous video sequence containing complete game logic (such as character behavior and scene events). No additional scripts or complex configurations are required, which greatly reduces the creation threshold.

[0077] (3) Ensure long-term scene consistency: Innovative design of "memory-consistency constraint mechanism" to dynamically store key scene features (such as pixel element position and lighting style), and verify and correct style drift in real time during video generation to ensure that scene elements (such as terrain and environment) of videos longer than 3 minutes remain highly consistent.

[0078] (4) Optimize computational efficiency: Based on the DiT architecture (VIT-VAE encoder → DiT → VIT-VAE decoder), an efficient model is built, which reduces computational complexity and supports a real-time generation speed of 50ms / frame. At the same time, the low-resolution and high-recognition style unique to pixel art is preserved, solving the computational bottleneck caused by storing historical data in traditional models.

[0079] (5) Pixel Art Style Preservation: The style constraint module ensures that the input pixel style is preserved at a rate of 98.7%. When the input is an 8-bit style image (such as a 32×32 pixel image), the 8-bit texture recognition of the output video is improved by 40% (from 58.2% to 98.7%), completely avoiding interference from realistic rendering. In the Minecraft scene, the character color consistency score reaches 4.8 / 5.0 (compared to only 2.1 / 5.0 for traditional models).

[0080] (6) This invention promotes the upgrade of pixel-style games from "static preview" to "dynamic co-creation", providing real-time, efficient and consistent video generation capabilities for development, live streaming, education and other scenarios.

[0081] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1. A method for real-time generation of interactive pixel-style game videos, characterized in that, Includes the following steps: A real-time interactive driving mechanism is established based on the diffusion converter; A memory consistency constraint mechanism is established based on the Transformer memory mapping module; A pixel-style optimization module was built using an encoder and decoder. An end-to-end diffusion converter architecture is adopted to reconstruct pixel-style video frames and generate interactive pixel-style game videos in real time based on the pixel-style video frames.

2. The method for real-time generation of interactive pixel-style game videos as described in claim 1, characterized in that, Establish a real-time interactive driving mechanism as follows: By embedding keyboard and mouse streaming input into the diffusion transformer generation process, millisecond-level response is achieved. The keyboard and mouse streaming input is as follows: ,in, Indicates streaming input. Represents a time frame. Represents the input vector space. This indicates the number of types of keyboard and mouse streaming operations; Interactive signals are incorporated into the latent space through linear transformation: ； in, For a moment Latent variables, This is the interaction weight matrix. Indicates streaming input. For a moment -1 is a hidden variable. This is the noise term.

3. The method for real-time generation of interactive pixel-style game videos as described in claim 2, characterized in that, The memory consistency constraint mechanism is established as follows: A dynamic context memory is introduced to store key scene features, including pixel element positions and lighting patterns, and a consistency loss function is designed: ； in, For the key scene features of frame t, This is a Transformer-based memory mapping module. Here, t represents the weighting coefficient, t is the t-th frame, and T is the total number of frames. For the key scene features of frame t-1, These are the key scene features for frame 0.

4. The method for real-time generation of interactive pixel-style game videos as described in claim 3, characterized in that, The pixel art style optimization module is constructed as follows: Embed style constraints in the encoder and / or decoder: ； in, This is a pixel-style extractor that forces the output video to maintain a low resolution consistency with the input image. For input video timing The video frames, This is the input image.

5. The method for real-time generation of interactive pixel-style game videos as described in claim 4, characterized in that, The end-to-end diffusion converter architecture is: VIT-VAE Encoder → DiT → VIT-VAE Decoder.

6. The method for real-time generation of interactive pixel-style game videos as described in claim 5, characterized in that, An end-to-end diffusion converter architecture is used to reconstruct pixel-art video frames as follows: The encoder will input the image Encoded as , for Latent feature encoding ,in yes Dimensions space ; Interactive input Plus Generate through multi-head self-attention , for Encoding of latent features after encoding; The decoder will Reconstructed into pixelated video frames .

Images