Video generation method and device and electronic equipment

Abstract

The invention provides a video generation method and apparatus, and an electronic device. The method comprises the steps of displaying a generated first video frame on a graphical user interface; the first video frame displays a virtual scene in a preset virtual field of view; the virtual scene comprises at least one scene element; acquiring operation information input by the input equipment in real time; determining a composite interaction instruction based on the operation information and the first video frame, a view updating instruction for the virtual view range, and / or an interaction action instruction for scene elements in the virtual scene; inputting the composite interaction instruction and the first video frame into a pre-trained video generation model, updating the first video frame based on the composite interaction instruction through the video generation model, and generating a plurality of video frames associated with the virtual scene; and displaying an interactive video formed by the plurality of video frames in the graphical user interface. According to the mode, the generation efficiency of the video comprising the interactive elements is improved, and the video generation cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method, apparatus, and electronic device for generating video. Background Technology

[0002] Interactive videos that users can interact with are typically generated by human animation or game engine recording. Both methods are labor-intensive and have low generation efficiency. While artificial intelligence (AI) technology can generate videos, these only support automatic playback and lack interactivity, failing to solve the problems of high cost and low efficiency in generating interactive videos. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a video generation method, apparatus and electronic device to improve the generation efficiency of interactive videos and reduce the generation cost of interactive videos.

[0004] In a first aspect, embodiments of the present invention provide a video generation method, which provides a graphical user interface through a terminal device; the method includes: displaying a generated first video frame in the graphical user interface; the first video frame displays a virtual scene within a preset virtual field of view; the virtual scene includes at least one scene element; acquiring operation information input through an input device in real time, the operation information including at least one of the following: operation position, operation direction, operation duration, and operation method; determining a composite interaction instruction based on the operation information and the first video frame, the composite interaction instruction including: a field of view update instruction for the virtual field of view, and / or, an interaction action instruction for scene elements in the virtual scene; inputting the composite interaction instruction and the first video frame into a pre-trained video generation model, updating the first video frame based on the composite interaction instruction through the video generation model, generating multiple video frames associated with the virtual scene; and displaying an interactive video formed by the multiple video frames in the graphical user interface.

[0005] Secondly, embodiments of the present invention provide a video generation apparatus that provides a graphical user interface via a terminal device; the apparatus includes: a first video frame determination module, used to display a generated first video frame in the graphical user interface; the first video frame displays a virtual scene within a preset virtual field of view; the virtual scene includes at least one scene element; a composite interaction instruction acquisition module, used to acquire operation information input through an input device in real time, the operation information including at least one of the following: operation position, operation direction, operation duration, and operation method; determining composite interaction instructions based on the operation information and the first video frame, the composite interaction instructions including: a field of view update instruction for the virtual field of view, and / or, an interaction action instruction for scene elements in the virtual scene; a video generation module, used to input the composite interaction instructions and the first video frame into a pre-trained video generation model, and update the first video frame based on the composite interaction instructions through the video generation model to generate multiple video frames associated with the virtual scene; and a video display module, used to display the interactive video formed by the multiple video frames in the graphical user interface.

[0006] Thirdly, embodiments of the present invention provide an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the above-described video generation method. Fourthly, embodiments of the present invention provide a machine-readable storage medium storing machine-executable instructions. When the machine-executable instructions are invoked and executed by a processor, the machine-executable instructions cause the processor to implement the aforementioned video generation method.

[0007] The embodiments of the present invention bring the following beneficial effects: The aforementioned video generation method, apparatus, and electronic device display a generated first video frame on a graphical user interface; the first video frame displays a virtual scene within a preset virtual field of view; the virtual scene includes at least one scene element; real-time acquisition of operation information input via an input device, the operation information including at least one of the following: operation position, operation direction, operation duration, and operation method; determination of a composite interaction instruction based on the operation information and the first video frame, the composite interaction instruction including: a field of view update instruction for the virtual field of view, and / or, an interaction action instruction for scene elements in the virtual scene; inputting the composite interaction instruction and the first video frame into a pre-trained video generation model, updating the first video frame based on the composite interaction instruction through the video generation model, generating multiple video frames associated with the virtual scene; and displaying the interactive video formed by the multiple video frames in the graphical user interface. This method improves the generation efficiency of videos including interactive elements and reduces video generation costs.

[0008] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0009] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0010] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a video generation method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the internal structure of an AG-MFM module provided in an embodiment of the present invention; Figure 3 A flowchart illustrating the working process of a video generation model provided in an embodiment of the present invention; Figure 4 A flowchart illustrating the training process of a video generation model and its sub-modules provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a video generation device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] Traditional video production processes, whether involving human animation or game engine recording, are extremely time-consuming and labor-intensive. While artificial intelligence (AI) video generation technology has developed rapidly in recent years, it still suffers from several shortcomings: 1. Most AI video generation models can only passively "play" videos and cannot be interactive in real time like playing a game. Users cannot control the "characters" on the screen to move, turn around, shoot, or pick up items using a keyboard and mouse.

[0014] 2. Some models support simple movement controls but fail to understand the meaning of objects in the scene. For example, they cannot distinguish between shooting an oil drum causing an explosion and shooting a wall leaving a bullet hole. Therefore, worlds generated in this way lack realism and playability.

[0015] 3. To achieve this kind of intelligent interaction, relying on large, general-purpose AI models for real-time analysis would result in huge computational delays, which would completely fail to meet the requirements for smooth interaction.

[0016] Based on this, embodiments of the present invention provide a video generation method, apparatus, and electronic device, which can be applied to interactive video generation scenarios.

[0017] The video generation method disclosed in one embodiment of the present invention can run on a local terminal device or a server. When the video generation method runs on a server, it can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and client devices.

[0018] In an optional implementation, various cloud applications, such as cloud gaming, can run under the cloud interaction system. Taking cloud gaming as an example, cloud gaming refers to a gaming method based on cloud computing. In the cloud gaming operating mode, the game program and the game screen presentation are separated. The storage and execution of video generation are completed on the cloud gaming server. The client device is used for data reception, transmission, and game screen presentation. For example, the client device can be a display device with data transmission capabilities located close to the user, such as a mobile terminal, television, computer, or PDA; however, the information processing is performed by the cloud gaming server in the cloud. When playing the game, the player operates the client device to send operation commands to the cloud gaming server. The cloud gaming server runs the game according to the operation commands, encodes and compresses the game screen and other data, returns it to the client device via the network, and finally, the client device decodes and outputs the game screen.

[0019] In an optional implementation, taking a game as an example, the local terminal device stores the game program and is used to display the game screen. The local terminal device is used to interact with the player through a graphical user interface (GUI), i.e., conventionally by downloading, installing, and running the game program via an electronic device. The local terminal device can provide the GUI to the player in various ways, such as rendering it on the terminal's display screen or providing it to the player via holographic projection. For example, the local terminal device can include a display screen for displaying the GUI, which includes game screens, and a processor for running the game, generating the GUI, and controlling the display of the GUI on the display screen.

[0020] See Figure 1 First, a video generation method provided by an embodiment of the present invention will be introduced. In this method, a graphical user interface is provided through a terminal device. The method includes the following steps: Step S102: Display the generated first video frame in the graphical user interface; the first video frame displays a virtual scene within a preset virtual field of view; the virtual scene includes at least one scene element.

[0021] The aforementioned first video frame can be drawn manually, generated from real-world scenes, extracted from a video, or generated by an artificial intelligence model based on text descriptions or images. For example, it can be generated using Deepseek, Chatgpt, or the video generation model described in this document. Specific settings can be configured according to requirements, and will not be discussed here.

[0022] The virtual scene displayed in the first video frame typically corresponds to a virtual 3D space. To give users the interactive experience of being in a virtual 3D space, the way humans see the world in the real world can be simulated; that is, the portion within the human field of vision is visible, corresponding to the virtual 3D space, and thus displayed on the first video frame. The aforementioned virtual field of vision is used to simulate the actual human field of vision. A virtual camera can also be determined based on the virtual field of vision corresponding to the first video frame. By controlling the movement and rotation of the virtual camera, the virtual field of vision can be updated, causing changes to the displayed content of the video frame.

[0023] The aforementioned virtual field of view is typically determined during the process of defining the first video frame. For example, if the first video frame is manually drawn, the artist usually sets the virtual field of view consciously or unconsciously during the drawing process. When the first video frame is generated by capturing a scene, the virtual field of view is determined by the position, orientation, and shooting parameters of the shooting device. The specific details can be determined based on the actual situation and are not limited here.

[0024] The aforementioned virtual scene includes one or more scene elements. These scene elements can be virtual creatures or virtual non-living things within the virtual scene. Virtual creatures can be virtual characters, animals, etc., while virtual non-living things can be virtual plants, virtual buildings, other virtual items, etc. These scene elements can also be text, patterns, colors, etc., displayed within the virtual scene; the specific settings can be configured according to requirements and are not limited here.

[0025] Step S104: Real-time acquisition of operation information input through the input device. The operation information includes at least one of the following: operation position, operation direction, operation duration, and operation method.

[0026] Users typically generate action information for virtual scenes through input devices connected to a terminal device. The terminal device can be electrically connected to the input device or a communication connection. Input devices can include keyboards, mice, game controllers, touchscreens, wearable interactive devices, etc.

[0027] The aforementioned operation position typically refers to the location of the signal generated by the input device within the graphical user interface (GUI), usually represented by screen coordinates. GUIs typically have four preset directions: up, down, left, and right. The aforementioned operation direction usually refers to the direction corresponding to these preset directions; for example, a swipe touch operation could be from the top of the GUI downwards. The aforementioned operation duration typically refers to the time between the start and end of the operation; for example, a user could press a key on a keyboard for 2 seconds. The aforementioned operation method can be one or more of the following: click, long press, swipe, etc.

[0028] Step S106: Determine a composite interaction instruction based on the operation information and the first video frame. The composite interaction instruction includes: a field of view update instruction for the virtual field of view, and / or, an interaction action instruction for scene elements in the virtual scene.

[0029] The aforementioned composite interaction instructions may include field-view update instructions for updating the virtual field of view, composite interaction instructions for scene elements in the virtual scene, or both.

[0030] As mentioned above, the virtual field of view changes as the virtual camera set for the virtual scene moves and rotates. The aforementioned field of view update command can be a command to control the movement of the virtual camera, a command to control the rotation of the virtual camera, or a command to control both the movement and rotation of the virtual camera.

[0031] To create a sense of interaction between the user and the virtual scene, commands to control the movement of the virtual camera can also evolve into commands to control the movement of the virtual character. In this case, the virtual camera and the virtual character typically have a fixed relative position. Specifically, users can generate commands to control the movement of the virtual character using different keys on a keyboard, the joystick on a game controller, or by touching the virtual joystick controls on a touchscreen.

[0032] There are typically several ways to generate rotation commands for the virtual camera. Users can usually generate rotation commands by holding down the mouse button and moving the mouse, or by using a touchscreen to operate preset view direction controls. The specific methods can be configured according to needs and are not limited here.

[0033] Scene elements in a virtual scene are typically interactive, meaning that their form and state will change after receiving an interaction command. However, some scene elements can be pre-defined as non-interactive. Even if interaction commands are generated for these non-interactive elements, the commands can be directly deemed invalid, or the corresponding interaction action can be determined as a null action.

[0034] Interaction instructions for scene elements can clearly specify the user's desired interaction with that element. For example, multiple trigger controls for different interactions can be pre-defined. A user can select a scene element and then trigger the corresponding trigger control, or trigger a control and then perform a swipe operation from that control to the scene element. In this case, the generated interaction instructions for that scene element contain information about the interaction action corresponding to the triggered control.

[0035] Interaction instructions for scene elements may not include information about a specific interaction action, but only the location information of the scene element. For example, a user can tap or long-press the location of the scene element via a touchscreen, or single-click or double-click the location of the scene element with a mouse. In this case, the interaction instruction only indicates the user's intention to interact with the scene element, but does not specify what kind of interaction action is required.

[0036] During the process of generating video frames using a video generation model, it is necessary to monitor the input signals of the input devices connected to the terminal device in real time to promptly determine the complex interaction commands for the virtual scene. When a complex interaction command is obtained, the video frame currently displayed in the graphical user interface at the time the complex interaction command was generated can be used as the video frame corresponding to the complex interaction command.

[0037] Step S106: Input the composite interaction command and the first video frame into the pre-trained video generation model. The video generation model updates the first video frame based on the composite interaction command to generate multiple video frames associated with the virtual scene.

[0038] If a composite interaction command is received when the first video frame is displayed in the graphical user interface, the composite interaction command and the first video frame can be input into a pre-trained video generation model, which will then generate the next video frame based on the first video frame and the composite interaction command. If no composite interaction command is received when the first video frame is displayed in the graphical user interface, only the first video frame can be input into the video generation model, which will then generate the next video frame based on the first video frame.

[0039] After generating the next video frame, it needs to be displayed on the graphical user interface. At this point, the same operations as for the first video frame are repeated. That is, during the display of the video frame, if a complex interaction command is obtained, the complex interaction command and the video frame are input into the video generation model, which then generates the next video frame; if no complex interaction command is obtained, only the video frame is input into the video generation model, which then generates the next video frame.

[0040] Given input video frames and composite interaction commands, the video generation model needs to determine the difference between the display range of the virtual scene in the next video frame and the display range in the input video frame based on the field of view update command in the composite interaction command. It also needs to determine the scene area to be focused on based on the scene elements corresponding to the interaction action command, and further determine the changes that will occur in the scene area where the scene element is located based on the interaction action indicated in the interaction action command or the "experience" for the scene element. Then, it needs to change the input video frame into the next video frame based on the difference in the display range of the virtual scene and the changes that will occur in the scene area.

[0041] When only video frames are input, the video generation model needs to transform the input video frame into the next video frame based on multi-level features of the input video frame, such as deep semantic features and shallow visual features.

[0042] The way users interact with the video is similar to the process of playing a game. The game can be recorded to obtain a video. Then, the game logs are analyzed to determine the complex interaction commands generated by the user during gameplay. The video generation model is then trained using the game video and its corresponding complex interaction commands. During training, the video generation model generates the next video frame based on the input game video frame and its corresponding complex interaction commands. The loss value of the video generation model is calculated based on the generated video frame and the next video frame after the game video frame. The model parameters are then adjusted based on the loss value to train the video generation model.

[0043] Step S108: Display the interactive video formed by multiple video frames in the graphical user interface.

[0044] Each time the video generation model generates a video frame, it is displayed in the graphical user interface (GUI). Users can input complex interactive commands for this video frame. The video generation model can then generate the next video frame based on this frame and the complex interactive commands, and subsequently display the next video frame in the GUI. The continuous display of video frames in the GUI creates an interactive video.

[0045] In practical implementation, the response frequency of composite interactive commands and the processing speed of the video generation model can be combined to interpolate between two video frames generated by the video generation model, thereby obtaining video frames between the two video frames, making the interactive video displayed by the graphical user interface smoother.

[0046] The aforementioned video generation method involves displaying a generated first video frame on a graphical user interface; the first video frame displays a virtual scene within a preset virtual field of view; the virtual scene includes at least one scene element; real-time acquisition of operation information input via an input device, the operation information including at least one of the following: operation position, operation direction, operation duration, and operation method; determining a composite interaction instruction based on the operation information and the first video frame, the composite interaction instruction including: a field of view update instruction for the virtual field of view, and / or, an interaction action instruction for scene elements in the virtual scene; inputting the composite interaction instruction and the first video frame into a pre-trained video generation model, updating the first video frame based on the composite interaction instruction through the video generation model, generating multiple video frames associated with the virtual scene; and displaying the interactive video formed by the multiple video frames in the graphical user interface. This method improves the generation efficiency of videos including interactive elements and reduces video generation costs.

[0047] The following embodiments provide an implementation for generating a first video frame.

[0048] In practical applications, an initial image can be input into a video generation model, which then generates the first video frame based on the initial image. The initial image can be a hand-drawn sketch, concept map, photograph, video screenshot, etc., and there are no restrictions on its type.

[0049] Scene feature data can be pre-generated for the desired video frames. For example, scene description text data can be generated as scene feature data; interactive scene element data describing the types, shapes, and interaction methods of interactive scene elements can also be used as scene feature data; pre-generated depth maps and normal maps can also be used as scene feature data; or a combination of these can be used simultaneously. After inputting the initial image and scene feature data into the video generation model, the model can generate the first video frame based on the initial image and scene feature data.

[0050] The following embodiments provide an implementation method for updating a first video frame based on composite interactive instructions using a video generation model to generate multiple video frames associated with a virtual scene.

[0051] In practical applications, multiple video frames need to be generated in the following way: (1) Generate the next video frame of the first video frame based on the first video frame and the composite interactive instructions for the first video frame through the video generation model.

[0052] (2) Determine the next video frame as the current video frame.

[0053] (3) Based on the operation information input by the input device and the current video frame when the current video frame is displayed, generate the composite interactive instruction corresponding to the current video frame.

[0054] (4) Generate the next video frame of the current video frame based on the current video frame and the composite interactive instructions for the current video frame through the video generation model.

[0055] (5) Determine whether the preset conditions are met; if not, execute the above step (2); if yes, execute the following step (6).

[0056] (6) Stop.

[0057] The process of generating multiple video frames can be viewed as a loop, where the next video frame generated in the current loop is used as the current video frame in the next loop, thus generating multiple video frames before the preset conditions are met.

[0058] The aforementioned preset conditions can be one or more, such as the number of video frames generated by the video generation model reaching a preset threshold, the playback duration of the interactive video reaching a preset duration threshold, the appearance of a specified scene element in the virtual scene, or the change of the state of a specified scene element in the virtual scene to a specified state. Specific settings can be configured according to requirements and are not limited here.

[0059] In practical applications, a video generation model may include a semantic analysis module, an interactive coding module, and an image generation module connected in sequence; in order to realize the automatic loop of the video generation process, the image generation module is also connected to the semantic analysis module.

[0060] The process of generating the next video frame from the first video frame using a video generation model is similar to that of generating the next video frame from the current video frame using a video generation model. Taking the generation of the next video frame from the current video frame as an example, the specific implementation process is as follows: 1. Based on the current video frame and the composite interaction commands for the current video frame, the semantic analysis module determines the interaction semantic features corresponding to the current video frame. The interaction semantic features are used to indicate the basic attributes and interaction attributes of the target scene elements displayed in the current video frame. The interaction attributes are used to indicate the probability of executing each preset feasible interaction action for the scene elements.

[0061] The aforementioned composite interaction instructions may include interaction behavior instructions. Interaction behavior instructions typically include the target interaction location in the current video frame, usually the user's mouse click location in the current video frame, or the touch point location in the current video frame where the user's touch operation used to generate the interaction behavior instruction is performed.

[0062] After inputting the current video frame and the target interaction position into the semantic analysis model, the model can output the interaction semantic features corresponding to the target scene element displayed at the target interaction position in the current video frame. The interaction semantic features can include the basic attributes of the target scene element, such as its position and category, and can also include interaction attributes; the interaction attributes can indicate the probability of performing each feasible interaction action on the target scene element.

[0063] Feasible interactive actions are usually predetermined. For example, the type of virtual scene displayed in the interactive video can be predetermined, such as a combat scene or a retrieval scene. Interactive actions that would appear in this type of virtual scene can then be identified as feasible interactive actions. For example, feasible interactive actions in a combat scene could include hitting, picking up, throwing, and reloading. A list of feasible interactive actions can be provided in advance, listing the possible interactive actions that might appear in the virtual scene displayed in the interactive video. This list of feasible interactive actions can be obtained through manual statistics or generated through game data analysis; no restriction is placed here.

[0064] In one embodiment, the semantic analysis module described above can be a target recognition model. The target recognition model can be trained in the following manner: (1) Establish the first initial model. The structure of the first initial model can be a lightweight convolutional neural network (CNN, such as MobileNetV3 or a customized ResNet with fewer than 5M parameters), which can be selected according to the requirements.

[0065] (2) Based on the pre-generated first sample set, determine the first training data; the first training data includes training images and labeled data of training images; the labeled data includes the labeled position, labeled category and labeled interaction attributes of the first scene elements displayed in the training images; the labeled position, labeled category and labeled interaction attributes of the first scene elements are output by the pre-defined visual language model for the training images.

[0066] When training the initial model, it can be used as the "student model" and the visual language model as the "teacher model." Through "knowledge distillation," the scene understanding capabilities of a powerful but limited teacher model are pre-compressed and taught to a high-performing and efficient student model. It's worth noting that the distillation target has been expanded from traditional object categories or features to the interactive attributes of objects, which can also be called "affordance."

[0067] The first sample set needs to be generated using a visual language model, and the specific method is as follows: (a) Acquire multiple game scene images; each game scene image includes at least one scene element.

[0068] In practical implementation, a large number of diverse game scene screenshots can be prepared as game scene images, for example, an image dataset containing more than 1 million frames.

[0069] (b) Perform the following operations on each game scene image in multiple game scene images: input the game scene image into the visual language model, guide the visual language model to output the position, category and interaction attributes of the scene elements in the game scene image through preset prompt words; determine the position, category and interaction attributes output by the visual language model as the annotation data of the game scene image.

[0070] The aforementioned visual language model can be, for example, GPT-4V or other visual language models. Guided by prompts, this model outputs information about the scene elements in a game scene image, including their locations, categories, and supported interactions. In practice, for each scene element (usually interactive) in an image, the visual language model outputs the element's category name, bounding box coordinates, and a list of availability options. For example, for an "exploding barrel," the availability list would be ["shoot", "push"], indicating support for both "shoot" and "push" actions. The data output by the visual language model for a specific game scene image can be used as annotation data for that game scene image.

[0071] (c) Generate a first sample set based on multiple game scene images and the labeled data of each game scene image.

[0072] After determining the annotation data for each game scene image, multiple game scene images and their annotation data can be directly used as the data in the first set of samples. Alternatively, multiple game scene images can be filtered; for example, images whose annotation data indicates a number of scene elements greater than or equal to a specified number can be selected, and these selected images and their annotation data can be used as the data in the first set of samples. Specific settings can be configured according to requirements and are not limited here.

[0073] After generating the first sample set, training data needs to be selected from it. This can usually be done in a preset order or randomly. The training data typically includes a game scene image and its labeled data, which can be called the "training image". The labeled data consists of the labeled location, labeled category, and labeled interaction attributes of the visual language model in response to its output.

[0074] (3) Input the training image and the labeled position of the first scene element into the first initial model, and output the first interactive semantic feature corresponding to the training image through the first initial model; the first interactive semantic feature is used to indicate: the position, category and interactive attribute of the first scene element.

[0075] The training objective of the first initial model is as follows: given an image and any location on that image (usually location coordinates), the first initial model can predetermine the category, center point or bounding box, and interaction attributes matching the visual language model annotations for the scene elements located at that location on the image. This is typically represented by an availability distribution vector p. The dimension of vector p is equal to the total number of predefined interaction actions, and the values ​​in the vector represent the applicability probability of each interaction.

[0076] Therefore, the training image and the labeled positions of the first scene elements need to be input into the first initial model, and then the first initial model outputs the first interactive semantic features corresponding to the training image. The first interactive semantic features are usually divided into three parts, which respectively represent the position, category and interactive attributes of the first scene elements.

[0077] (4) Calculate the loss value corresponding to the first initial model based on the annotation location, annotation category, annotation interaction attributes and first interaction semantic features.

[0078] The loss values ​​for the first initial model can be calculated separately for location, category, and interaction attributes. Specifically, cross-entropy loss can be used to calculate the loss value for the category, mean squared error loss can be used to calculate the loss value for the location, and binary cross-entropy loss (BCE Loss) can be used to calculate the loss value for the interaction attributes of multiple tags.

[0079] (5) Update the model parameters of the first initial model based on the loss value.

[0080] (6) Determine whether the model parameters of the first initial model have converged; if not, proceed to step (2); if yes, proceed to step (7).

[0081] If convergence fails, step (2) needs to be performed again to continue training the first initial model. If convergence occurs, the training ends.

[0082] (7) The first initial model after training is determined as the target recognition model.

[0083] After training the object recognition model, in the real-time video generation loop, the semantic analysis module receives coordinate parameters (specifically, the coordinates of the mouse cursor position) from the previous video frame and interactive action commands. It then completes a forward propagation at extremely high speed, outputting the object information (category, location, and interaction attributes) corresponding to that coordinate point, and passes this information to the interaction encoding module. On a typical hardware configuration such as an NVIDIA RTX 4060, the semantic analysis module's inference time is less than 10 milliseconds.

[0084] Based on the above training process, the semantic analysis module can also be called the Affordance-Distilled Lightweight Grounding module (AD-LG). This module is implemented through offline knowledge distillation. First, a large visual language model (VLM) is used as the teacher model to annotate massive amounts of game image data, not only annotating the object category and bounding box, but also its interaction affordance, i.e., the set of interactive actions supported by the object. Then, a lightweight student network is trained so that it can predict the target column, normalized position coordinates, and an affordance distribution vector representing the probabilities of multiple interactive actions in real time based on the input image frame and screen coordinates.

[0085] 2. The interactive encoding module determines the interactive behavior features corresponding to the current video frame based on the composite interactive instructions and interactive semantic features for the current video frame.

[0086] The interactive encoding module can translate fragmented and heterogeneous signals (equivalent to "composite interactive instructions") input by users through various input devices (such as keyboards and mice) into a unified, machine-readable, structured instruction containing a clear intent—the aforementioned interactive behavior characteristics. Compared to related technologies that use simple splicing or independent processing of instructions, the interactive encoding module enables the entire video generation model to more robustly understand complex instructions, such as aiming at object A and firing while simultaneously panning to the left.

[0087] Updates to the field of view in a virtual scene are typically achieved through the movement or rotation of the virtual camera, or both. Correspondingly, the field-of-view update command includes movement and rotation parameters. In practice, input devices include a keyboard and a mouse; movement parameters can be determined based on the key states of specified keys on the keyboard; rotation parameters can be determined based on the displacement of the mouse.

[0088] Terminal devices typically monitor external input devices in real-time at a high frequency (e.g., 60 times / second). For keyboards, the states of keys such as W / A / S / D can be recorded to determine positional changes between the virtual camera and the virtual scene, i.e., movement parameters. For mice, mouse displacement (Δx, Δy) can be recorded to record viewpoint rotation, i.e., rotation parameters. The mouse wheel can be used for zooming or other functions, and left and right clicks are used for discrete actions (such as shooting and interaction). When the terminal device can also connect to a game controller, the signals generated by the controller can be used to acquire joystick states and key events.

[0089] The aforementioned composite interaction commands typically include both view update commands and interaction action commands. When a composite interaction command includes only one of these commands, the other command (which is not included in the command) can be replaced with a preset parameter.

[0090] In one embodiment, the interactive encoding module includes a first encoding unit, a second encoding unit, a third encoding unit, and a connection unit. The first encoding unit, the second encoding unit, and the third encoding unit are each connected to the connection unit. Based on the functionality of the interactive encoding module, this module can also be referred to as a Hierarchically Decoupled Multi-modal Action Encoder (HD-MAE).

[0091] In a specific implementation, the field-of-view update command can be encoded by a first encoding unit to obtain the first feature vector. This first encoding unit, also known as the navigation channel encoder, encodes continuous six-degree-of-freedom (6-DoF) camera motion parameters. This unit inputs the acquired continuous motion parameters (such as movement parameters mapped from keyboard states and rotation quaternion change parameters mapped from mouse displacements) into a multilayer perceptron (MLP) containing several fully connected layers and activation functions (such as ReLU), outputting a fixed-dimensional navigation vector a_cam, which is the aforementioned first feature vector. In a specific embodiment, the dimension of this vector can be set to 32 dimensions.

[0092] In practical implementation, the second encoding unit can encode the interactive action instructions based on a preset set of interactive actions to obtain the second feature vector. This set of interactive actions typically includes multiple predefined discrete interactive actions, such as jump, shoot, interact, and reload. Each interactive action can be assigned a unique integer identifier (also called an "integer ID"). The second encoding unit can also be called discrete action channel encoding.

[0093] When a user triggers an interaction, the resulting interaction command includes an identifier for that interaction. First, the target interaction command is determined based on its identifier, and the target integer identifier corresponding to that interaction command is searched from the set of interaction commands. Then, the target integer identifier is converted into a second feature vector by a second encoding unit. This second encoding unit can include a lookup table structure, such as an nn.Embedding layer. The lookup table structure converts the target integer identifier into a dense high-order vector a_discrete, which is the aforementioned second feature vector. The second feature vector is a dense vector of fixed dimensions, which can be set to 64 dimensions.

[0094] When the composite interactive instruction does not include an interactive action instruction, the preset specified integer identifier can be converted into a second feature vector through the second encoding unit; the specified integer identifier corresponds to an empty action and is the integer identifier of the empty action.

[0095] In practical implementation, the interaction semantic features can be encoded by a third encoding unit to obtain a third feature vector. The third encoding unit is also called the target semantic channel encoding. The input to this unit comes from the semantic analysis module. In one specific embodiment, the interaction semantic features input to the semantic analysis module include a vector indicating the category of the target scene element (referred to as the "target category embedding"), normalized two-dimensional coordinates, and an availability distribution vector P. The normalized two-dimensional coordinates are the normalized coordinates (x, y) of the mouse click position on the screen, with a value range of [0, 1]. The availability distribution vector p is a probability vector, where each dimension corresponds to the confidence level of a possible interaction method. The third encoding unit concatenates these three elements into a single vector and inputs the concatenated vector into an independent multilayer perceptron (MLP). The MLP encodes this vector into a target semantic vector, i.e., the aforementioned third feature vector. The dimension of the third feature vector can be set to 64 dimensions. If no interaction action instruction is generated, the third feature vector is set to a zero vector, or a specific vector representing no target embedding.

[0096] In the specific implementation, the interaction behavior feature vector corresponding to the current video frame can be generated by the connection unit based on the first feature vector, the second feature vector, and the third feature vector. Specifically, the first, second, and third feature vectors can be concatenated, and then the concatenated vector is input into a shallow Transformer encoder layer with a self-attention mechanism. This encoder layer can contain two or more attention heads. This encoding layer can learn the intrinsic relationships between different vectors and output a highly condensed interaction behavior feature vector (also known as "the same action vector a_final"). When the first feature vector has a dimension of 32, the second feature vector has a dimension of 64, and the third feature vector has a dimension of 64, the interaction behavior feature vector is usually projected to 128 dimensions.

[0097] 3. Based on the interactive behavior features and the current video frame, the image generation module generates the next video frame of the current video frame.

[0098] The aforementioned image generation module may include a diffusion model. The diffusion model can generate a noise map based on the input current video frame, and then generate the next video frame by denoising the noise map. To enable interactive behavior features to modulate the image generation process of the diffusion model, interactive behavior feature vectors can be introduced to modulate the intermediate images during the denoising process, so that the next video frame output by the diffusion model conforms to the visual effects such as field-of-view adjustment and scene element interaction indicated by the interactive behavior features.

[0099] However, diffusion models are inefficient for image processing. In practical implementations, latent diffusion models (LDMs) can be used. LDMs perform denoising operations in a low-dimensional latent space, efficiently generating high-resolution images. A latent space is a low-dimensional space used in machine learning to represent data. It refers to a compressed representation of all useful information contained in the data. The latent space is typically lower in dimension than the original data space, making it easier to analyze and understand.

[0100] In one specific embodiment, the image generation module includes a latent representation generation submodule and an image generation submodule. The latent representation generation submodule can also be referred to as an Action-Guided Multi-dimensional Feature Modulation network (AG-MFM), and its internal structure is as follows: Figure 2 As shown, the process begins with the latent representation generation submodule generating a latent representation for the next video frame based on the interaction behavior features and the current video frame. Then, the image generation submodule generates the next video frame based on the latent representation.

[0101] The aforementioned latent representation generation submodule includes a feature mapping unit, an encoder, a feature modulation unit, and a decoder. In a specific embodiment, the latent representation generation submodule typically includes encoders and decoders corresponding to multiple resolution levels, i.e., the encoders and decoders appear in pairs, and a U-Net structure can be adopted. The latent representation generation submodule also includes at least one feature mapping unit corresponding to a specified resolution level, i.e., the number of feature mapping units can be one or more.

[0102] Encoders at multiple resolution levels are connected sequentially, as are decoders at multiple resolution levels. Without a feature modulation unit (IMU), encoders and decoders at the same resolution level are directly connected. Since the encoder is connected to the encoder at the next resolution level and the decoder at the corresponding resolution level, this connection method can be called a skip connection. When an IMU corresponding to a specific resolution level is introduced, the first encoder, IMU, and first decoder at that resolution level are connected sequentially; that is, the IMU is inserted at the skip connection point between the encoder and decoder. Second encoders corresponding to other non-specified resolution levels remain directly connected.

[0103] In one specific embodiment, the current video frame is first mapped to the latent space using a feature mapping unit to obtain the initial latent representation corresponding to the current video frame. The mapping of the latent space can be implemented in various ways, such as Principal Component Analysis (PCA), Deep Neural Networks (DNNs), etc., and is not limited here. Then, the encoder outputs the feature image corresponding to the initial latent representation. The encoder extracts deep features from the input initial latent representation step by step, which is to downsample the image to be processed to obtain the feature image (also called a "visual feature map," denoted as h_visual) corresponding to the initial latent representation.

[0104] Then, the feature image is modulated based on the interaction behavior features by the feature modulation unit to obtain the modulated feature image. The specific structure of the feature modulation unit can be a cross-attention layer. The feature modulation unit calculates the query vector Query(Q) in the cross-attention mechanism based on the interaction behavior features. Specifically, the interaction behavior features can be input to a linear projection layer to map and obtain the query vector.

[0105] The feature modulation unit also calculates the key vector Key(K) and value vector Value(V) in the cross-attention mechanism based on the feature image output by the encoder. Typically, the feature image is flattened into a sequence, which is then input into different linear projection layers to map and obtain Key(K) and Value(V) respectively. In a specific embodiment, the feature dimension d_k of Q, K, and V can be set to 256, and the number of attention heads (Multi-Heads) in the cross-attention module can be set to 8. Further, the feature modulation unit generates a modulation image based on the cross-attention principle, the query vector, the key vector, and the value vector. The modulation image indicates the modulation method corresponding to each pixel in the feature image; the modulation method is to enhance or weaken pixel features.

[0106] Specifically, the calculation can be performed using the following formula: Attention(Q, K, V) = softmax(Q·K^T / sqrt(d_k)) · V.

[0107] A feature vector can be computed. This feature vector has the same dimension as the action vector a_final, but has already incorporated visual context information. This output vector is then processed by an MLP, upsampling it through transposed convolution or bilinear interpolation, and reshaped to obtain a modulation map (also called a "temporal mapping") of the same size as h_visual, which can be represented by the symbol M. Furthermore, the feature map can be modulated using the modulation map to obtain the modulated feature map h_modulated.

[0108] In one specific embodiment, the modulated image can be used as a gating signal, and the gating signal is multiplied pixel by pixel with the feature image to obtain the modulated feature image. A simple multiplicative gating calculation method can be used: h_modulated = M * h_visual.

[0109] Specifically, the pixel values ​​of each pixel in the feature image are normalized to the range of 0 to 1, forming a gating signal map. Then, the gating signal map is multiplied element-wise with the feature image to obtain the adjusted feature image. In the adjusted feature image, regions in the gating signal map close to 1 retain the original pixel features, while regions close to 0 suppress or 'turn off' the original pixel features, thereby achieving dynamic filtering and adjustment of features.

[0110] Alternatively, based on the principle of affine transformation, the modulated image can be element-wise decomposed into a scaling factor M_scale and a bias factor M_bias. Then, for each pixel in the feature image (for convenience, each pixel in the feature image is referred to as the "target pixel"), the scaling factor and bias factor derived from the pixel decomposed from the corresponding pixel in the modulated image are multiplied by the target pixel to obtain the modulated feature image. The pixel in the modulated image corresponding to the target pixel is defined as the pixel whose position in the modulated image is the same as the target pixel's position in the feature image. This can be calculated using the following formula: h_modulated = M_scale * h_visual + M_bias.

[0111] The scaling factor indicates whether the original feature value of each channel and each pixel in the feature image should be enhanced or weakened. For example, a scaling factor greater than 1 will enhance the feature, and vice versa. The bias factor indicates what offset should be added to or subtracted from the original feature value of each channel and each pixel in the feature image.

[0112] After obtaining the modulated feature image, the decoder can generate the latent representation of the next video frame based on the modulated feature image. As mentioned above, the encoder and decoder typically consist of multiple components. The AG-MFM modulated feature image h_modulated replaces the original feature image h_visual and continues to be transmitted in the network. By performing this precise, action-guided modulation at multiple layers, the entire denoising process is directed in a direction consistent with the player's intent. After several denoising steps, a latent representation consistent with the player's intent can be obtained. In a specific embodiment, the number of inference steps K can be set to 4-8 steps.

[0113] The aforementioned image generation sub-model can be implemented using the decoder module of a Variational Autoencoder (VAE). Specifically, the latent representation output from the latent representation generation sub-module can be input into a pre-trained VAE decoder, which decodes it from the low-dimensional latent space into a high-resolution pixel image, becoming the next video frame. This video frame is displayed on the graphical user interface and provided to the user, serving as the current video frame in the next round of video frame generation loops, repeating continuously to form a smooth, interactive video stream. Under typical hardware configurations, the end-to-end latency of the entire loop (from input to output) can be controlled between 100-150 milliseconds, meeting real-time interactive requirements.

[0114] In practical applications, the workflow of video generation models can be achieved through methods such as... Figure 3 The flowchart is shown above for details.

[0115] In one specific embodiment, the video generation model is trained in the following manner: 1. Establish a second initial model.

[0116] To distinguish it from the first initial model described above, the initial model of the video generation model is referred to as the "second initial model." There is no order requirement between the second initial model and the first initial model. The structure of the second initial model is as described above, including a semantic analysis module, an interactive coding module, and an image generation module.

[0117] 2. Based on a pre-set second sample set, train the second initial model to obtain an intermediate model. The second sample set includes multiple first game videos and first composite interaction instructions during the formation process of the first game videos; the first composite interaction instructions include field-of-view update instructions, but do not include interaction action instructions.

[0118] Similarly, the second sample set is only distinguished from the first sample set mentioned above and the third sample set described below, and there is no order requirement. The data in the second sample set can be referred to as initial data. Initial data only includes game videos where the player controls the virtual character's movement and camera rotation, but does not perform discrete interactive actions generated by interactive action commands such as shooting, jumping, or interacting. To distinguish it from subsequent game videos, it is referred to here as the "first game video". The aforementioned first composite interactive command can be obtained by parsing the logs of the first game video. It is usually necessary to determine the first composite interactive command and its corresponding game video frame.

[0119] In one specific embodiment, the intermediate model can be trained in the following way: (1) Determine the first training data from the preset second sample set; the first training data includes training game videos and training compound interactive instructions in the process of forming the training game videos.

[0120] (2) Input the first game video frame of the training compound interaction instruction and the training game video into the second initial model, and generate multiple training video frames based on the training compound interaction instruction and the first game video frame through the second initial model.

[0121] The second initial model can generate the next training video frame based on the first game video frame and its corresponding training compound interaction instruction, then determine the next training video frame as the current training video frame, and determine the training compound interaction instruction corresponding to the current training video frame from the training compound interaction instruction, and continue to generate the next training video frame based on the current training video frame and its corresponding training compound interaction instruction, thereby generating multiple training frame videos.

[0122] It is worth noting that since the training compound interaction instructions do not include interaction action instructions, the second encoding module of the second initial model can input a predefined integer identifier of a "null action." The second encoding module can then convert this integer identifier into a dense vector, which serves as the second feature vector. When no interaction action instructions are included, the semantic analysis module has no input and therefore cannot output interaction semantic features to the third encoding module. The third encoding module can be input with a zero vector or a specific "no-goal" embedding vector.

[0123] (3) Based on multiple training video frames and training game videos, calculate the first loss value of the second initial model, and update the model parameters of the second initial model based on the first loss value.

[0124] When calculating the loss value, the pixel-level mean squared error (MSE) or learned perceptual patch similarity (LPIPS) loss between the training video frames and their corresponding game video frames in the first game video is usually calculated as the first loss value of the second initial model. Then, the model parameters of the second initial model are updated based on the first loss value.

[0125] (4) Determine whether the first loss value has converged; if not, execute step (1); if yes, execute step (5).

[0126] When the first loss value converges to a stable and low value, and the first loss value no longer shows a decrease during multiple consecutive training processes, it is considered that the training based on the second sample set can be terminated, and the model has already acquired basic navigation capabilities.

[0127] (5) The second initial model after training is determined as the intermediate model.

[0128] 3. Based on a pre-set third sample set, the intermediate model is trained to obtain a video generation model; the third sample set includes multiple second game videos and second composite interaction instructions in the formation process of the second game videos; the second composite interaction instructions include field of view update instructions and interaction action instructions.

[0129] The model training process based on the third sample set is similar to that based on the second sample set. The main difference lies in the fact that the second composite interaction command in the formation process of the second game video in the third sample set includes both field-of-view update commands and interaction action commands. The model training process based on the third sample set simultaneously trains the model's navigation function and its ability to understand the user's interaction intent. When determining the model's loss value, it is necessary to consider both the loss caused by the field-of-view update command and the logical consistency of the interaction actions. Specifically, an auxiliary evaluation network or manual evaluation can be used to judge the logical consistency of the interaction actions, for example, whether a corresponding visual effect is produced after shooting.

[0130] The training process based on the third sample set can also be divided into two stages. Correspondingly, the third sample set includes a first subset and a second subset. In the first subset, the number of interactive actions corresponding to the interactive action commands is one, or there are multiple corresponding interactive actions, and the time interval between these multiple interactive actions is greater than a preset first duration threshold. That is, the game videos in the first subset correspond to a single interactive action, or the time interval between the interactive actions is relatively long and not continuous. In the second subset, the interactive action commands correspond to multiple interactive actions, and the time interval between these multiple interactive actions is less than or equal to a preset second duration threshold; the second duration threshold is less than the first duration threshold. In other words, the game videos in the second subset are typically formed based on multiple consecutive interactive actions.

[0131] First, the intermediate model needs to be trained based on the first subset to obtain the trained intermediate model. Then, based on the second subset, the trained intermediate model is trained to obtain the video generation model.

[0132] All or part of the data in the second subset mentioned above can be generated in the following way: (1) During the training process of the intermediate model, the loss value of the intermediate model and the corresponding interactive action are sampled to obtain multiple sampling results.

[0133] Specifically, an adversarial action sampler can be set up based on reinforcement learning (RL) or evolutionary algorithms. This sampler is usually an independent, small agent that can sample the loss value of the intermediate model and the corresponding interaction action during the training of the intermediate model.

[0134] (2) Based on the sampling results, determine the sequence of interactive actions; the loss value corresponding to at least some of the interactive actions in the sequence of interactive actions is greater than or equal to the preset loss value threshold, or the loss value corresponding to at least some of the interactive actions is greater than or equal to the loss value corresponding to other interactive actions in multiple sampling results.

[0135] The "environment" of this sampler is the video generation model being trained, and its "action space" is all possible sequences of action instructions. A "reward" can be set for it, representing the loss the video generation model incurs on that action sequence. Therefore, the goal of this sampler is to learn to generate action sequences that cause the main generation model the "most pain" (maximum loss). This sampler can select multiple interactive actions with relatively high loss thresholds from the sampling results to generate interactive action sequences. When certain interactive actions require connection actions, these connecting actions can also be added to the interactive action sequence.

[0136] (3) Generate the second composite interactive instruction and the second game video corresponding to the interactive action sequence, and add the generated second composite interactive instruction and the second game video to the second subset.

[0137] A second composite interactive command corresponding to the sequence of interactive actions can be generated manually or via script to control the game scene. Recording the game screen during the control process yields the corresponding second game video.

[0138] The data in the sample set generated in the above manner is mainly used in the mid-to-late stages of training. In the initial stage, the video generation model is too weak, and any action may result in high loss, making adversarial training meaningless. Only after the video generation model has mastered the basic capabilities can this data effectively identify the model's "weaknesses" and perform targeted reinforcement training, thereby improving the model's robustness and generalization ability.

[0139] Training is considered complete when the video generation model achieves target levels in various metrics, such as video fidelity, motion tracking accuracy, and interaction logic correctness, on a complete and complex test set, and the loss stabilizes.

[0140] The training process of the entire video generation model and its sub-modules is as follows: Figure 4 The flowchart shown above illustrates this; see the description above for details.

[0141] In one specific embodiment, the specific method for generating interactive videos is illustrated by taking the example of generating a player's action of designing an oil drum: 1. Initialization: Input a concept art of an abandoned gas station twilight scene drawn by an artist into the video generation model. The video generation model initializes the player's perspective to be at the gas station entrance. This concept art becomes the first video frame of the deduction loop. 2. Interactive loop begins: Frame t: The player uses the W / A / S / D keys and the mouse to control the viewpoint, which is located near a gas drum at a gas station. The graphical user interface displays a continuous walking animation generated by the video generation model from the previous rounds. The player's action is to move the mouse, aim the game crosshair (located in the center of the screen) at a red oil drum, and then click the left mouse button to trigger the design interaction. The following events were captured by the motion capture device: (1) No keyboard input (stationary); (2) No mouse movement (viewpoint not moving); (3) Left mouse button click event. The first encoding unit of the interactive encoding module encodes the first feature vector a_cam representing "still"; the second encoding unit converts the integer identifier of "shooting" (assumed to be 2) into the second feature vector a_discrete through the embedding table. At this point, the coordinates of the mouse click (screen center) and the current video frame are sent to the semantic analysis module. The semantic analysis module receives the image and the screen center coordinates. Having undergone offline distillation training, it quickly identifies the object within that coordinate area as an "exploding oil drum," assuming a category ID of 15, and predicts its availability distribution vector p. In this vector, the probability value corresponding to the "shooting" action will be very high (e.g., 0.95), while the probability of actions such as "interact (open)" will be very low. The semantic analysis module outputs the category of "exploding oil drum," the screen center coordinates, and the availability vector p as interactive semantic features. The third encoding unit of the interactive encoding module receives the interactive semantic features from the semantic analysis module and encodes them into a third feature vector a_goal. The connection unit of the interactive encoding module fuses the three vectors a_cam, a_discrete, and a_goal into the final interactive action feature vector a_final. This vector contains rich information: "At the current position and from the current viewpoint, perform the 'shoot' action on the 'exploding oil drum' in the center of the screen." The current video frame is processed by the feature mapping unit of the latent representation generation submodule to obtain its corresponding initial latent representation. This initial latent representation and the interaction action feature vector a_final are then input into the U-Net denoising network. Inside U-Net, the Feature Modulation Unit (IMU) is activated. It uses `a_final` as the query to perform cross-attention computation on the visual features of U-Net. Since `a_final` contains strong "shooting" and "exploding oil drum" signals, the modulated image generated by the IMU highly focuses on the spatial region of the corresponding oil drum in the feature image and applies a feature transformation that tends to "decompose, emit light, and generate flames and smoke." After multiple denoising steps, the latent diffusion model finally outputs a latent representation describing "the oil drum explodes after being hit." The VAE decoder renders the latent representation of this explosion into a high-resolution pixel image, which is the (t+1)th video frame. In the image, the oil drum begins to burst into flames. This frame is shown to the player and becomes the "current video frame" for the next round. In the following frames, even if the player does nothing, because the previous frame was "in the midst of an explosion," the visual generation model will continue to generate a series of images based on this strong visual context: the oil drum explodes, expands, generates a shockwave, and eventually turns into a cloud of black smoke, until the scene returns to calm. Through the above loop, a realistic physical interaction with an oil drum can be experienced in a scene generated from a single image, verifying its fun and visual performance in the game. The entire process requires no intervention from 3D modelers and programmers, making it extremely efficient.

[0142] For the above method embodiments, see Figure 5 The illustrated embodiment of the present invention provides a video generation apparatus. A graphical user interface is provided via a terminal device; the apparatus includes: The first video frame display module 502 is used to display the generated first video frame on the graphical user interface; the first video frame displays a virtual scene within a preset virtual field of view; the virtual scene includes at least one scene element; The operation information acquisition module 504 is used to acquire operation information input through the input device in real time. The operation information includes at least one of the following: operation position, operation direction, operation duration, and operation method. The composite interaction instruction determination module 506 determines composite interaction instructions based on operation information and the first video frame. The composite interaction instructions include: a field of view update instruction for the virtual field of view range, and / or, an interaction action instruction for scene elements in the virtual scene. The video generation module 508 is used to input the composite interaction command and the first video frame into the pre-trained video generation model, and update the first video frame based on the composite interaction command through the video generation model to generate multiple video frames associated with the virtual scene. The video display module 510 is used to display an interactive video formed by multiple video frames in a graphical user interface.

[0143] The aforementioned video generation apparatus displays a generated first video frame on a graphical user interface. The first video frame displays a virtual scene within a preset virtual field of view. The virtual scene includes at least one scene element. It acquires operation information input via an input device in real time, including at least one of the following: operation position, operation direction, operation duration, and operation method. Based on the operation information and the first video frame, it determines a composite interaction instruction, including: a field of view update instruction for the virtual field of view, and / or, an interaction action instruction for scene elements in the virtual scene. The composite interaction instruction and the first video frame are input to a pre-trained video generation model, which updates the first video frame based on the composite interaction instruction to generate multiple video frames associated with the virtual scene. The interactive video formed by the multiple video frames is displayed in the graphical user interface. This method improves the generation efficiency of videos including interactive elements and reduces video generation costs.

[0144] The aforementioned video generation module is further configured to: generate a next video frame of the first video frame based on the first video frame and a composite interactive instruction for the first video frame using a video generation model; determine the next video frame as the current video frame; generate a composite interactive instruction corresponding to the current video frame based on the operation information input by the input device when the current video frame is displayed and the current video frame; generate a next video frame of the current video frame based on the current video frame and the composite interactive instruction for the current video frame using a video generation model; and continue executing the step of determining the next video frame as the current video frame until a preset condition is met.

[0145] The aforementioned video generation model includes a semantic analysis module, an interaction encoding module, and an image generation module connected in sequence. The image generation module is also connected to the semantic analysis module. The video generation module is further used to: determine the interaction semantic features corresponding to the current video frame based on the current video frame and the composite interaction instructions for the current video frame through the semantic analysis module; the interaction semantic features are used to indicate the basic attributes and interaction attributes of the target scene elements displayed in the current video frame; the interaction attributes are used to indicate the probability of performing each preset feasible interaction action for the scene elements; determine the interaction behavior features corresponding to the current video frame based on the composite interaction instructions for the current video frame and the interaction semantic features through the interaction encoding module; and generate the next video frame of the current video frame based on the interaction behavior features and the current video frame through the image generation module.

[0146] The aforementioned semantic analysis module includes a target recognition model; basic attributes include location and category; the aforementioned device also includes a target recognition model training module, used for: establishing a first initial model; determining first training data based on a pre-generated first sample set; the first training data includes training images and labeled data of the training images; the labeled data includes labeled positions, labeled categories, and labeled interaction attributes of first scene elements displayed in the training images; the labeled positions, labeled categories, and labeled interaction attributes of the first scene elements are output by a preset visual language model for the training images; inputting the training images and the labeled positions of the first scene elements into the first initial model, and outputting first interactive semantic features corresponding to the training images through the first initial model; the first interactive semantic features are used to indicate: the position, category, and interaction attributes of the first scene elements; calculating the loss value corresponding to the first initial model based on the labeled position, labeled category, labeled interaction attributes, and the first interactive semantic features; updating the model parameters of the first initial model based on the loss value; continuing to execute the step of determining the first training data based on the preset first sample set until the model parameters of the first initial model converge, and determining the trained first initial model as the target recognition model.

[0147] The aforementioned apparatus further includes a first sample set generation module, configured to: acquire multiple game scene images; each game scene image includes at least one scene element; perform the following operations on each of the multiple game scene images; input the game scene image into a visual language model, and guide the visual language model to output the position, category, and interaction attributes of the scene element in the game scene image through preset prompts; determine the position, category, and interaction attributes output by the visual language model as the annotation data of the game scene image; and generate a first sample set based on the multiple game scene images and the annotation data of each game scene image.

[0148] The aforementioned composite interaction instructions include interaction behavior instructions; the interaction behavior instructions include the target interaction position of the current video frame; the aforementioned video generation module is also used to: output the interaction semantic features corresponding to the target scene elements displayed at the target interaction position of the current video frame through a semantic analysis model.

[0149] The aforementioned composite interactive instructions include field-of-view update instructions and interactive action instructions; the interactive encoding module includes a first encoding unit, a second encoding unit, a third encoding unit, and a connection unit; the first encoding unit, the second encoding unit, and the third encoding unit are respectively connected to the connection unit; the aforementioned video generation module is further configured to: encode the field-of-view update instructions through the first encoding unit to obtain a first feature vector; encode the interactive action instructions based on a preset set of interactive actions through the second encoding unit to obtain a second feature vector; encode the interactive semantic features through the third encoding unit to obtain a third feature vector; and generate the interactive behavior feature vector corresponding to the current video frame through the connection unit based on the first feature vector, the second feature vector, and the third feature vector.

[0150] The aforementioned field-of-view update command includes movement parameters and rotation parameters; the input devices include a keyboard and a mouse; the movement parameters are determined based on the key states of specified keys on the keyboard; and the rotation parameters are determined based on the displacement of the mouse.

[0151] Each interactive action in the aforementioned set of interactive actions has a corresponding integer identifier; the aforementioned video generation module is further used to: determine the target integer identifier corresponding to the target interactive action corresponding to the interactive action instruction; convert the target integer identifier into a second feature vector through the second encoding unit; the second feature vector is a dense vector of fixed dimensions.

[0152] The aforementioned composite interactive instructions do not include interactive action instructions; the aforementioned device further includes: a feature conversion module, used to convert a preset specified integer identifier into a second feature vector through a second encoding unit; the specified integer identifier corresponds to an empty action.

[0153] The aforementioned image generation module includes a latent representation generation submodule and an image generation submodule. The step of generating the next video frame of the current video frame based on the interaction behavior features and the current video frame by the image generation module includes: generating the latent representation corresponding to the next video frame of the current video frame based on the interaction behavior features and the current video frame by the latent representation generation submodule; and generating the next video frame of the current video frame based on the latent representation by the image generation submodule.

[0154] The aforementioned latent representation generation submodule includes a feature mapping unit, an encoder, a feature modulation unit, and a decoder; the aforementioned video generation module is further configured to: map the current video frame to the latent space through the feature mapping unit to obtain the initial latent representation corresponding to the current video frame; output the feature image corresponding to the initial latent representation through the encoder; perform feature modulation on the feature image based on interactive behavior features through the feature modulation unit to obtain the modulated feature image; and generate the latent representation corresponding to the next video frame of the current video frame through the decoder based on the modulated feature image.

[0155] The aforementioned latent representation generation submodule includes encoders and decoders corresponding to multiple resolution levels and at least one feature mapping unit corresponding to a specified resolution level; encoders of multiple resolution levels are connected in sequence; decoders of multiple resolution levels are connected in sequence; a first encoder, feature modulation unit, and first decoder corresponding to a specified resolution level are connected in sequence; and a second encoder corresponding to a non-specified resolution level is directly connected to the first encoder.

[0156] The aforementioned video generation module is also used to: calculate a query vector based on interactive behavior features; calculate a key vector and a value vector based on a feature image; and generate a modulated image based on the cross-attention principle, the query vector, the key vector, and the value vector. The modulated image is used to indicate the modulation method corresponding to each pixel in the feature image. The modulation method is to enhance or weaken pixel features. The modulated image is used to perform feature modulation on the feature image to obtain the modulated feature image.

[0157] The aforementioned video generation module is also used to: use the modulated image as a gating signal, multiply the gating signal with the feature image pixel by pixel to obtain the modulated feature image; or, based on the principle of affine transformation, decompose the modulated image element by element into scaling factors and bias factors; for each target pixel in the feature image, multiply the scaling factors and bias factors obtained by decomposing the pixels corresponding to the target pixel in the modulated image with the target pixel to obtain the modulated feature image.

[0158] The aforementioned device further includes a first video frame determination module, used to: input an initial image into a video generation model, and generate a first video frame based on the initial image through the video generation model.

[0159] The aforementioned first video frame determination module is further configured to: input the initial image and scene feature data into the video generation model, and generate the first video frame based on the initial image and scene feature data through the video generation model; the scene feature data includes one or more of the following: scene description text data, interactive scene element data, depth map and normal map.

[0160] The aforementioned device further includes a video generation model training module, used for: establishing a second initial model; training the second initial model based on a preset second sample set to obtain an intermediate model; the second sample set includes multiple first game videos and a first composite interaction instruction during the formation process of the first game videos; the first composite interaction instruction includes a field-view update instruction but does not include an interaction action instruction; training the intermediate model based on a preset third sample set to obtain a video generation model; the third sample set includes multiple second game videos and a second composite interaction instruction during the formation process of the second game videos; the second composite interaction instruction includes a field-view update instruction and an interaction action instruction.

[0161] The aforementioned video generation model training module is further configured to: determine first training data from a preset second sample set; the first training data includes training game videos and training compound interaction instructions during the formation of the training game videos; input the first game video frame of the training compound interaction instructions and the training game video into a second initial model, and generate multiple training video frames based on the training compound interaction instructions and the first game video frame through the second initial model; calculate a first loss value of the second initial model based on the multiple training video frames and the training game videos, and update the model parameters of the second initial model based on the first loss value; continue to execute the step of determining the first training data from the preset second sample set until the first loss value converges, and determine the trained second initial model as an intermediate model.

[0162] The aforementioned third sample set includes a first subset and a second subset; the number of interactive actions corresponding to the interactive action instructions in the first subset is one, or multiple, and the time interval between multiple interactive actions is greater than a preset first duration threshold; the number of interactive actions corresponding to the interactive action instructions in the second subset is multiple, and the time interval between multiple interactive actions is less than or equal to a preset second duration threshold; the second duration threshold is less than the first duration threshold; the aforementioned video generation model training module is further used to: train the intermediate model based on the first subset to obtain the trained intermediate model; and train the trained intermediate model based on the second subset to obtain the video generation model.

[0163] The aforementioned apparatus further includes a second subset generation model, used for: sampling the loss value of the intermediate model and the corresponding interactive actions during the training process of the intermediate model to obtain multiple sampling results; determining an interactive action sequence based on the sampling results; wherein the loss value corresponding to at least some interactive actions in the interactive action sequence is greater than or equal to a preset loss value threshold, or, the loss value corresponding to at least some interactive actions is greater than or equal to the loss value corresponding to other interactive actions in the multiple sampling results; generating a second composite interactive instruction and a second game video corresponding to the interactive action sequence, and adding the generated second composite interactive instruction and the second game video to the second subset.

[0164] This embodiment also provides an electronic device, including a processor and a memory. The memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the above-described video generation method. Specifically: The system displays a first video frame generated in a graphical user interface; the first video frame displays a virtual scene within a preset virtual field of view; the virtual scene includes at least one scene element; it acquires operation information input via an input device in real time, the operation information including at least one of the following: operation position, operation direction, operation duration, and operation method; it determines a composite interaction instruction based on the operation information and the first video frame, the composite interaction instruction including: a field of view update instruction for the virtual field of view, and / or, an interaction action instruction for scene elements in the virtual scene; it inputs the composite interaction instruction and the first video frame into a pre-trained video generation model, and updates the first video frame based on the composite interaction instruction through the video generation model to generate multiple video frames associated with the virtual scene; and it displays the interactive video formed by the multiple video frames in the graphical user interface.

[0165] The above methods improve the efficiency of video generation, including interactive elements, and reduce video generation costs.

[0166] Optionally, the above-mentioned step of updating the first video frame based on composite interaction instructions using a video generation model to generate multiple video frames associated with the virtual scene includes: generating the next video frame of the first video frame based on the first video frame and composite interaction instructions for the first video frame using a video generation model; determining the next video frame as the current video frame; generating composite interaction instructions corresponding to the current video frame based on the operation information input by the input device when the current video frame is displayed and the current video frame; generating the next video frame of the current video frame based on the current video frame and composite interaction instructions for the current video frame using a video generation model; and continuing to execute the step of determining the next video frame as the current video frame until a preset condition is met.

[0167] Optionally, the aforementioned video generation model includes a semantic analysis module, an interaction encoding module, and an image generation module connected in sequence; the image generation module is also connected to the semantic analysis module; the step of generating the next video frame based on the current video frame and the composite interaction instructions for the current video frame by the video generation model includes: determining the interaction semantic features corresponding to the current video frame by the semantic analysis module based on the current video frame and the composite interaction instructions for the current video frame; the interaction semantic features are used to indicate: the basic attributes and interaction attributes of the target scene elements displayed in the current video frame; the interaction attributes are used to indicate: the probability of performing each preset feasible interaction action for the scene elements; determining the interaction behavior features corresponding to the current video frame by the interaction encoding module based on the composite interaction instructions for the current video frame and the interaction semantic features; and generating the next video frame based on the interaction behavior features and the current video frame by the image generation module.

[0168] Optionally, the semantic analysis module includes a target recognition model; basic attributes include location and category; the target recognition model is trained in the following ways: establishing a first initial model; determining first training data based on a pre-generated first sample set; the first training data includes training images and labeled data of the training images; the labeled data includes the labeled position, labeled category, and labeled interaction attributes of the first scene elements displayed in the training images; the labeled position, labeled category, and labeled interaction attributes of the first scene elements are output by a preset visual language model for the training images; the training images and the labeled positions of the first scene elements are input into the first initial model, and the first initial model outputs the first interactive semantic features corresponding to the training images; the first interactive semantic features are used to indicate: the position, category, and interaction attributes of the first scene elements; calculating the loss value corresponding to the first initial model based on the labeled position, labeled category, labeled interaction attributes, and the first interactive semantic features; updating the model parameters of the first initial model based on the loss value; continuing to execute the step of determining the first training data based on the preset first sample set until the model parameters of the first initial model converge, and determining the trained first initial model as the target recognition model.

[0169] Optionally, the first sample set is generated as follows: multiple game scene images are acquired; each game scene image includes at least one scene element; the following operations are performed on each of the multiple game scene images; the game scene images are input into a visual language model, and the visual language model is guided by preset prompts to output the position, category, and interaction attributes of the scene elements in the game scene images; the position, category, and interaction attributes output by the visual language model are determined as the annotation data of the game scene images; and the first sample set is generated based on the multiple game scene images and the annotation data of each game scene image.

[0170] Optionally, the aforementioned composite interaction instruction includes an interaction behavior instruction; the interaction behavior instruction includes the target interaction position of the current video frame; the step of determining the interaction semantic features corresponding to the current video frame based on the composite interaction instruction and the current video frame by the semantic analysis module includes: outputting the interaction semantic features corresponding to the target scene element displayed at the target interaction position of the current video frame through the semantic analysis model.

[0171] Optionally, the aforementioned composite interaction instructions include a view update instruction and an interaction action instruction; the interaction encoding module includes a first encoding unit, a second encoding unit, a third encoding unit, and a connection unit; the first encoding unit, the second encoding unit, and the third encoding unit are respectively connected to the connection unit; the step of determining the interaction behavior features corresponding to the current video frame based on the composite interaction instructions and interaction semantic features for the current video frame by the interaction encoding module includes: encoding the view update instruction by the first encoding unit to obtain a first feature vector; encoding the interaction action instruction based on a preset set of interaction actions by the second encoding unit to obtain a second feature vector; encoding the interaction semantic features by the third encoding unit to obtain a third feature vector; and generating the interaction behavior feature vector corresponding to the current video frame by the connection unit based on the first feature vector, the second feature vector, and the third feature vector.

[0172] Optionally, the aforementioned field of view update command includes movement parameters and rotation parameters; the input devices include a keyboard and a mouse; the movement parameters are determined based on the key states of specified keys on the keyboard; and the rotation parameters are determined based on the displacement of the mouse.

[0173] Optionally, each interactive action in the aforementioned set of interactive actions has a corresponding integer identifier; the step of encoding the interactive action instruction based on the preset set of interactive actions to obtain the second feature vector includes: determining the target integer identifier corresponding to the target interactive action corresponding to the interactive action instruction; converting the target integer identifier into the second feature vector through the second encoding unit; the second feature vector is a dense vector of fixed dimensions.

[0174] Optionally, the above-mentioned composite interactive instructions do not include interactive action instructions; the above method further includes: converting a preset specified integer identifier into a second feature vector through a second encoding unit; the specified integer identifier corresponds to an empty action.

[0175] Optionally, the image generation module includes a latent representation generation submodule and an image generation submodule; the step of generating the next video frame of the current video frame based on the interaction behavior features and the current video frame by the image generation module includes: generating the latent representation corresponding to the next video frame of the current video frame based on the interaction behavior features and the current video frame by the latent representation generation submodule; and generating the next video frame of the current video frame based on the latent representation by the image generation submodule.

[0176] Optionally, the aforementioned latent representation generation submodule includes a feature mapping unit, an encoder, a feature modulation unit, and a decoder. The step of generating a latent representation corresponding to the next video frame based on interactive behavior features and the current video frame by the latent representation generation submodule includes: mapping the current video frame to the latent space through the feature mapping unit to obtain an initial latent representation corresponding to the current video frame; outputting a feature image corresponding to the initial latent representation through the encoder; performing feature modulation on the feature image based on interactive behavior features through the feature modulation unit to obtain a modulated feature image; and generating a latent representation corresponding to the next video frame based on the modulated feature image through the decoder.

[0177] Optionally, the aforementioned latent representation generation submodule includes an encoder and a decoder corresponding to multiple resolution levels and at least one feature mapping unit corresponding to a specified resolution level; the encoders of multiple resolution levels are connected in sequence; the decoders of multiple resolution levels are connected in sequence; the first encoder, feature modulation unit, and first decoder corresponding to the specified resolution level are connected in sequence; and the second encoder corresponding to a non-specified resolution level is directly connected.

[0178] Optionally, the above steps of performing feature modulation on the feature image based on interaction behavior features to obtain a modulated feature image include: calculating a query vector based on the interaction behavior features; calculating a key vector and a value vector based on the feature image; generating a modulated image based on the cross-attention principle, the query vector, the key vector, and the value vector; wherein, the modulated image is used to indicate the modulation method corresponding to each pixel in the feature image; the modulation method is to enhance pixel features or weaken pixel features; and performing feature modulation on the feature image through the modulated image to obtain a modulated feature image.

[0179] Optionally, the above steps of performing feature modulation on the feature image using the modulated image to obtain the modulated feature image include: using the modulated image as a gate signal, multiplying the gate signal with the feature image pixel by pixel to obtain the modulated feature image; or, based on the principle of affine transformation, decomposing the modulated image element by element into scaling factors and bias factors; for each target pixel in the feature image, multiplying the scaling factors and bias factors obtained by decomposing the pixels corresponding to the target pixel in the modulated image with the target pixel to obtain the modulated feature image.

[0180] Optionally, the first video is generated by: inputting an initial image into a video generation model, and generating a first video frame based on the initial image using the video generation model.

[0181] Optionally, the above-mentioned step of inputting the initial image into a pre-trained video generation model and generating the first video frame of the interactive video based on the initial image by the video generation model includes: inputting the initial image and scene feature data into the video generation model and generating the first video frame based on the initial image and scene feature data by the video generation model; the scene feature data includes one or more of the following: scene description text data, interactive scene element data, depth map and normal map.

[0182] Optionally, the above video generation model is trained in the following manner: establishing a second initial model; training the second initial model based on a preset second sample set to obtain an intermediate model; the second sample set includes multiple first game videos and a first composite interaction instruction in the formation process of the first game videos; the first composite interaction instruction includes a field-view update instruction but does not include an interaction action instruction; training the intermediate model based on a preset third sample set to obtain a video generation model; the third sample set includes multiple second game videos and a second composite interaction instruction in the formation process of the second game videos; the second composite interaction instruction includes a field-view update instruction and an interaction action instruction.

[0183] Optionally, the step of training the second initial model based on a preset second sample set to obtain an intermediate model includes: determining first training data from the preset second sample set; the first training data includes training game videos and training compound interaction instructions during the formation of the training game videos; inputting the training compound interaction instructions and the first game video frame of the training game videos into the second initial model, and generating multiple training video frames based on the training compound interaction instructions and the first game video frames through the second initial model; calculating a first loss value of the second initial model based on the multiple training video frames and the training game videos, and updating the model parameters of the second initial model based on the first loss value; continuing to execute the step of determining the first training data from the preset second sample set until the first loss value converges, and determining the trained second initial model as the intermediate model.

[0184] Optionally, the aforementioned third sample set includes a first subset and a second subset; the number of interactive actions corresponding to the interactive action instructions in the first subset is one, or there are multiple corresponding interactive actions, and the time interval between multiple interactive actions is greater than a preset first duration threshold; the number of interactive actions corresponding to the interactive action instructions in the second subset is multiple, and the time interval between multiple interactive actions is less than or equal to a preset second duration threshold; the second duration threshold is less than the first duration threshold; the step of training the intermediate model based on the preset third sample set to obtain a video generation model includes: training the intermediate model based on the first subset to obtain a trained intermediate model; and training the trained intermediate model based on the second subset to obtain a video generation model.

[0185] Optionally, at least a portion of the second subset is generated in the following manner: during the training of the intermediate model, the loss value of the intermediate model and the corresponding interactive actions are sampled to obtain multiple sampling results; based on the sampling results, an interactive action sequence is determined; the loss value corresponding to at least a portion of the interactive actions in the interactive action sequence is greater than or equal to a preset loss value threshold, or the loss value corresponding to at least a portion of the interactive actions is greater than or equal to the loss value corresponding to other interactive actions in the multiple sampling results; a second composite interactive instruction and a second game video corresponding to the interactive action sequence are generated, and the generated second composite interactive instruction and the second game video are added to the second subset.

[0186] See Figure 6 As shown, the electronic device includes a processor 100 and a memory 101. The memory 101 stores machine-executable instructions that can be executed by the processor 100. The processor 100 executes the machine-executable instructions to implement the above-described video generation method.

[0187] Furthermore, Figure 6 The electronic device shown also includes a bus 102 and a communication interface 103, with the processor 100, the communication interface 103 and the memory 101 connected via the bus 102.

[0188] The memory 101 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 103 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 102 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0189] Processor 100 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 100 or by instructions in software form. Processor 100 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 101, and the processor 100 reads the information from memory 101 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0190] This embodiment also provides a machine-readable storage medium storing machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the above-described video generation method.

[0191] The present invention provides a video generation method, apparatus, and electronic device, including a computer-readable storage medium storing program code. The program code includes instructions that can be used to execute the methods described in the preceding method embodiments. Specifically: The system displays a first video frame generated in a graphical user interface; the first video frame displays a virtual scene within a preset virtual field of view; the virtual scene includes at least one scene element; it acquires operation information input via an input device in real time, the operation information including at least one of the following: operation position, operation direction, operation duration, and operation method; it determines a composite interaction instruction based on the operation information and the first video frame, the composite interaction instruction including: a field of view update instruction for the virtual field of view, and / or, an interaction action instruction for scene elements in the virtual scene; it inputs the composite interaction instruction and the first video frame into a pre-trained video generation model, and updates the first video frame based on the composite interaction instruction through the video generation model to generate multiple video frames associated with the virtual scene; and it displays the interactive video formed by the multiple video frames in the graphical user interface.

[0192] The above methods improve the efficiency of video generation, including interactive elements, and reduce video generation costs.

[0193] Optionally, the above-mentioned step of updating the first video frame based on composite interaction instructions using a video generation model to generate multiple video frames associated with the virtual scene includes: generating the next video frame of the first video frame based on the first video frame and composite interaction instructions for the first video frame using a video generation model; determining the next video frame as the current video frame; generating composite interaction instructions corresponding to the current video frame based on the operation information input by the input device when the current video frame is displayed and the current video frame; generating the next video frame of the current video frame based on the current video frame and composite interaction instructions for the current video frame using a video generation model; and continuing to execute the step of determining the next video frame as the current video frame until a preset condition is met.

[0194] Optionally, the aforementioned video generation model includes a semantic analysis module, an interaction encoding module, and an image generation module connected in sequence; the image generation module is also connected to the semantic analysis module; the step of generating the next video frame based on the current video frame and the composite interaction instructions for the current video frame by the video generation model includes: determining the interaction semantic features corresponding to the current video frame by the semantic analysis module based on the current video frame and the composite interaction instructions for the current video frame; the interaction semantic features are used to indicate: the basic attributes and interaction attributes of the target scene elements displayed in the current video frame; the interaction attributes are used to indicate: the probability of performing each preset feasible interaction action for the scene elements; determining the interaction behavior features corresponding to the current video frame by the interaction encoding module based on the composite interaction instructions for the current video frame and the interaction semantic features; and generating the next video frame based on the interaction behavior features and the current video frame by the image generation module.

[0195] Optionally, the semantic analysis module includes a target recognition model; basic attributes include location and category; the target recognition model is trained in the following ways: establishing a first initial model; determining first training data based on a pre-generated first sample set; the first training data includes training images and labeled data of the training images; the labeled data includes the labeled position, labeled category, and labeled interaction attributes of the first scene elements displayed in the training images; the labeled position, labeled category, and labeled interaction attributes of the first scene elements are output by a preset visual language model for the training images; the training images and the labeled positions of the first scene elements are input into the first initial model, and the first initial model outputs the first interactive semantic features corresponding to the training images; the first interactive semantic features are used to indicate: the position, category, and interaction attributes of the first scene elements; calculating the loss value corresponding to the first initial model based on the labeled position, labeled category, labeled interaction attributes, and the first interactive semantic features; updating the model parameters of the first initial model based on the loss value; continuing to execute the step of determining the first training data based on the preset first sample set until the model parameters of the first initial model converge, and determining the trained first initial model as the target recognition model.

[0196] Optionally, the first sample set is generated as follows: multiple game scene images are acquired; each game scene image includes at least one scene element; the following operations are performed on each of the multiple game scene images; the game scene images are input into a visual language model, and the visual language model is guided by preset prompts to output the position, category, and interaction attributes of the scene elements in the game scene images; the position, category, and interaction attributes output by the visual language model are determined as the annotation data of the game scene images; and the first sample set is generated based on the multiple game scene images and the annotation data of each game scene image.

[0197] Optionally, the aforementioned composite interaction instruction includes an interaction behavior instruction; the interaction behavior instruction includes the target interaction position of the current video frame; the step of determining the interaction semantic features corresponding to the current video frame based on the composite interaction instruction and the current video frame by the semantic analysis module includes: outputting the interaction semantic features corresponding to the target scene element displayed at the target interaction position of the current video frame through the semantic analysis model.

[0198] Optionally, the aforementioned composite interaction instructions include a view update instruction and an interaction action instruction; the interaction encoding module includes a first encoding unit, a second encoding unit, a third encoding unit, and a connection unit; the first encoding unit, the second encoding unit, and the third encoding unit are respectively connected to the connection unit; the step of determining the interaction behavior features corresponding to the current video frame based on the composite interaction instructions and interaction semantic features for the current video frame by the interaction encoding module includes: encoding the view update instruction by the first encoding unit to obtain a first feature vector; encoding the interaction action instruction based on a preset set of interaction actions by the second encoding unit to obtain a second feature vector; encoding the interaction semantic features by the third encoding unit to obtain a third feature vector; and generating the interaction behavior feature vector corresponding to the current video frame by the connection unit based on the first feature vector, the second feature vector, and the third feature vector.

[0199] Optionally, the aforementioned field of view update command includes movement parameters and rotation parameters; the input devices include a keyboard and a mouse; the movement parameters are determined based on the key states of specified keys on the keyboard; and the rotation parameters are determined based on the displacement of the mouse.

[0200] Optionally, each interactive action in the aforementioned set of interactive actions has a corresponding integer identifier; the step of encoding the interactive action instruction based on the preset set of interactive actions to obtain the second feature vector includes: determining the target integer identifier corresponding to the target interactive action corresponding to the interactive action instruction; converting the target integer identifier into the second feature vector through the second encoding unit; the second feature vector is a dense vector of fixed dimensions.

[0201] Optionally, the above-mentioned composite interactive instructions do not include interactive action instructions; the above method further includes: converting a preset specified integer identifier into a second feature vector through a second encoding unit; the specified integer identifier corresponds to an empty action.

[0202] Optionally, the image generation module includes a latent representation generation submodule and an image generation submodule; the step of generating the next video frame of the current video frame based on the interaction behavior features and the current video frame by the image generation module includes: generating the latent representation corresponding to the next video frame of the current video frame based on the interaction behavior features and the current video frame by the latent representation generation submodule; and generating the next video frame of the current video frame based on the latent representation by the image generation submodule.

[0203] Optionally, the aforementioned latent representation generation submodule includes a feature mapping unit, an encoder, a feature modulation unit, and a decoder. The step of generating a latent representation corresponding to the next video frame based on interactive behavior features and the current video frame by the latent representation generation submodule includes: mapping the current video frame to the latent space through the feature mapping unit to obtain an initial latent representation corresponding to the current video frame; outputting a feature image corresponding to the initial latent representation through the encoder; performing feature modulation on the feature image based on interactive behavior features through the feature modulation unit to obtain a modulated feature image; and generating a latent representation corresponding to the next video frame based on the modulated feature image through the decoder.

[0204] Optionally, the aforementioned latent representation generation submodule includes an encoder and a decoder corresponding to multiple resolution levels and at least one feature mapping unit corresponding to a specified resolution level; the encoders of multiple resolution levels are connected in sequence; the decoders of multiple resolution levels are connected in sequence; the first encoder, feature modulation unit, and first decoder corresponding to the specified resolution level are connected in sequence; and the second encoder corresponding to a non-specified resolution level is directly connected.

[0205] Optionally, the above steps of performing feature modulation on the feature image based on interaction behavior features to obtain a modulated feature image include: calculating a query vector based on the interaction behavior features; calculating a key vector and a value vector based on the feature image; generating a modulated image based on the cross-attention principle, the query vector, the key vector, and the value vector; wherein, the modulated image is used to indicate the modulation method corresponding to each pixel in the feature image; the modulation method is to enhance pixel features or weaken pixel features; and performing feature modulation on the feature image through the modulated image to obtain a modulated feature image.

[0206] Optionally, the above steps of performing feature modulation on the feature image using the modulated image to obtain the modulated feature image include: using the modulated image as a gate signal, multiplying the gate signal with the feature image pixel by pixel to obtain the modulated feature image; or, based on the principle of affine transformation, decomposing the modulated image element by element into scaling factors and bias factors; for each target pixel in the feature image, multiplying the scaling factors and bias factors obtained by decomposing the pixels corresponding to the target pixel in the modulated image with the target pixel to obtain the modulated feature image.

[0207] Optionally, the first video is generated by: inputting an initial image into a video generation model, and generating a first video frame based on the initial image using the video generation model.

[0208] Optionally, the above-mentioned step of inputting the initial image into a pre-trained video generation model and generating the first video frame of the interactive video based on the initial image by the video generation model includes: inputting the initial image and scene feature data into the video generation model and generating the first video frame based on the initial image and scene feature data by the video generation model; the scene feature data includes one or more of the following: scene description text data, interactive scene element data, depth map and normal map.

[0209] Optionally, the above video generation model is trained in the following manner: establishing a second initial model; training the second initial model based on a preset second sample set to obtain an intermediate model; the second sample set includes multiple first game videos and a first composite interaction instruction in the formation process of the first game videos; the first composite interaction instruction includes a field-view update instruction but does not include an interaction action instruction; training the intermediate model based on a preset third sample set to obtain a video generation model; the third sample set includes multiple second game videos and a second composite interaction instruction in the formation process of the second game videos; the second composite interaction instruction includes a field-view update instruction and an interaction action instruction.

[0210] Optionally, the step of training the second initial model based on a preset second sample set to obtain an intermediate model includes: determining first training data from the preset second sample set; the first training data includes training game videos and training compound interaction instructions during the formation of the training game videos; inputting the training compound interaction instructions and the first game video frame of the training game videos into the second initial model, and generating multiple training video frames based on the training compound interaction instructions and the first game video frames through the second initial model; calculating a first loss value of the second initial model based on the multiple training video frames and the training game videos, and updating the model parameters of the second initial model based on the first loss value; continuing to execute the step of determining the first training data from the preset second sample set until the first loss value converges, and determining the trained second initial model as the intermediate model.

[0211] Optionally, the aforementioned third sample set includes a first subset and a second subset; the number of interactive actions corresponding to the interactive action instructions in the first subset is one, or there are multiple corresponding interactive actions, and the time interval between multiple interactive actions is greater than a preset first duration threshold; the number of interactive actions corresponding to the interactive action instructions in the second subset is multiple, and the time interval between multiple interactive actions is less than or equal to a preset second duration threshold; the second duration threshold is less than the first duration threshold; the step of training the intermediate model based on the preset third sample set to obtain a video generation model includes: training the intermediate model based on the first subset to obtain a trained intermediate model; and training the trained intermediate model based on the second subset to obtain a video generation model.

[0212] Optionally, at least a portion of the second subset is generated in the following manner: during the training of the intermediate model, the loss value of the intermediate model and the corresponding interactive actions are sampled to obtain multiple sampling results; based on the sampling results, an interactive action sequence is determined; the loss value corresponding to at least a portion of the interactive actions in the interactive action sequence is greater than or equal to a preset loss value threshold, or the loss value corresponding to at least a portion of the interactive actions is greater than or equal to the loss value corresponding to other interactive actions in the multiple sampling results; a second composite interactive instruction and a second game video corresponding to the interactive action sequence are generated, and the generated second composite interactive instruction and the second game video are added to the second subset.

[0213] See Figure 6 As shown, the electronic device includes a processor 100 and a memory. See Figure 6 As shown, the electronic device includes a processor 100 and a memory 101. The memory 101 stores machine-executable instructions that can be executed by the processor 100. The processor 100 executes the machine-executable instructions to implement the above-described video generation method.

[0214] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0215] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0216] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0217] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0218] Finally, it should be noted that the above embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating video, characterized in that, The method includes providing a graphical user interface via a terminal device; the method includes: The generated first video frame is displayed in the graphical user interface; the first video frame displays a virtual scene within a preset virtual field of view; the virtual scene includes at least one scene element; Real-time acquisition of operation information input through an input device, the operation information including at least one of the following: operation position, operation direction, operation duration, and operation method; Based on the operation information and the first video frame, a composite interaction instruction is determined, which includes: a field of view update instruction for the virtual field of view, and / or, an interaction action instruction for scene elements in the virtual scene. The composite interaction command and the first video frame are input into a pre-trained video generation model. The video generation model updates the first video frame based on the composite interaction command to generate multiple video frames associated with the virtual scene. The interactive video formed by the plurality of video frames is displayed in the graphical user interface.

2. The method according to claim 1, characterized in that, The step of updating the first video frame based on the composite interaction command using the video generation model to generate multiple video frames associated with the virtual scene includes: The video generation model generates the next video frame based on the first video frame and the composite interactive instructions for the first video frame. The next video frame is determined as the current video frame; Based on the operation information input by the input device when the current video frame is displayed and the current video frame, a composite interactive instruction corresponding to the current video frame is generated; The video generation model generates the next video frame based on the current video frame and the composite interactive instructions for the current video frame. Continue executing the step of determining the next video frame as the current video frame until the preset conditions are met.

3. The method according to claim 2, characterized in that, The video generation model includes a semantic analysis module, an interactive encoding module, and an image generation module connected in sequence; the image generation module is also connected to the semantic analysis module. The step of generating the next video frame based on the current video frame and a composite interactive command for the current video frame using the video generation model includes: The semantic analysis module determines the interactive semantic features corresponding to the current video frame based on the current video frame and the composite interactive instructions for the current video frame. The interactive semantic features are used to indicate the basic attributes and interactive attributes of the target scene elements displayed in the current video frame. The interactive attributes are used to indicate the probability of performing each preset feasible interactive action for the scene elements. The interactive encoding module determines the interactive behavior features corresponding to the current video frame based on the composite interactive instructions for the current video frame and the interactive semantic features. The image generation module generates the next video frame based on the interactive behavior features and the current video frame.

4. The method according to claim 3, characterized in that, The composite interaction instruction includes an interaction behavior instruction; the interaction behavior instruction includes the target interaction position of the current video frame. The step of determining the interactive semantic features corresponding to the current video frame based on the composite interactive command and the current video frame by the semantic analysis module includes: The semantic analysis model outputs the interactive semantic features corresponding to the target scene elements displayed at the target interactive position in the current video frame.

5. The method according to claim 3, characterized in that, The composite interaction command includes a view update command and an interaction action command; the interaction encoding module includes a first encoding unit, a second encoding unit, a third encoding unit, and a connection unit; the first encoding unit, the second encoding unit, and the third encoding unit are respectively connected to the connection unit; The step of determining the interactive behavior features corresponding to the current video frame through the interactive encoding module based on the composite interactive instructions for the current video frame and the interactive semantic features includes: The first encoding unit encodes the view update instruction to obtain a first feature vector. The second encoding unit encodes the interactive action instruction based on a preset set of interactive actions to obtain a second feature vector; The interaction semantic features are encoded by the third encoding unit to obtain a third feature vector; The connection unit generates an interactive behavior feature vector corresponding to the current video frame based on the first feature vector, the second feature vector, and the third feature vector.

6. The method according to claim 5, characterized in that, The field-of-view update command includes movement parameters and rotation parameters; the input device includes a keyboard and a mouse; the movement parameters are determined based on the key state of a specified key on the keyboard; the rotation parameters are determined based on the displacement of the mouse.

7. The method according to claim 5, characterized in that, Each interactive action in the set of interactive actions has a corresponding integer identifier; The step of encoding the interactive action instructions based on a preset set of interactive actions to obtain a second feature vector includes: Determine the target integer identifier corresponding to the target interactive action corresponding to the interactive action instruction; The target integer identifier is converted into a second feature vector by the second encoding unit; the second feature vector is a dense vector of fixed dimensions.

8. The method according to claim 7, characterized in that, The composite interactive instructions do not include interactive action instructions. The method further includes: The second encoding unit converts a preset specified integer identifier into a second feature vector; the specified integer identifier corresponds to an empty action.

9. The method according to claim 3, characterized in that, The image generation module includes a latent representation generation submodule and an image generation submodule; The step of generating the next video frame of the current video frame based on the interaction behavior features and the current video frame by the image generation module includes: The latent representation generation submodule generates a latent representation for the next video frame based on the interaction behavior features and the current video frame. The image generation submodule generates the next video frame based on the latent representation of the current video frame.

10. The method according to claim 9, characterized in that, The latent representation generation submodule includes a feature mapping unit, an encoder, a feature modulation unit, and a decoder; The step of generating a latent representation for the next video frame based on the interaction behavior features and the current video frame through the latent representation generation submodule includes: The feature mapping unit maps the current video frame to the latent space to obtain the initial latent representation corresponding to the current video frame. The encoder outputs the feature image corresponding to the initial latent representation; The feature modulation unit modulates the feature image based on the interaction behavior features to obtain a modulated feature image. The decoder generates a potential representation of the next video frame corresponding to the current video frame based on the modulated feature image.

11. The method according to claim 10, characterized in that, The latent representation generation submodule includes an encoder and a decoder corresponding to multiple resolution levels and at least one feature mapping unit corresponding to a specified resolution level. Encoders at multiple resolution levels are connected sequentially; decoders at multiple resolution levels are connected sequentially. The first encoder, feature modulation unit, and first decoder corresponding to the specified resolution level are connected in sequence; The second encoder, which corresponds to a resolution level other than the specified resolution level, is directly connected to the second encoder.

12. The method according to claim 10, characterized in that, The step of performing feature modulation on the feature image based on the interaction behavior features to obtain the modulated feature image includes: Calculate the query vector based on the aforementioned interactive behavior characteristics; Based on the feature image, calculate the key vector and value vector; Based on the cross-attention principle, the query vector, the key vector, and the value vector, a modulation image is generated; wherein, the modulation image is used to indicate the modulation method corresponding to each pixel in the feature image; the modulation method is to enhance pixel features or weaken pixel features; The feature image is modulated by the modulated image to obtain the modulated feature image.

13. The method according to claim 12, characterized in that, The step of performing feature modulation on the feature image using the modulated image to obtain the modulated feature image includes: The modulated image is used as a gate signal, and the gate signal is multiplied pixel-by-pixel with the feature image to obtain the modulated feature image; or... Based on the principle of affine transformation, the modulated image is decomposed element by element into scaling factor and bias factor; For each target pixel in the feature image, the modulated feature image is obtained by multiplying the target pixel by the scaling factor obtained by splitting the pixel corresponding to the target pixel in the modulated image and the bias factor.

14. The method according to claim 1, characterized in that, The first video frame is generated in the following way: An initial image is input into the video generation model, and the video generation model generates a first video frame based on the initial image.

15. The method according to claim 14, characterized in that, The step of inputting an initial image into a pre-trained video generation model, and generating the first video frame of an interactive video based on the initial image by the video generation model, includes: An initial image and scene feature data are input into the video generation model, and the video generation model generates a first video frame based on the initial image and the scene feature data. The scene feature data includes one or more of the following: scene description text data, interactive scene element data, depth map and normal map.

16. A video generation apparatus, characterized in that, A graphical user interface is provided via a terminal device; the device includes: A first video frame display module is used to display a generated first video frame on the graphical user interface; the first video frame displays a virtual scene within a preset virtual field of view; the virtual scene includes at least one scene element; The operation information acquisition module is used to acquire operation information input through the input device in real time. The operation information includes at least one of the following: operation position, operation direction, operation duration, and operation mode. A composite interaction instruction determination module is used to determine composite interaction instructions based on the operation information and the first video frame; the composite interaction instructions include: a field of view update instruction for the virtual field of view range, and / or, an interaction action instruction for scene elements in the virtual scene; The video generation module is used to input the composite interaction command and the first video frame into a pre-trained video generation model, and generate multiple video frames associated with the virtual scene based on the composite interaction command and the first video frame through the video generation model. A video display module is used to display an interactive video formed by the plurality of video frames in the graphical user interface.

17. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the video generation method according to any one of claims 1-15.

18. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the video generation method according to any one of claims 1-15.

Images