Autonomous Game Operation AI Agent Method and Device

Abstract

This application provides an autonomous game operation AI agent method and device. The method includes: S10, capturing the game screen in real time after the AI ​​agent takes over the control of the game operation; S11, analyzing the game screen and tasks through a large language model (LLM) to generate tactical intentions; S12, generating operation instructions based on the tactical intentions and game screens through a pre-trained action execution model, and executing the operation instructions. During the execution of the above scheme, the game screen is analyzed to monitor whether the task is completed or failed. When the task is completed or failed, the control of the game operation is returned to the user. This fundamentally solves the pain point of players' "operation not keeping up with awareness". It can adapt to the dynamic changes and uncertainties in the game and execute highly complex and non-linear tasks, providing substantial operational assistance to users in playing games.

Description

Technical Field

[0001] This invention relates to the field of gaming, and in particular to a method and apparatus for autonomous game operation AI agent. Background Technology

[0002] Currently, when players encounter difficulties in games, they can use game assistance systems and automation tools. Existing game assistance systems can "understand" the game and "hear" the player's questions, providing strategy suggestions in voice format. These systems solve the problem of "not knowing what to do." However, they cannot solve the dilemma of "knowing how to do it, but not being able to execute it." For challenges requiring high precision, fast reactions, or complex operation sequences, simple voice guidance cannot provide players with substantial operational assistance, creating a "knowing is easy, doing is hard" gap. Existing automation tools, such as macro programs or in-game macro functions, can only record and replay fixed operation sequences. They lack the ability to perceive the game environment in real time and cannot make intelligent adjustments based on dynamic changes in the battle situation. Once the game scene or enemy behavior deviates slightly from the preset script, these tools are prone to errors and lack true intelligence and adaptability. Summary of the Invention

[0003] Therefore, in order to solve the problem of players getting stuck due to insufficient operation skills, this application provides an autonomous game operation AI agent method and device, which fundamentally solves the pain point of players' "operation not keeping up with awareness", can adapt to the dynamic changes and uncertainties in the game, and perform highly complex and non-linear tasks, providing users with substantial operation assistance in playing games.

[0004] In a first aspect, embodiments of this application provide an autonomous game operation AI agent method, including:

[0005] S10. After the AI ​​agent takes over the control of the game operation, capture the game screen in real time;

[0006] S11. Analyze game visuals and missions using a large language model (LLM) to generate tactical intentions;

[0007] S12. Based on tactical intent and game screen, the pre-trained action execution model generates operation commands and executes them. During the execution of the above scheme, the game screen is analyzed to monitor whether the task is completed or failed. When the task is completed or failed, the control of the game operation is returned to the user.

[0008] Secondly, embodiments of this application also provide an autonomous game operation AI agent device, comprising:

[0009] The capture unit is used to capture game footage in real time after the AI ​​agent takes over control of the game operation;

[0010] The generation unit is used to analyze game visuals and missions using a large language model (LLM) to generate tactical intentions.

[0011] The execution unit is used to generate operation instructions based on tactical intentions and game screens using a pre-trained action execution model, and to execute the operation instructions. During the execution of the above scheme, the game screen is analyzed to monitor whether the task is completed or failed. When the task is detected to be completed or failed, the control of the game operation is returned to the user.

[0012] In summary, the autonomous game operation AI agent method and device provided in this application adopts a multi-model collaborative architecture. After taking over the control of game operation, the Large Language Model (LLM) is responsible for understanding the user's intent and the current game screen, decomposing high-level tasks into specific tactical intents. The Action Execution Model receives the tactical intents and the real-time game screen, generates and executes precise, low-latency control command streams, and automatically returns control to the user after completing the task or failing the operation. The entire solution fundamentally solves the pain point of players' "operation not keeping up with awareness," providing substantial operational assistance rather than just suggestions. Moreover, unlike rigid macros and other automation tools, the entire solution makes decisions based on the understanding of the real-time screen, can adapt to the dynamic changes and uncertainties in the game, and execute highly complex, non-linear tasks. This greatly helps players who have difficulty passing certain game challenges due to physical reasons, reaction speed, or operational skill limitations, enabling them to experience the complete game content without obstacles and improving the inclusivity of the game. Attached Figure Description

[0013] Figure 1 This application provides a flowchart illustrating an embodiment of an AI agent method for autonomous game operation.

[0014] Figure 2 This is a schematic diagram of an embodiment of an autonomous game operation AI agent device provided in this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0016] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0017] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0018] Reference Figure 1 As shown in the figure, this application provides a flowchart of an autonomous game operation AI agent method, which includes:

[0019] S10. After the AI ​​agent takes over the control of the game operation, capture the game screen in real time;

[0020] In this embodiment, it should be noted that during the user's game, it is possible to continuously monitor whether the user has triggered a takeover request signal. If a takeover request signal is detected, the user's physical input signals (such as keyboard, mouse, gamepad, etc.) can be intercepted through the operating system-level API (Application Programming Interface) or driver, and the user's permission to simulate input commands can be obtained. The control of the game operation is then handed over to the AI ​​agent, and the AI ​​agent executes steps S10, S11, and S12.

[0021] S11. Analyze game visuals and missions using a large language model (LLM) to generate tactical intentions;

[0022] In this embodiment, it should be noted that the Large Language Model (LLM) can be either a GPT-4o or a Llama 3 model. The LLM analyzes the game screen and mission to understand the mission objective and formulate advanced tactics. For example, if the mission is "Help me get there," the current game screen and mission are input into the LLM. After analysis, the LLM generates a tactical intent: sprint to dodge the first wave of arrows. After generating the tactical intent, the LLM can send it to a pre-trained action execution model.

[0023] S12. Based on tactical intent and game screen, the pre-trained action execution model generates operation commands and executes them. During the execution of the above scheme, the game screen is analyzed to monitor whether the task is completed or failed. When the task is completed or failed, the control of the game operation is returned to the user.

[0024] In this embodiment, it should be noted that after receiving the tactical intent, the action execution model will generate and output specific, low-level, and time-sequential control command streams based on the real-time game screen (such as the command to move the mouse 350 pixels to the right, the command to press the left mouse button, etc.), and inject control commands into the game. Then, it enters the next loop, executing steps S10, S11, and S12. During this process, the Large Language Model (LLM) can continuously analyze the game screen, judging whether the task is completed (e.g., the boss's health bar disappearing, or the appearance of specific visual features when the character reaches the target area indicating task completion) or failed (e.g., the appearance of a "YOU DIED" message indicating player character death indicating task failure). Once task completion or failure is detected, control over the physical input signals is immediately released, seamlessly returning game control to the user. This prevents the AI ​​agent from getting bogged down in ineffective, infinite attempts, and ensures that the user always has ultimate control and a good experience. Monitoring task completion or failure can be performed after step S11 or S12. Additionally, it should be noted that the game can be a game running on a terminal or a cloud game running on a cloud server. If the game is running on a terminal, then executing the operation instructions in step S12 refers to executing operation instructions on the game on the terminal to control the game; while if the game is running on a cloud server, then executing the operation instructions in step S12 refers to sending the operation instructions to the cloud server and controlling the game on the cloud server based on the operation instructions. For example, the operation instructions can be sent directly to the operating system of the cloud server, or the operation instructions can be converted into corresponding operation instructions on the cloud server side and then sent to the operating system of the cloud server.

[0025] The autonomous game operation AI agent method provided in this application adopts a multi-model collaborative architecture. After taking over the control of game operation, the Large Language Model (LLM) is responsible for understanding the user's intent and the current game screen, decomposing high-level tasks into specific tactical intents. The Action Execution Model receives the tactical intents and the real-time game screen, generates and executes precise, low-latency control command streams, and automatically returns control to the user after completing the task or failing the operation. The entire solution fundamentally solves the pain point of players' "operation not keeping up with awareness", providing substantial operational assistance rather than just suggestions. Moreover, unlike rigid macros and other automation tools, the entire solution makes decisions based on the understanding of the real-time screen, can adapt to the dynamic changes and uncertainties in the game, and execute highly complex, non-linear tasks. This greatly helps players who have difficulty passing certain game challenges due to physical reasons, reaction speed, or operational skill limitations, enabling them to experience the complete game content without obstacles and improving the inclusivity of the game.

[0026] Based on the aforementioned method embodiments, before step S12, the following may also be included:

[0027] Acquire game video streams, as well as expert operation commands synchronized with the game video streams (such as keyboard key events, mouse movement trajectories and click data, gamepad joystick coordinates and button status data, etc.).

[0028] The action execution model is trained by using game video streams as input and expert operation commands as output.

[0029] In this embodiment, it should be noted that the action execution model can be a Transformer model, or it can employ a Transformer architecture that includes a visual encoder and a recurrent neural network (using the output of the Transformer-based visual encoder as the input of the sequence decoder based on the recurrent neural network RNN). When training the action execution model, the collected dataset can be used for training, with game screen frame sequences as input and corresponding expert operation commands as output, to learn a direct mapping from what is seen to what is done.

[0030] Based on the aforementioned method embodiments, the triggering conditions for the AI ​​agent to take over the control of game operations may include the user pressing a specific button, and the task can be obtained by analyzing the user's voice commands captured by the microphone.

[0031] In this embodiment, it should be noted that the specific key press can be a combination key, such as Ctrl + Shift + G. The task can be the conversion of the user's voice command into text. For example, if the user's voice command is "Help me defeat him," then the task is: Help me defeat him. Besides controlling specific keys to trigger the AI ​​agent to take over game control, the control of specific keys can also be combined with the user's voice input to trigger the AI ​​agent to take over game control. For example, when the user presses Ctrl + Shift + G and voice inputs "Help me defeat him," a takeover request signal is triggered, transferring game control to the AI ​​agent. By using simple key presses, or simple key presses combined with voice input to trigger the AI ​​agent to take over game control, the intervention of the AI ​​agent is explicitly authorized and controlled by the user. The process of taking over control is smooth and intuitive, minimizing disruption to the game's immersion.

[0032] Based on the aforementioned method embodiments, the step of returning control of game operations to the user may further include:

[0033] The results of the task are provided via voice feedback.

[0034] In this embodiment, it should be noted that when handing over control, text-to-speech technology can be used to generate voice feedback to clearly report the result of the task execution to the user, such as "Task completed, control has been handed over" or "Sorry, I failed, please take over".

[0035] The following example illustrates the solution of an embodiment of the present invention.

[0036] First, several game experts were invited to repeatedly challenge the high-difficulty boss "Forge Knight" in a certain game. Using a configured data acquisition tool, their first-person game screen and the corresponding operation commands for each precise keyboard, mouse, and gamepad operation (e.g., pressing Shift key -> pressing W key -> clicking the right mouse button -> releasing Shift key) were collected. The large amount of collected data (game screen video frame sequence, operation command sequence) was used to train an action execution model. The goal of the model learning is to predict the most likely operation command that the experts will input in the next frame when given a continuous game screen (e.g., 24 frames of the past second).

[0037] After the action execution model was trained, a user, frustrated after failing to defeat the boss "Forge Knight" multiple times while playing the game, pressed the hotkey Ctrl+Shift+G and said into the microphone, "Help me defeat him!" This triggered a takeover request signal, transferring control of the game to an AI agent. The AI ​​agent then began intercepting input from the user's physical input devices such as the keyboard and mouse, while simultaneously converting the audio into the text "Help me defeat him!".

[0038] Next, the Large Language Model (LLM) analyzes the current game screen (which contains the text "Forge Knight") and the text to understand the mission objective and begin formulating advanced tactics. It first generates the initial tactical intent: "Maintain distance, observe its attack patterns, and primarily dodge." Based on the mission objective, it determines whether the mission is completed or failed. If the mission is neither completed nor failed, the tactical intent is sent to the Action Execution Model (ALM). The ALM then generates specific control commands based on the real-time game screen (such as the command to press the roll button backward or to the side) and injects them into the game. This process continues in a loop.

[0039] When the Large Language Model (LLM) recognizes from the game screen that "Forge Knight" has performed an attack with a long recovery time (such as stomping the ground), it will immediately update the tactical intent in the next loop to "find an attack window and launch a quick counterattack" and send the tactical intent to the action execution model. Upon receiving the new intent, the action execution model will immediately adjust its output.

[0040] This loop of LLM planning -> action execution model execution -> LLM observation and replanning will continue until the LLM detects that the "Forge Knight's" health bar has reached zero, or the player character's death screen appears. The final result falls into two categories: Scenario 1 (Mission Success): The "Forge Knight" is defeated, the AI ​​agent immediately releases control of the physical input device, and plays the voice message: "Mission complete, control returned," and the user regains control; Scenario 2 (Mission Failure): The player character dies, the AI ​​agent also releases control, and plays the voice message: "Sorry, I failed, please take over," and the user regains control and can perform other operations.

[0041] Reference Figure 2 The diagram shown is a structural schematic of an autonomous game operation AI agent device provided in an embodiment of this application. The device includes:

[0042] The capture unit 20 is used to capture game footage in real time after the AI ​​agent takes over control of the game operation.

[0043] Generation unit 21 is used to analyze game visuals and tasks through a large language model (LLM) to generate tactical intentions;

[0044] The execution unit 22 is used to generate operation instructions based on tactical intentions and game screens using a pre-trained action execution model, and to execute the operation instructions. During the execution of the above scheme, the game screen is analyzed to monitor whether the task is completed or failed. When the task is detected to be completed or failed, the control of the game operation is returned to the user.

[0045] The autonomous game operation AI agent device provided in this application adopts a multi-model collaborative architecture. After taking over the control of game operation, the Large Language Model (LLM) is responsible for understanding the user's intent and the current game screen, decomposing high-level tasks into specific tactical intents. The Action Execution Model receives the tactical intents and the real-time game screen, generates and executes precise, low-latency control command streams, and automatically returns control to the user after completing the task or failing the operation. The entire solution fundamentally solves the pain point of players' "operation not keeping up with awareness", providing substantial operational assistance rather than just suggestions. Moreover, unlike rigid macros and other automation tools, the entire solution makes decisions based on the understanding of the real-time screen, can adapt to the dynamic changes and uncertainties in the game, and execute highly complex, non-linear tasks. This greatly helps players who have difficulty passing certain game challenges due to physical reasons, reaction speed, or operational skill limitations, allowing them to experience the complete game content without obstacles and improving the inclusivity of the game.

[0046] Based on the aforementioned device embodiments, it may further include:

[0047] The training unit is used to acquire game video streams and expert operation instructions synchronized with the game video streams before the execution unit operates, and to train the action execution model with the game video streams as input and the expert operation instructions as output.

[0048] Based on the aforementioned device embodiments, the triggering conditions for the AI ​​agent to take over the control of game operations may include the user pressing a specific button, and the task can be obtained by analyzing the user's voice commands captured by the microphone.

[0049] Based on the aforementioned device embodiments, the step of returning control of game operations to the user may further include:

[0050] The results of the task are provided via voice feedback.

[0051] The autonomous game operation AI agent device provided in this application embodiment is implemented in the same way as the autonomous game operation AI agent method provided in this application embodiment, and the effect it can achieve is also the same as the autonomous game operation AI agent method provided in this application embodiment, so it will not be described again here.

[0052] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for autonomous game operation AI agent, characterized in that, include: S10. After the AI ​​agent takes over the control of the game operation, capture the game screen in real time; S11. Analyze game visuals and missions using a large language model (LLM) to generate tactical intentions; S12. Based on tactical intent and game screen, the pre-trained action execution model generates operation commands and executes them. During the execution of the above scheme, the game screen is analyzed to monitor whether the task is completed or failed. When the task is completed or failed, the control of the game operation is returned to the user.

2. The method as described in claim 1, characterized in that, Before step S12, the following is also included: Capture game video streams, as well as expert operation commands synchronized with the game video streams; The action execution model is trained by using game video streams as input and expert operation commands as output.

3. The method as described in claim 1 or 2, characterized in that, The triggering conditions for the AI ​​agent to take over control of the game include the user pressing a specific key, and the task is obtained by analyzing the user's voice commands captured by the microphone.

4. The method as described in claim 1, characterized in that, The process of returning control of the game to the user also includes: The results of the task are provided via voice feedback.

5. An autonomous game operation AI agent device, characterized in that, include: The capture unit is used to capture game footage in real time after the AI ​​agent takes over control of the game operation; The generation unit is used to analyze game visuals and missions using a large language model (LLM) to generate tactical intentions. The execution unit is used to generate operation instructions based on tactical intentions and game screens using a pre-trained action execution model, and to execute the operation instructions. During the execution of the above scheme, the game screen is analyzed to monitor whether the task is completed or failed. When the task is detected to be completed or failed, the control of the game operation is returned to the user.

6. The apparatus as claimed in claim 5, characterized in that, Also includes: The training unit is used to acquire game video streams and expert operation instructions synchronized with the game video streams before the execution unit operates, and to train the action execution model with the game video streams as input and the expert operation instructions as output.

7. The apparatus as described in claim 5 or 6, characterized in that, The triggering conditions for the AI ​​agent to take over control of the game include the user pressing a specific key, and the task is obtained by analyzing the user's voice commands captured by the microphone.

8. The apparatus as claimed in claim 5, characterized in that, The process of returning control of the game to the user also includes: The results of the task are provided via voice feedback.

Images