A method and system for autonomous operation of a game GUI

Abstract

This invention discloses a method and system for autonomous operation of a game GUI, primarily relating to the fields of artificial intelligence and human-computer interaction. It includes: acquiring game guides and video knowledge to construct a knowledge base; visually recognizing interface elements and game states in the game screen; integrating the knowledge base, interface element, and game state information to generate an operation decision sequence through multi-agent collaboration; converting the decision sequence into human-like control commands for execution; and optimizing the decision strategy based on feedback. The beneficial effects of this invention are: it achieves cross-game autonomous learning and adaptation without manual scripting; it ensures the universality and concealment of operations through a purely visual approach; and it improves the intelligence, human-likeness, and overall efficiency of complex task execution through multi-agent collaboration and reinforcement learning optimization, making it particularly suitable for scenarios such as automated testing.

Description

Technical Field

[0001] This invention relates to the field of technology, specifically to a method and system for autonomous operation of a game GUI based on pure vision and multi-agent collaboration. Background Technology

[0002] Currently, automated operations for game graphical user interfaces (GUIs) mainly rely on the following technical solutions, but all of them have significant limitations: Firstly, tools based on fixed scripts or preset operation sequences perform specific tasks by manually writing rules or recording operations. Their core flaw lies in their extreme lack of flexibility and adaptability. Once the game version is updated, causing changes in the interface layout, element positions, or rules, the pre-written scripts will not work properly and must be manually re-debugged or rewritten, resulting in high maintenance costs. Furthermore, they are completely incapable of handling new games or complex, dynamically changing interfaces (such as multi-layered tactical panels). Secondly, tools that rely on the game's internal application programming interface (API) or directly read and modify game memory data, while able to accurately obtain the game state, have fatal weaknesses. First, their implementation is tightly coupled with a specific game engine or program structure, resulting in extremely poor cross-game compatibility. Second, such behavior that directly intrudes into the game process is easily detected and blocked by the game's security and anti-cheat system. More importantly, this method is not based on an understanding of the graphical interface and cannot achieve truly universal GUI operations. Third, based on preliminary explorations of computer vision and single-agent reinforcement learning, in recent years, some studies have attempted to use screen images as input to make decisions through a single agent. However, when faced with game scenarios that require long trajectory planning, multi-step reasoning, and complex task division, the single-agent architecture is often insufficient. It is difficult to effectively handle the coordination of multiple links such as knowledge acquisition, interface parsing, state perception, strategy planning, and human-like execution, resulting in low decision-making efficiency. Moreover, the generated mechanical operation sequences lack the coherence and rationality of human players and are easily identified as non-human operations.

[0003] Therefore, there is an urgent need for a game GUI autonomous operation method and system based on pure vision multi-agent collaboration to solve the above problems. Summary of the Invention

[0004] The purpose of this invention is to provide a game GUI autonomous operation method and system based on pure vision multi-agent collaboration. It realizes cross-game autonomous learning and adaptation without the need for manual script writing, improves the intelligence, anthropomorphism and overall efficiency of complex task execution, and is especially suitable for scenarios such as automated testing.

[0005] To achieve the above objectives, the present invention employs the following technical solution: On one hand, the present invention provides a method for autonomous operation of a game GUI, comprising the following steps: Step S1: Obtain knowledge information related to the target game, and construct a knowledge base based on the obtained knowledge information; the knowledge information includes: operation rules extracted from game strategy texts and operation sequences parsed from game videos; Step S2: Perform GUI parsing on the real-time captured game footage to identify the interface elements in the game footage; Step S3: Perform visual perception on the real-time captured game screen and extract game state information from the game screen; Step S4: Based on the knowledge base, identified interface elements, and extracted game state information, generate an operation decision sequence for the game; Step S5: Transform the operation decision sequence into control commands that simulate human operation and execute them. At the same time, optimize the strategy for generating the operation decision sequence based on the game feedback information after execution.

[0006] Preferably, step S1 includes: Step S11: Crawl strategy texts and game videos related to the target game from internet resources; Step S12: Use a text processing model to extract semantic information from the strategy text to obtain text strategy information. ,in Indicates the first A strategy for extracting data from text; Step S13: Analyze the game video using a visual model, mapping the video frame sequence to an action sequence to obtain video strategy information. ,in Indicates the first The analyzed action; Step S14: Integrate text policy information and video policy information to construct a structured knowledge base. The knowledge base Each policy in the table is represented as a triple. ,in This indicates a description of the game state. Indicates a suggested action. This indicates the confidence level or priority weight of the strategy.

[0007] Preferably, in step S14, the integration process includes: deduplicating and standardizing duplicate or conflicting policy information; mapping text rules and video actions into a unified structured format; and assigning confidence to each policy. .

[0008] Preferably, step S2 specifically involves: using the first visual model The game footage captured in real time Process the data and output the identified set of GUI elements. ,in Indicates time, The parameters of the first visual model are defined as follows: the GUI elements include one or more of the following: skill buttons, health bars, skill cooldown icons, item inventory, and minimap.

[0009] Preferably, step S3 specifically involves: using a second visual model The game footage captured in real time Process the data and output a set of extracted game state information. ,in Indicates time, The parameters of the second visual model include one or more of the following: enemy unit position, friendly unit position, neutral unit position, own level, and skill availability.

[0010] Preferably, step S4 specifically includes: Step S41: Set the current time Identified UI elements Extracted game status information and the first N historical action sequences Combined into the current decision state ; Step S42: Based on the knowledge base and current decision-making status Through strategy model Generate action decision for the current moment ,Right now ,in These are the parameters of the strategy model; Step S43: Repeat steps S41-S42 to generate a continuous sequence of operation decisions. .

[0011] Preferably, step S4 further includes applying reinforcement learning to the policy model. Optimization steps: Step S44: Define the reward function The reward function is a weighted sum of rewards from multiple dimensions, including at least: action efficiency reward. Risk control rewards Collaboration performance rewards and rewards for handling uncertainty ; Step S45: Set the optimization objective to maximize the expected cumulative reward. ,in This represents a trajectory consisting of states and actions; Step S46: Update the parameters of the policy model using the policy gradient method. , represented as: ,in This represents the estimated cumulative reward over the past N steps; Step S47: During the iterative optimization process, dynamically adjust the weight of each dimension's reward in the total reward function according to the improvement magnitude of each dimension's reward.

[0012] Preferably, step S5 includes: Step S51: Through the action mapping function Action decisions in the operation decision sequence Transform into specific equipment control commands ,Right now The control commands include simulated mouse clicks, keyboard keys, touchscreen touches, or gamepad input; Step S52: Issue the control command Apply delay or random perturbation And take into account the control commands from the previous moment. Through coherent processing functions Generate anthropomorphic final control commands Execute after; Step S53: After executing the final control command, obtain the new game screen as feedback, extract the new game state information, calculate the reward value, and use the reward value to return to step S4 to optimize the strategy for generating the operation decision sequence.

[0013] On the other hand, the present invention also provides a game GUI autonomous operating system for implementing the above-described game GUI autonomous operation method, comprising: The knowledge acquisition module is used to acquire knowledge information related to the target game and build a knowledge base based on the acquired knowledge information; The GUI parsing module is used to parse the game footage captured in real time and identify the interface elements within it; The visual perception module is used to perceive the real-time captured game screen and extract the game state information therein; The strategy planning module is used to generate an operation decision sequence for the game based on the knowledge base, the identified interface elements, and the extracted game state information. The execution and optimization module is used to convert the operation decision sequence into control instructions that simulate human operation and execute them, and at the same time optimize the strategy of the strategy planning module based on the game feedback information after execution. The knowledge acquisition module, GUI parsing module, visual perception module, strategy planning module, and execution and optimization module communicate and collaborate through shared memory or message passing.

[0014] Preferably, the knowledge acquisition module includes: The crawler unit is used to crawl strategy texts and game videos related to the target game from Internet resources; The text processing unit is used to extract semantic information from the strategy text using a text processing model to obtain text strategy information. The video parsing unit is used to parse the game video using a visual model to obtain video strategy information; The knowledge base construction unit is used to integrate the text policy information and video policy information to construct a structured knowledge base; The strategy planning module integrates a strategy model, which is a model based on reinforcement learning training. Its optimization objective is to maximize the cumulative reward across multiple dimensions, taking into account action efficiency, risk control, collaborative performance, and uncertainty handling.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Excellent cross-game adaptability and portability: This invention automatically acquires and parses game guides and videos from the Internet to build a transferable knowledge base, enabling the system to autonomously learn the rules and strategies of new games, thereby getting rid of dependence on fixed scripts and being able to quickly adapt to different games and version updates.

[0016] 2. Achieves highly compatible and highly covert universal operation: This invention adopts a pure visual solution for GUI element recognition and game state perception, which does not rely on internal game APIs or memory data at all. This fundamentally avoids the risk of being detected by security mechanisms due to intrusion into the game process, and has true cross-platform and cross-game operation potential.

[0017] 3. Improved intelligence and collaboration in handling complex tasks: By constructing a multi-agent collaborative architecture encompassing knowledge acquisition, GUI parsing, visual perception, strategy planning and execution, this invention achieves effective task decomposition and collaboration, enabling the handling of complex game scenarios requiring long-sequence planning and multi-step reasoning, significantly improving decision-making efficiency and task completion reliability.

[0018] 4. Enhanced human-likeness and naturalness of operation: This invention incorporates modeling of human operating habits into decision generation and introduces random delay and action continuity processing at the execution layer, making the automated operation process closer to the behavior pattern of real players, effectively reducing the possibility of being judged as "machine operation" by the game system, and ensuring the stability and naturalness of the operation.

[0019] 5. A complete self-learning and adaptive closed loop is formed: This invention uses environmental feedback (such as game wins and losses, state changes) after policy execution to drive reinforcement learning to optimize the policy, forming a complete autonomous loop from perception, decision-making, execution to feedback optimization, enabling the system to continuously evolve and improve its operational performance during use. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0021] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined in this application.

[0022] In this invention, terms such as "upper," "lower," "left," "right," "front," "back," "vertical," "horizontal," "side," and "bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only to facilitate the description of the structural relationships of the various components or elements of this invention and do not specifically refer to any component or element in this invention. They should not be construed as limiting the invention.

[0023] Example: like Figure 1 As shown, this embodiment provides a method for autonomous operation of a game GUI, including the following steps: Step S1: Obtain knowledge information related to the target game and build a knowledge base based on the obtained knowledge information; wherein, the knowledge information includes: operation rules extracted from game strategy texts and operation sequences parsed from game videos; Step S2: Perform GUI parsing on the real-time captured game footage to identify interface elements in the game footage; Step S3: Perform visual perception on the real-time captured game screen and extract game state information from the game screen; Step S4: Based on the knowledge base, identified interface elements, and extracted game state information, generate an operation decision sequence for the game; Step S5: Transform the operation decision sequence into control commands that simulate human operation and execute them. At the same time, optimize the strategy for generating the operation decision sequence based on the game feedback information after execution.

[0024] like Figure 2 As shown, this embodiment also provides a game GUI autonomous operating system, including: The knowledge acquisition module is used to acquire knowledge information related to the target game and build a knowledge base based on the acquired knowledge information; The GUI parsing module is used to parse the game footage captured in real time and identify the interface elements within it; The visual perception module is used to perceive the real-time captured game screen and extract the game state information therein; The strategy planning module is used to generate an operation decision sequence for the game based on the knowledge base, the identified interface elements, and the extracted game state information. The execution and optimization module is used to convert the operation decision sequence into control instructions that simulate human operation and execute them, and at the same time optimize the strategy of the strategy planning module based on the game feedback information after execution. Among them, the knowledge acquisition module, GUI parsing module, visual perception module, strategy planning module, and execution and optimization module communicate and collaborate through shared memory or message passing. The knowledge acquisition module includes: The crawler unit is used to crawl strategy texts and game videos related to the target game from Internet resources; The text processing unit is used to extract semantic information from the strategy text using a text processing model to obtain text strategy information. The video parsing unit is used to parse the game video using a visual model to obtain video strategy information; The knowledge base construction unit is used to integrate the text policy information and video policy information to construct a structured knowledge base; The strategy planning module integrates a strategy model, which is a model trained based on reinforcement learning. Its optimization objective is to maximize the cumulative reward across multiple dimensions, taking into account action efficiency, risk control, collaborative performance, and uncertainty handling.

[0025] The self-learning intelligent agent proposed in this embodiment, oriented towards game GUI, can autonomously learn the usage strategies of various games.

[0026] The system is divided into five types of intelligent agents: a knowledge agent responsible for information retrieval and knowledge extraction; a GUI parsing agent responsible for interface element recognition and semantic modeling; a visual perception agent responsible for image state and analysis, such as enemy and friendly positions and skill states; a strategy planning agent that generates action sequences based on a knowledge base and visual information; and an execution agent that simulates human operations to execute action sequences. The agents communicate via shared memory or natural language messages. Figure 1 The information exchange relationships among the agents are shown.

[0027] The knowledge-based intelligent agent comprises a web crawler model, a text model, and a visual model. The web crawler model scrapes publicly available resources such as game strategy websites, forums, and video platforms, using tools like GPT to extract semantics from the strategy text, extracting operational rules and tactical suggestions. The visual model detects player actions based on on-screen elements, mapping video frame sequences to action sequences to generate a training policy library. The visual model is... The parameters are The action sequence generation process can then be represented as: ,in Indicates the first The corresponding action for each frame; the strategy information extracted from the text guide is... The action sequence obtained from video analysis is Then, the construction of the knowledge base can be represented as ,in The integration function includes (1) deduplication and standardization: unifying the representation of duplicate or conflicting policy information; (2) structured encoding: mapping text rules and video actions into a unified structured format; and (3) probability or weight assignment: assigning confidence to each policy. For reinforcement learning or decision optimization, the action policy library can be further represented as: ; It indicates the game status, such as health, position, and cooldown status; This indicates suggested actions, such as skill activation, movement, or attacking a target. Indicates the confidence level or priority of the strategy.

[0028] The GUI parsing agent extracts the set of main GUI elements of the game using a visual model. ,in The game screen indicating the time. The identified set of GUI elements includes skill buttons, health bars, skill cooldowns, inventory, and minimap.

[0029] The visual perception agent further analyzes real-time images. Perform object detection, instance segmentation, and state analysis to extract game state information such as enemy hero positions, teammate positions, jungle monster positions, and your own level. , This indicates the status information of the current frame. The GUI parsing process and the visual perception process are entirely based on visual input, that is... It does not rely on game APIs, ensuring strong compatibility and high concealment.

[0030] The policy planning agent integrates knowledge base information with visual perception state to generate candidate operation sequences: ; ; in, This indicates the action performed by the agent at time step t. Representing the sequence of operations in the first N steps, the policy planning agent simultaneously utilizes visual perception of the state, GUI parsing results, historical action sequences, and knowledge base K to generate operational decisions, thereby achieving multimodal fusion of information.

[0031] This embodiment models agent decision optimization as a reinforcement learning problem. State symbols are used. express Agent strategy In state Select action (e.g., skill release, movement, attacking targets, etc.), the strategy uses a reward function. Learning to select the optimal action under multi-dimensional objectives. The reward function can integrate multiple metrics, such as: ; in, It indicates the efficiency with which an action completes a task. This indicates risk control during the execution of actions. This indicates the performance in collaboration with other intelligent agents or teammates. This represents how an intelligent agent handles uncertainties in the state of the environment.

[0032] The optimization objective of reinforcement learning strategies is to maximize the expected cumulative reward. ; in These are the parameters of the strategy. Representation strategy Expected returns Representative according to strategy Expected value of the sample. The length of the operation sequence.

[0033] The policy gradient update process can be expressed as: ; in This represents the state at step t. This represents the action taken at step t. It is the cumulative reward over the past N steps.

[0034] To improve the robustness of agent decision-making and the balance of multi-dimensional objectives, an iterative closed-loop optimization mechanism is introduced: First, based on the initial weights... Calculate rewards for each metric First, conduct an initial evaluation. Second, optimize the strategy parameters based on the reward signals. ; in, For learning rate, Indicates the first Round iteration.

[0035] Finally, we analyze the extent of improvement for each indicator in this round. Dynamically adjust weights: ; Through repeated state inputs, reward evaluation, policy updates, and weight adjustments, the agent's policy gradually converges, and the generated decision sequences continuously improve in terms of efficiency, risk control, collaboration, and environmental adaptability.

[0036] The agent will perform actions Transform into executable device operations: ; in This represents the action mapping function. To simulate actual human actions, such as mouse clicks or drags, keyboard presses, and gamepad operations, This is the set of all executable actions.

[0037] To avoid external features and unnatural operations, delay and continuity constraints are introduced: ,in This indicates an operational delay or random disturbance. Ensure that the movements are fluid, natural, and in line with human operating habits.

[0038] After executing the agent's operation, the system obtains the real-time status. and rewards The feedback information is used to update the parameters of the policy learning agent, achieving closed-loop optimization. ; This selected embodiment is a collaborative project for a mobile device manufacturer, designing a fully automated intelligent agent system for controlling the opening, playing, and restarting of games on a mobile phone. Its core requirement is to conduct long-term, high-intensity automated performance and stability testing of mainstream large-scale mobile games on newly released mobile phones or chip platforms. The system flow is as follows: 1. Knowledge acquisition and strategy base construction: First, the knowledge agent uses web crawling models, text models, and visual models to automatically crawl strategy texts, tactical suggestions, and high-scoring player videos for games A and B from the internet. A large model is used to extract semantics from the strategy text, and a visual model is used to map video frame sequences to confidence-based sequences. The action strategy sequence. The construction of the knowledge base K enables the system to adapt and quickly adapt to new game versions without the need for manually writing fixed scripts.

[0039] 2. Pure visual state perception: The system relies entirely on pure visual input. To sense the game state, thereby achieving high compatibility and stealth. The GUI parsing agent is responsible for recognizing common interface elements. This includes the location and semantics of elements such as the "Start Match" button, skill cooldown status, health bars, and the results panel. The visual perception agent further performs target detection and state analysis on the real-time screen to extract the battlefield state. Such as the location of enemy and friendly heroes or vehicles, projectile trajectory information, and monster spawns. Status and elements The fusion of visual input forms the basis of strategic decision-making. Relying entirely on visual input ensures that decisions are based on real-time battlefield conditions rather than preset coordinates, thus avoiding the risk of detection.

[0040] 3. Strategy planning and reinforcement learning optimization: Integrating external knowledge base K and GUI parsing results and real-time status Optimal actions are generated by maximizing the expected cumulative reward through reinforcement learning. The strategy planning agent is endowed with a high degree of self-recovery capability, enabling it to generate appropriate recovery actions (such as clicking "reconnect" or "confirm") based on its uncertainty handling ability, avoiding process deadlock and thus significantly improving the system's robustness. Through an iterative optimization mechanism, the weights of multi-dimensional indicators such as task efficiency and collaborative performance are dynamically adjusted, allowing decision-making to find the optimal balance between efficiently completing the game loop and human-like operation.

[0041] 4. Execution of agent operations: The executing agent will perform the actions generated by the policy planner. This is transformed into simulated human touchscreen operation. To avoid external features and unnatural operations, random delays or perturbations are introduced into the execution agent during actual operation. Furthermore, logically coherent algorithms are applied to ensure the naturalness and human-likeness of the actions. For example, in game A, it simulates the slight shaking when a human shoots, or in game B, it simulates the non-linear movement when a human walks. This highly human-like operation, combined with the risk control decision-making of the strategy planning agent, ensures that the test system will not be detected as a human-machine interface by the game's security mechanisms during long-term continuous operation, thus meeting stringent testing requirements.

[0042] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for autonomous operation of a game GUI, characterized in that, Includes the following steps: Step S1: Obtain knowledge information related to the target game, and build a knowledge base based on the obtained knowledge information; The knowledge information includes: operation rules extracted from game strategy texts and operation sequences parsed from game videos; Step S2: Perform GUI parsing on the real-time captured game footage to identify the interface elements in the game footage; Step S3: Perform visual perception on the real-time captured game screen and extract game state information from the game screen; Step S4: Based on the knowledge base, identified interface elements, and extracted game state information, generate an operation decision sequence for the game; Step S5: Convert the operation decision sequence into control instructions that simulate human operation and execute them. At the same time, optimize the strategy for generating the operation decision sequence based on the game feedback information after execution. Step S3 specifically involves: using the second visual model The game footage captured in real time Process the data and output a set of extracted game state information. ,in Indicates time, The parameters of the second visual model include one or more of the following: enemy unit position, friendly unit position, neutral unit position, own level, and skill availability.

2. The game GUI autonomous operation method according to claim 1, characterized in that, Step S1 includes: Step S11: Crawl strategy texts and game videos related to the target game from internet resources; Step S12: Use a text processing model to extract semantic information from the strategy text to obtain text strategy information. ,in Indicates the first A strategy for extracting data from text; Step S13: Analyze the game video using a visual model, mapping the video frame sequence to an action sequence to obtain video strategy information. ,in Indicates the first The analyzed action; Step S14: Integrate text policy information and video policy information to construct a structured knowledge base. The knowledge base Each policy in the table is represented as a triple. ,in This indicates a description of the game state. Indicates a suggested action. This indicates the confidence level or priority weight of the strategy.

3. The game GUI autonomous operation method according to claim 2, characterized in that, In step S14, the integration process includes: deduplicating and standardizing duplicate or conflicting policy information; mapping text rules and video actions into a unified structured format; and assigning confidence to each policy. .

4. The game GUI autonomous operation method according to claim 1, characterized in that, Step S2 specifically involves: using the first visual model The game footage captured in real time Process the data and output the identified set of GUI elements. ,in Indicates time, The parameters of the first visual model are defined as follows: the GUI elements include one or more of the following: skill buttons, health bars, skill cooldown icons, item inventory, and minimap.

5. The game GUI autonomous operation method according to claim 1, characterized in that, Step S4 specifically includes: Step S41: Set the current time Identified UI elements Extracted game status information and the first N historical action sequences Combined into the current decision state ; Step S42: Based on the knowledge base and current decision-making status Through strategy model Generate action decision for the current moment ,Right now ,in These are the parameters of the strategy model; Step S43: Repeat steps S41-S42 to generate a continuous sequence of operation decisions. .

6. The game GUI autonomous operation method according to claim 5, characterized in that, Step S4 also includes applying reinforcement learning to the policy model. Optimization steps: Step S44: Define the reward function The reward function is a weighted sum of rewards from multiple dimensions, including at least: action efficiency reward. Risk control rewards Collaboration performance rewards and rewards for handling uncertainty ; Step S45: Set the optimization objective to maximize the expected cumulative reward. ,in This represents a trajectory consisting of states and actions; Step S46: Update the parameters of the policy model using the policy gradient method. , is represented as: ,in This represents the estimated cumulative reward over the past N steps; Step S47: During the iterative optimization process, dynamically adjust the weight of each dimension's reward in the total reward function according to the improvement magnitude of each dimension's reward.

7. The game GUI autonomous operation method according to claim 1, characterized in that, Step S5 includes: Step S51: Through the action mapping function Action decisions in the operation decision sequence Transform into specific equipment control commands ,Right now The control commands include simulated mouse clicks, keyboard keys, touchscreen touches, or gamepad input; Step S52: Issue the control command Apply delay or random perturbation And take into account the control commands from the previous moment. Through coherent processing functions Generate anthropomorphic final control commands Execute after; Step S53: After executing the final control command, obtain the new game screen as feedback, extract the new game state information, calculate the reward value, and use the reward value to return to step S4 to optimize the strategy for generating the operation decision sequence.

8. A game GUI autonomous operating system, used to implement the game GUI autonomous operation method as described in any one of claims 1-7, characterized in that, include: The knowledge acquisition module is used to acquire knowledge information related to the target game and build a knowledge base based on the acquired knowledge information; The GUI parsing module is used to parse the game footage captured in real time and identify the interface elements within it; The visual perception module is used to perceive the real-time captured game screen and extract the game state information therein; The strategy planning module is used to generate an operation decision sequence for the game based on the knowledge base, the identified interface elements, and the extracted game state information. The execution and optimization module is used to convert the operation decision sequence into control instructions that simulate human operation and execute them, and at the same time optimize the strategy of the strategy planning module based on the game feedback information after execution. The knowledge acquisition module, GUI parsing module, visual perception module, strategy planning module, and execution and optimization module communicate and collaborate through shared memory or message passing.

9. A game GUI autonomous operating system according to claim 8, characterized in that, The knowledge acquisition module includes: The crawler unit is used to crawl strategy texts and game videos related to the target game from Internet resources; The text processing unit is used to extract semantic information from the strategy text using a text processing model to obtain text strategy information. The video parsing unit is used to parse the game video using a visual model to obtain video strategy information; The knowledge base construction unit is used to integrate the text policy information and video policy information to construct a structured knowledge base; The strategy planning module integrates a strategy model, which is a model based on reinforcement learning training. Its optimization objective is to maximize the cumulative reward across multiple dimensions, taking into account action efficiency, risk control, collaborative performance, and uncertainty handling.

Images