A game automation system and method based on OCR recognition

Abstract

The application provides a game automation system and method based on OCR recognition, comprising a business module, an automation module, an OCR module and a scheduling module; the business module is used for game automation configuration information management, task management and execution device management; the automation module is used for matching automation scripts, controlling a task queue, distributing a task and standardizing an automation execution result; the OCR recognition module is used for receiving a screenshot sample in an automation execution process, judging a current automation execution state, extracting and calling key parameters in an automation scene; and the scheduling module is used for distributing device nodes for task execution and controlling a task execution queue; the application provides a high-stability and high-fault-tolerance game full-automation solution according to the business characteristics of great changes in game automation field elements and difficult positioning.

Description

Technical Field

[0001] This invention relates to the field of game automation, specifically to a game automation system and method based on OCR recognition. Background Technology

[0002] Game automation is a complex process involving technical challenges from multiple fields. Current game automation faces the following difficulties: 1. Game environments are diverse; different games have different rules, interfaces, and interaction methods, requiring automation systems to adapt to these differences. 2. Automation systems need to be able to identify and analyze visual information in games, including characters, items, and environments, which is particularly difficult in games with complex graphics or pixelated graphics. 3. Game automation needs to respond to changes in the game environment in real time, placing high demands on the efficiency and response speed of the algorithms. 4. Game developers may update game rules or fix vulnerabilities, requiring automation systems to adapt to these changes. Looking at existing game automation systems, while highly specialized, they are limited by element localization and image matching, and currently lack a comprehensive technical solution or system that addresses the combination of challenges, including the diversity of game interfaces and the complexity of graphics.

[0003] For example, PyAutoGame (a Python-based game automation framework) is based on screen recognition and event simulation. It utilizes OpenCV (a cross-platform computer vision library) for image processing, performing template matching or color recognition on screenshots to locate game elements. Then, it uses the PyAutoGUI module to simulate mouse and keyboard operations, precisely controlling in-game behavior. However, with technological advancements, the complexity of real-world business scenarios has surpassed previous levels, making it impossible to reliably locate game elements using only template matching and color recognition.

[0004] For example, Chinese patent CN115858358A provides a multi-terminal automated game testing system. The automatic testing module identifies and clicks on screen areas in the currently active client. The system divides the game into several segments for simultaneous testing, thereby improving testing efficiency and reducing the time cost of conventional testing. This technical solution can solve the problem of needing to repeat tests and fixes multiple times in game automation, but it lacks technical support for game scenes and elements that are constantly changing. Summary of the Invention

[0005] In view of the shortcomings of existing automation systems, the technical problem to be solved by the present invention is to provide a game automation system and method based on OCR recognition, which can solve the problems of difficulty in recognizing game elements and difficulty in locating game scenes caused by complex game environments and real-time page changes.

[0006] To achieve the above objectives, according to one aspect of the present invention, the present invention is implemented through the following technical measures: a game automation system based on OCR recognition, comprising a business module, an automation module, an OCR recognition module, and a scheduling module;

[0007] The business module is used for automated configuration information management, task management, and device management.

[0008] The automation module is used to match automation scripts, control task queues, distribute and schedule tasks, and standardize automation execution results.

[0009] The OCR recognition module is used to receive screenshot samples during the automated execution process, determine the current game state of automated execution, and extract and call back key parameters in the automated scene;

[0010] The scheduling module is used to allocate device nodes for task execution, control the task execution queue, and monitor the device status under the system.

[0011] Furthermore, the business module configures key information for scenarios such as parsing parameters and downloading code in automated tasks through an interface, modifies device alarm thresholds, and associates tasks with the execution device cluster.

[0012] Furthermore, the automation module receives valid request parameters that have passed verification, matches the device node cluster used to execute the corresponding game automation task according to the parameters, queries the available device nodes, and distributes the automation task to the matched available device nodes. According to the project address configured in the data module, the game automation code is downloaded from the cloud repository to the device node, the game automation code is executed, and the execution process is monitored. During the execution process, the current game status is determined by taking automated screenshots and requesting the OCR recognition module's service through the request library. The return result of the OCR recognition module determines the next automation operation strategy, so as to achieve the purpose of accurately recognizing game elements and status.

[0013] Furthermore, the OCR recognition module receives automated scripts that automatically capture screen images at key points in the game. It then preprocesses the screenshots using image processing technology to improve OCR accuracy. Next, image recognition technology is used to locate areas in the screenshots containing key information. OCR technology is then used to recognize the text within these areas, converting the recognized text into structured data. For example, if the recognized information is health, it is converted into a numerical value; if it is a task description, keywords or task IDs are extracted. Based on the extracted information, the automated script determines the current game state and sends the identified and extracted key parameters back to the automated script so that the script can make decisions based on these parameters. For example, if health is below a certain threshold, a healing mechanism is triggered when a healing operation is needed; if specific task keywords are identified, task navigation is triggered when a switch to the corresponding task execution flow is required. Finally, after executing the operation, the automated script takes another screenshot and performs recognition to monitor the effect of the operation and adjust the strategy as needed.

[0014] Furthermore, the key elements include the game's menu interface, battle scenes, and mission prompts, while the image processing techniques include grayscale conversion, binarization, noise reduction, and edge detection.

[0015] Furthermore, upon receiving an automated task, the scheduling module allocates device nodes for task execution, controls the task execution queue, monitors the connection status of the device node cluster, and issues a specific project group alarm when the device connection is lost. Simultaneously, it receives requests from the automation module and, in the event of abnormal situations such as game updates or abnormal network disconnections during automated execution, executes instructions and strategies to disable, restart, or reconnect device nodes.

[0016] In another aspect, the present invention provides a game automation method based on the aforementioned OCR-based game automation system, the method comprising the following steps:

[0017] S1. Design game automation scripts, in which the game interface recognition stage uses the recognition return of the OCR module to formulate an automation execution strategy;

[0018] S2. After the local automation script is debugged and passed, upload the code to GitLab, add the game automation scene configuration through the configuration management module, and set the parameters that the script needs to receive and the machine node to execute through the task configuration module.

[0019] S3. Initiate game automation tasks through the interface provided by the automation module. The task status can be queried based on the unique ID of the automation task returned by the interface. After receiving the request, the automation module will parse and verify the parameters. If the verification passes, it will notify the scheduling layer to query the available machines and assign the task to an idle available machine. If there are not enough available machines, the alarm policy will be executed.

[0020] S4. The automated task starts on the execution machine. The scheduling layer cleans up the workspace, then downloads the automated script from S1 to the execution machine and starts it. The game script drives the corresponding game program. During execution, it takes screenshots and uploads them, then requests the OCR interface. Based on the information returned by the OCR interface, it determines the current game state and executes the next strategy in the script. The above steps are repeated throughout the game automation process.

[0021] S5. After the game automation ends normally or an exception occurs, the automation script will send the execution result and status back to the automation module. The automation module will format the parameters and store them in the database. If an exception occurs, an alarm will be pushed according to the preset alarm configuration in the configuration management to guide the developers to locate the problem.

[0022] S6 allows developers to query task logs and task status using fields such as task ID, task type, and task name through the automation module, and retrieve data statistics reports based on game scenarios and time.

[0023] Compared with the prior art, the advantages of the present invention are as follows: This game automation system and method based on OCR recognition provides a highly stable and fault-tolerant fully automated game solution that addresses the business characteristics of game automation, such as large element changes and difficulty in positioning. Attached Figure Description

[0024] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0025] Figure 1 This is a schematic diagram of the system architecture of a game automation system based on OCR recognition as described in this invention;

[0026] Figure 2 This is an execution flowchart of a game automation method according to the present invention;

[0027] Figure 3 This is a flowchart illustrating the alarm strategy of a game automation method according to the present invention. Detailed Implementation

[0028] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present application can be combined with each other.

[0029] In the description of this invention, it should be understood that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", "bottom", "top", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0030] Please refer to Figure 1 This embodiment provides a game automation system based on OCR recognition, including a business module, an automation module, an OCR recognition module, and a scheduling module;

[0031] The business module is used for automated configuration information management, task management, and execution device management. The business module configures key information for scenarios such as parsing parameters and downloading code in automated tasks through interfaces, modifies device alarm thresholds, and associates tasks with execution device clusters.

[0032] The automation module is used to match automation scripts, control task queues, distribute and schedule tasks, and standardize automation execution results. The automation module receives valid request parameters that have passed verification, matches the device node clusters for executing the corresponding game automation tasks based on the parameters, queries the available device nodes, and distributes the automation tasks to the matched available device nodes. According to the project address configured in the data module, the game automation code is downloaded from the cloud repository to the device nodes, the game automation code is executed, and the execution process is monitored. During the execution process, the current game status is determined by taking automated screenshots and requesting the OCR recognition module's service through the request library. The return result of the OCR recognition module determines the next automation operation strategy, so as to achieve the purpose of accurately recognizing game elements and status.

[0033] The OCR recognition module receives screenshot samples during automated execution, determines the current game state, extracts and calls back key parameters in the automated scene. The module automatically captures screen images at key points in the game from the automated script and preprocesses the screenshots to improve OCR accuracy. Key points include game menus, battle scenes, and quest prompts. Image processing techniques include grayscale conversion, binarization, denoising, and edge detection. Then, image recognition techniques (such as template matching and feature point matching) are used to locate areas containing key information in the screenshot (e.g., identifying and locating a character's health bar, experience points, quest descriptions, etc.). Finally, OCR technology is used to recognize the text within the located area (the font may vary depending on the game). The system performs separate training to convert the identified text into structured data (e.g., if the identified text is health, it can be converted into a numerical value; if it is a task description, keywords or task IDs can be extracted). Then, based on the extracted information, the automated script determines the current game state (e.g., if the health is below a certain threshold, it determines that a healing operation may be needed; if specific task keywords are identified, it determines that a switch to the corresponding task execution flow may be needed). The key parameters identified and extracted are then fed back to the automated script so that the script can make decisions based on these parameters (e.g., health information may trigger a healing mechanism, task information may trigger task navigation). Finally, after executing the operation, the automated script takes a screenshot and performs recognition again to monitor the effect of the operation and adjust the strategy as needed.

[0034] The scheduling module is used to allocate device nodes for task execution, control the task execution queue, and monitor the device status under the system. When the scheduling module receives an automated task, it allocates device nodes for task execution, controls the task execution queue, monitors the connection status of the device node cluster, and issues alarms for specific project groups when the device connection is lost. At the same time, it receives requests from the automation module and executes instructions and strategies to disable, restart, or reconnect device nodes in case of abnormal situations such as game updates or abnormal network disconnection during the automated execution process.

[0035] Please refer to Figure 1 The system architecture of a game automation system based on OCR recognition provided in this embodiment includes the following:

[0036] 1. Business layer, including task management, configuration management, script management, and device management;

[0037] 1.1 Task Management: The task list provides dynamic parameter configuration via an interface. The task list is stored in the database as a dictionary, where the key represents the specific game automation scenario and the value represents the association identification code between the game scenario and the automation task. When a user initiates an automated game task request on the page, the automation module will parse the task parameters and match them with the dynamic parameter configuration of the task list after the permission verification is passed, thereby associating a unique automation task. The automation task will then schedule idle node machines to execute the corresponding automated game script. The task management provides interfaces for adding, modifying, and deleting tasks for managing the task list.

[0038] 1.2 Configuration Management: Provides management methods for configuration information in the form of interfaces, including project groups and mailing groups for pushing alarm information, keys for hashing task parameters, keys for pulling code repositories via GitLab, and execution machine thresholds for task initiation detection (when a task is initiated, idle execution machines are checked first, and specific alarm settings are applied when the number of idle machines is less than a certain parameter).

[0039] 1.3 Script Management: Scripts are managed uniformly on GitLab in the form of code repositories. The interface provides detection and remote scheduling capabilities, and supports one-click code updates and one-click code retrieval on execution nodes.

[0040] 1.4 Device Management: Device management capabilities are provided in the form of interfaces. The automation module integrates the Jenkins component library. All automated running devices are connected to the Jenkins components as nodes. Devices can be connected, disabled, started, queried, and log exported through the API provided by Jenkins. At the same time, these devices are also connected to the JumpServer bastion host, supporting visual remote connection and debugging via link sharing.

[0041] 2. Service layer, including automation module and OCR module;

[0042] 2.1 Automation Module: Provides automation services via interfaces, offering capabilities including but not limited to: requesting permission and parameter verification; initiating game automation tasks upon successful verification; monitoring the game automation status during task execution through mutual calls with the OCR module; receiving script execution result callbacks after the automation task is completed; storing the callbacks in a structured database; supporting querying game automation tasks; supporting task replay when game automation tasks encounter exceptions; supporting the uploading and downloading of various game screenshots; and supporting the receiving and querying of dynamic parameters such as CAPTCHAs during script execution.

[0043] 2.2 OCR Module: Provides OCR services via an interface, offering capabilities including but not limited to the following: preprocessing screenshots to improve OCR accuracy, which may include image processing techniques such as grayscale conversion, binarization, denoising, and edge detection; using image recognition techniques (such as template matching and feature point matching) to locate areas containing key information in the screenshot, such as identifying and locating a character's health bar, experience points, or task descriptions; using OCR technology to recognize text within the located area and converting the recognized text into structured data, such as converting health points into numerical values ​​if they are recognized; extracting keywords or task IDs if it is a task description; sending the identified and extracted key parameters back to the automation script so that the script can make decisions based on these parameters, such as health information triggering a healing mechanism or task information triggering task navigation; and designing an error handling mechanism to handle OCR recognition errors or automation operation failures, which may include retry mechanisms, exception handling, and logging.

[0044] 3. Scheduling layer:

[0045] 3.1 Scheduling Module: This module mainly consists of Jenkins and JumpServer. Jenkins, upon receiving a game automation task, allocates idle node machines to execute the script based on the task's associated node machine information. During execution, it first cleans up the local code space, then pulls the game script code from GitLab for execution based on the task's configuration information and key, while simultaneously tracking the task's execution status. It also provides an API for the automation module to manage tasks and nodes. JumpServer, on the other hand, provides the ability to remotely and visually connect to node machines, allowing developers to track the execution status of game automation tasks in real time.

[0046] 4. Data Layer:

[0047] 4.1 OSS: Provides screenshot management capabilities. During the execution of the automated script, screenshots are uploaded to OSS through the interface provided by the automation module. At the same time, the corresponding screenshot OSS external link is called back to the OCR module. The OCR module downloads the screenshot through the OSS external link, performs recognition and information extraction, and sends the results back to the game automation script.

[0048] 4.2 Redis: Provides caching services for task information, configuration information, and other data;

[0049] 4.3 MySQL: Provides storage services for information such as automated task parameters, automated execution results, dynamic parameter configuration, and node machine list.

[0050] 5. Operating environment:

[0051] 5.1 PC Nodes: Alibaba Cloud PC hosts are provided. These physical PC hosts are used for the automated hardware support of the client game. JDK, Git, and Python need to be installed on these PCs. JDK is used for Jenkins access, Git is used to pull game automation scripts from GitLab, and Python is used for the runtime environment of game automation. These PC hosts are all connected to the automation module as Jenkins nodes and are subject to unified scheduling and management by the automation module.

[0052] 5.2 Mobile Nodes: Provides mobile cloud real devices and mobile emulators. The mobile execution machine is connected to one or more computer nodes via USB. In addition to installing JDK, Git, and Python, these computer nodes also need to install a mobile automation environment, such as installing the adb environment for Android machines and the WebDriverAgent tool for iOS machines.

[0053] Please refer to Figure 2 This embodiment also provides a game automation method based on OCR recognition, including the following steps:

[0054] S1. Design game automation scripts, in which the game interface recognition stage uses the recognition return of the OCR module to formulate an automation execution strategy;

[0055] S2. After the local automation script is debugged and passed, upload the code to GitLab, add the game automation scene configuration through the configuration management module, and set the parameters that the script needs to receive and the machine node to execute through the task configuration module.

[0056] S3. Initiate a game automation task through the interface provided by the automation module. The task status can be queried using the unique automation task ID returned by the interface. After receiving the request, the automation module parses and validates the parameters. If the validation passes, it notifies the scheduling layer to query available machines and assign the task to an idle machine. If there are insufficient available machines, it will then... Figure 3 Implement alarm policies;

[0057] S4. The automated task starts on the execution machine. The scheduling layer cleans up the workspace, then downloads the automated script from S1 to the execution machine and starts it. The game script drives the corresponding game program. During execution, it takes screenshots and uploads them, then requests the OCR interface. Based on the information returned by the OCR interface, it determines the current game state and executes the next strategy in the script. The above steps are repeated throughout the game automation process.

[0058] S5. After the game automation ends normally or an exception occurs, the automation script will send the execution result and status back to the automation module. The automation module will format the parameters and store them in the database. If an exception occurs, an alarm will be pushed according to the preset alarm configuration in the configuration management to guide the developers to locate the problem.

[0059] S6 allows developers to query task logs and task status using fields such as task ID, task type, and task name through the automation module, and retrieve data statistics reports based on game scenarios and time.

[0060] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can have various changes and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A game automation system based on OCR recognition, characterized in that: It includes a business module, an automation module, an OCR recognition module, and a scheduling module; The business module is used for automated configuration information management, task management, and device management. The automation module is used to match automation scripts, control task queues, distribute and schedule tasks, standardize automation execution results, and generate statistical reports that include execution time and operation success rate. The OCR recognition module is used to receive screenshot samples during automated execution, determine the current game state of automated execution, extract and call back key parameters in the automated scene. The OCR recognition module receives screen images automatically captured by the automated script at key points in the game, and uses image processing technology to preprocess the screenshots to improve the accuracy of OCR recognition. The key elements of the game include the game's menu interface, battle scenes, and mission prompts; Then, image recognition technology using feature point matching is used to locate the area containing key information in the screenshot. The feature point matching technology is based on the ORB algorithm to extract key points in the image, and then OCR technology is used to recognize the text in the located area. The recognized text is converted into structured data. If the recognized information is health, it is converted into a numerical value; if it is a task description, keywords or task IDs are extracted. Then, based on the extracted information, the current game state is determined by an automated script, and the key parameters of recognition and extraction are called back to the automated script so that the script can make decisions based on these parameters. If the health is lower than a certain threshold, a health replenishment mechanism will be triggered when it is determined that a health replenishment operation needs to be performed. If a specific task keyword is recognized, a task navigation will be triggered when it is determined that the corresponding task execution process needs to be switched. Finally, after the automated script performs the operation, it takes screenshots and performs recognition again to monitor the effect of the operation and adjust the strategy as needed. Design an error handling mechanism to deal with OCR recognition errors and automation operation failures, including retry mechanism, exception capture and logging; The scheduling module is used to allocate device nodes for task execution, control the task execution queue, and monitor the device status under the system.

2. The game automation system based on OCR recognition according to claim 1, characterized in that: The business module configures the key information for parsing parameters and downloading code in automated tasks through an interface, modifies device alarm thresholds, and associates tasks with the execution device cluster.

3. The game automation system based on OCR recognition according to claim 1, characterized in that: The automation module receives valid request parameters that have passed verification, matches the device node clusters used to execute the corresponding game automation tasks based on the parameters, queries the available device nodes, and distributes the automation tasks to the matched available device nodes. According to the project address configured in the data module, the game automation code is downloaded from the cloud repository to the device nodes, the game automation code is executed, and the execution process is monitored. During the execution process, the current game status is determined by taking automated screenshots and requesting the OCR recognition module's service through the request library. The return result of the OCR recognition module determines the next automation operation strategy, so as to achieve the purpose of accurately recognizing game elements and status.

4. The game automation system based on OCR recognition according to claim 1, characterized in that: The image processing techniques include grayscale conversion, binarization, denoising, and edge detection using the Sobe operator.

5. The game automation system based on OCR recognition according to claim 1, characterized in that: When the scheduling module receives an automated task, it allocates device nodes for task execution, controls the task execution queue, monitors the connection status of the device node cluster, and issues a specific project group alarm when the device connection is lost. At the same time, it receives requests from the automation module and executes instructions and strategies to disable, restart, or reconnect device nodes in case of game updates or abnormal network disconnections during the automated execution process.

6. A game automation method based on the OCR-based game automation system according to any one of claims 1-5, the method comprising the following steps: S1. Design game automation scripts, in which the game interface recognition stage uses the recognition return of the OCR module to formulate an automation execution strategy; S2. After the local automation script is debugged and passed, upload the code to GitLab, add game automation scene configuration through the configuration management module, and set the parameters that the script needs to receive and the machine node to execute through the task configuration module. S3. Initiate game automation tasks through the interface provided by the automation module. The task status can be queried based on the unique ID of the automation task returned by the interface. After receiving the request, the automation module will parse and verify the parameters. If the verification passes, it will notify the scheduling layer to query the available machines and assign the task to an idle available machine. If there are not enough available machines, the alarm policy will be executed. S4. The automated task starts on the execution machine. The scheduling layer cleans up the workspace, then downloads the automated script from S1 to the execution machine and starts it. The game script drives the corresponding game program. During execution, it takes screenshots and uploads them, then requests the OCR interface. Based on the information returned by the OCR interface, it determines the current game state and executes the next strategy in the script. The above steps are repeated throughout the game automation process. S5. After the game automation ends normally or an exception occurs, the automation script will send the execution result and status back to the automation module. The automation module will format the parameters and store them in the database. If an exception occurs, an alarm will be pushed according to the preset alarm configuration in the configuration management to guide the developers to locate the problem. S6. Developers can use the automation module to query task logs and task status based on task ID, task type, and task name fields, and retrieve data statistics reports based on game scene and time.

Images