Cloud mobile phone intelligent automatic operation method based on AI picture recognition

Abstract

The invention provides a cloud mobile phone intelligent automatic operation method based on AI picture recognition, and belongs to the technical field of data processing, and the method specifically comprises the steps: receiving a task execution request and a corresponding task configuration file; collecting a current screen picture of the cloud mobile phone in real time; the method comprises the following steps: preprocessing a current screen picture, identifying and analyzing, establishing structured description information of the current screen picture, and forming an interactive element list according to the structured description information; according to the target of the current execution step and the interactive element list, intelligent decision making is carried out through a rule engine or a decision making model, a target element to be operated is determined, and a corresponding simulation user operation instruction is generated; executing a simulation user operation instruction through a system input event injection mode; after the operation instruction is executed, a screen picture is collected and recognized, whether operation succeeds or not is verified based on interface changes before and after operation, and the task execution state is updated according to a verification result. According to the scheme, the operation success rate and adaptability are improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a cloud phone intelligent automated operation method based on AI image recognition. Background Technology

[0002] Currently, cloud phones primarily employ UI automation testing frameworks (such as Appium and UiAutomator) or accessibility services in scenarios such as application automation testing, game hosting, and batch operations. These solutions obtain the application's UI control tree structure, locate elements based on control IDs, text, XPath, and other attributes, and then execute operations. Alternatively, they listen to UI events on the screen to obtain control node information and simulate user clicks, swipes, and other actions. Some solutions utilize recording and playback technology to record the user's operation trajectory (coordinates, time, action type) and replay it under the same scenario.

[0003] However, in the process of researching and practicing the existing technology, the inventors of this invention discovered that the existing technology has the following defects: 1. Strong dependence on application structure: UI control-based solutions require open accessibility of the application. Custom drawing interfaces for applications such as games and videos cannot be effectively recognized. Furthermore, changes in control IDs and hierarchical structures after application version updates can cause scripts to fail. 2. High adaptation costs: Different applications and versions require separate writing and maintenance of automation scripts, making it impossible to quickly adapt to new applications and resulting in huge manual maintenance costs; 3. Poor flexibility: The recording and playback solution is based on fixed coordinates, which makes it weak in terms of fault tolerance and robustness when facing devices with different resolutions or changes in the position of interface elements. 4. Lack of intelligent decision-making: Existing solutions are mostly simple conditional judgments and fixed process executions, which cannot make intelligent decisions and handle anomalies according to complex scenarios. They are prone to freezing when encountering unexpected pop-ups, network latency, etc. 5. Limited recognition capabilities: Unable to understand the semantic information of the screen content, and unable to handle complex tasks that require content understanding, such as identifying specific products or judging the state of a game battle.

[0004] These shortcomings result in existing cloud phone automation solutions having problems such as low success rate, difficult maintenance, and limited applicable scenarios in practical applications, making it difficult to meet the needs of large-scale commercial applications.

[0005] It is evident that there is an urgent need for a cloud phone intelligent automation operation method based on AI image recognition with high success rate and adaptability. Summary of the Invention

[0006] In view of this, the present disclosure provides a cloud phone intelligent automated operation method based on AI image recognition, which at least partially solves the problems of poor operation success rate and adaptability in the prior art.

[0007] This disclosure provides a cloud phone intelligent automated operation method based on AI image recognition, including: Step 1: Receive the task execution request and the corresponding task configuration file, wherein the task configuration file defines an automated process with multiple steps. Step 2: In response to the task execution request, collect the current screen image of the cloud phone in real time based on system service permissions; Step 3: After preprocessing the current screen image, input it into the multimodal AI recognition engine for recognition and analysis, establish the structured description information of the current screen image, and form a list of interactive elements accordingly; Step 4: Based on the goal of the current execution step and the list of interactive elements, make intelligent decisions through a rule engine or decision model to determine the target element to be operated and generate the corresponding simulated user operation instructions. Step 5: Execute simulated user operation commands through system input event injection; Step 6: After executing the operation command, capture and recognize the screen image again, verify whether the operation was successful based on the changes in the interface before and after the operation, and update the task execution status according to the verification result.

[0008] According to a specific implementation of an embodiment of this disclosure, step 2 specifically includes: Step 2.1: Access the screen frame buffer directly through the SurfaceFlinger mechanism in the Android system service to collect raw frame data; Step 2.2: Dynamically adjust the image capture frequency and capture area according to the current task type; Step 2.3: Perform format conversion and size normalization on the original frame data to obtain the current screen image.

[0009] According to a specific implementation of an embodiment of this disclosure, step 3 specifically includes: Step 3.1: After preprocessing the current screen image, input it into the scene classification model to determine the current interface type; Step 3.2: Based on the current interface type, use the OCR engine, object detection model and icon matching algorithm to identify the text content and position, UI controls and bounding boxes, and specific icons in the image. Step 3.3: Integrate the recognition results of the OCR engine, object detection model and icon matching algorithm to generate structured description information containing element type, text content, coordinate position and confidence information, and form an interactive element list accordingly.

[0010] According to a specific implementation of an embodiment of this disclosure, step 4 specifically includes: Step 4.1: Based on the target element features defined in the current step, locate the target element in the list of interactive elements using exact matching, fuzzy matching, or combined matching. Step 4.2: Based on the predefined decision rule base or the decision model trained on historical data, determine the type of operation and parameters to be performed in conjunction with the target elements; Step 4.3: Generate specific operation instructions based on the operation type and parameters, and add random offsets to the coordinates in the operation instructions and / or generate curves that simulate human operation for the sliding trajectory.

[0011] According to a specific implementation of an embodiment of this disclosure, step 5 specifically includes: Step 5.1: Inject touch events directly into the / dev / input / event device node through the Android system's Input subsystem; Step 5.2: Before performing the operation, check the device screen status, network connection, and application foreground status; Step 5.3: After performing the operation, set a reasonable waiting time according to the operation type, and continuously monitor the changes on the screen during the waiting period.

[0012] According to a specific implementation of this disclosure, the step of verifying whether the operation was successful based on the changes in the interface before and after the operation, and updating the task execution status according to the verification result, includes: Step 6.1: If the operation is successful, the task status will be moved to the next sequential step, conditional branch, or loop step according to the definition in the task configuration file. Step 6.2: If the operation fails or the result does not meet expectations, retry or perform error recovery operation according to the predefined exception handling strategy. The exception handling strategy includes delaying retry, closing the unexpected pop-up window, performing a return operation, or restarting the application.

[0013] According to a specific implementation of this disclosure, before step 1, the method further includes: The AI ​​automation controller is initialized when the Android system service starts with the system. Load a pre-set AI model file into memory, wherein the AI ​​model file includes a lightweight OCR model, an object detection model, and a scene classification model; Establish a connection channel with the screen capture service.

[0014] According to a specific implementation of this disclosure, after step 6, the method further includes: Collect screen images that have a confidence level below the threshold or have failed to be identified during task execution; After labeling the screen images, they are added to the training dataset, which is then used to incrementally train or fine-tune the model in the multimodal AI recognition engine. Based on the success and failure feedback of task execution, the strategy in the decision-making model is optimized through a reinforcement learning mechanism.

[0015] According to a specific implementation of this disclosure, after step 6, the method further includes: During the execution of the detection task, target data is monitored and collected in real time. The target data includes task progress, recognition accuracy, operation success rate, system resource consumption indicators, as well as timestamps, screenshots, recognition results and operation instructions for each step. After the task is completed, an execution report is generated based on the collected data, including total time consumption, step time distribution, success and failure statistics, and exception logs.

[0016] The AI-based screen recognition-based intelligent automated operation scheme for cloud phones in this embodiment includes: Step 1, receiving a task execution request and a corresponding task configuration file, wherein the task configuration file defines an automated process including multiple steps; Step 2, responding to the task execution request, collecting the current screen image of the cloud phone in real time based on system service permissions; Step 3, preprocessing the current screen image, inputting it to a multimodal AI recognition engine for recognition and analysis, establishing a structured description of the current screen image, and forming an interactive element list accordingly; Step 4, based on the target of the current execution step and the interactive element list, making intelligent decisions through a rule engine or decision model, determining the target element to be operated, and generating corresponding simulated user operation instructions; Step 5, executing the simulated user operation instructions through system input event injection; Step 6, after executing the operation instructions, collecting the screen image again and performing recognition, verifying the success of the operation based on the interface changes before and after the operation, and updating the task execution status based on the verification results.

[0017] The beneficial effects of the embodiments disclosed herein are as follows: 1. High versatility and low adaptation cost: Independent of the application's UI control structure, it is entirely based on visual recognition. One system can adapt to multiple application types without requiring separate script development for each application. When application versions are updated, as long as the visual characteristics of the interface elements are basically maintained, the system can still function normally, significantly reducing maintenance costs.

[0018] 2. Strong recognition capability and wide applicability: Combining multiple AI technologies such as OCR, object detection, and image classification, it can not only recognize UI controls, but also understand the semantics of the screen content. It can handle custom-drawn interfaces such as games and videos, as well as complex tasks that require content understanding.

[0019] 3. Robust and fault-tolerant: Through an intelligent decision-making system, it can handle various abnormal situations such as changes in the position of interface elements, adaptation to different resolutions, unexpected pop-ups, and network latency. Operation execution uses relative positioning and feature matching, rather than fixed coordinates, making it highly adaptable to environmental changes.

[0020] 4. High level of intelligence: By incorporating deep learning and reinforcement learning technologies, the system can make intelligent decisions based on the scenario, supporting continuous learning and strategy optimization. As execution data accumulates, the system becomes increasingly intelligent, and the success rate continues to improve.

[0021] 5. Thorough performance optimization: Through optimization techniques such as model quantization, hardware acceleration, and dynamic acquisition strategies, system resource consumption is kept within a reasonable range while ensuring recognition accuracy. It supports large-scale concurrent execution, meeting the performance requirements of cloud phone scenarios.

[0022] 6. System-level integration with ample permissions: Integrated into Android system services, it has system-level permissions, allowing direct access to the screen frame buffer and injection of input events without user authorization, resulting in optimal performance and the best user experience.

[0023] 7. Robust architecture and strong scalability: Utilizing a layered architecture and modular design, each layer has clearly defined responsibilities and standardized interfaces. It supports plug-in extensions, allowing for easy addition of new recognition models, decision-making algorithms, and execution methods. Equipped with a comprehensive task management, monitoring and alerting, and log analysis system, it possesses full capabilities for commercial applications. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart illustrating an AI-based intelligent automated operation method for cloud phones, provided in an embodiment of this disclosure; Figure 2 This is a schematic diagram illustrating the specific implementation process of a cloud phone intelligent automated operation method based on AI image recognition, provided in an embodiment of this disclosure. Detailed Implementation

[0026] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0027] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0029] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0030] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0031] This disclosure provides a cloud phone intelligent automated operation method based on AI screen recognition. The method can be applied to the automated operation process of cloud phone interface in scenarios such as automated testing, game hosting, and batch operations.

[0032] See Figure 1 This is a flowchart illustrating a cloud phone intelligent automated operation method based on AI image recognition, provided in an embodiment of this disclosure. Figure 1 and Figure 2 As shown, the method mainly includes the following steps: Step 1: Receive the task execution request and the corresponding task configuration file, wherein the task configuration file defines an automated process with multiple steps. In practice, during the system initialization phase: when the system starts, the AI ​​automation controller is initialized in the Lifecycle of CloudGameService, the task configuration file and AI model file are loaded, a connection channel with the SurfaceFlinger service is established, and screen capture capability is prepared. Specifically, the system is integrated into the existing CloudGameService framework, utilizing the SystemService lifecycle management mechanism to complete initialization during the PHASE_BOOT_COMPLETED phase. Simultaneously, AI models such as OCR models, scene classification models, and object detection models are loaded into memory, using inference engines like TensorFlow Lite or NCNN, supporting GPU / NPU hardware acceleration.

[0033] Task startup and configuration loading: When a task execution request is received, the task configuration file (JSON or XML format) is parsed to obtain information such as task objectives, execution step sequence, element identification rules, judgment conditions, and exception handling strategies, and the task execution state machine is constructed.

[0034] The task configuration describes a complete automated workflow. Each step includes: step name, target element characteristics (text, icon, position, etc.), action type (click, swipe, input, wait), success criteria, timeout, and number of retries. It supports sequential execution, conditional branching, and loop repetition control logic.

[0035] Step 2: In response to the task execution request, collect the current screen image of the cloud phone in real time based on system service permissions; In practice, the screen image is captured in real time: the screen frame buffer is directly accessed through the SurfaceFlinger mechanism to capture the currently displayed image in real time. The capture frequency (2-10 frames / second) and capture area (full screen or ROI area) are dynamically adjusted according to the task type, and the captured raw images are preprocessed such as format conversion and size normalization. Specifically, it can leverage the high privileges of system services to directly read screen frame data without user authorization, offering superior performance compared to the MediaProjection API. By calculating the perceptual hash values ​​of adjacent frames, it skips recognition when the screen change is less than a threshold, saving computational resources. Preprocessing includes operations such as RGBA to RGB conversion and scaling to the model input size (e.g., 640x640).

[0036] Step 3: After preprocessing the current screen image, input it into the multimodal AI recognition engine for recognition and analysis, establish the structured description information of the current screen image, and form a list of interactive elements accordingly; In practice, the preprocessed image can be input into the AI ​​recognition engine for multi-level recognition and analysis. (1) Use the scene classification model to quickly determine the current interface type (desktop, login page, main interface, settings page, etc.) and output the scene category and confidence level; (2) Select the recognition strategy according to the scene type, call the OCR engine to recognize the text content and position on the screen, call the object detection model to recognize UI controls (buttons, input boxes, icons, etc.) and bounding boxes, and call the icon matching algorithm to recognize specific icons; (3) The results of multiple recognition modules are integrated to establish a structured description of the current interface and form a column of interactive elements.

[0037] For example, the OCR engine uses lightweight solutions such as PaddleOCR, supporting Chinese, English, and multilingual recognition. The object detection model uses mobile-optimized models such as YOLO or MobileNet-SSD; after model quantization (FP32 to INT8), the single-frame inference time is controlled within 100-150ms. Modules can be executed in parallel, improving overall efficiency. Recognition results are stored in a JSON structure, containing information such as element type, text content, coordinates, and confidence level.

[0038] Step 4: Based on the goal of the current execution step and the list of interactive elements, make intelligent decisions through a rule engine or decision model to determine the target element to be operated and generate the corresponding simulated user operation instructions. In practice, target elements can be retrieved from the recognition results based on the objective of the current execution step. Multiple location methods are supported: matching by text content (e.g., "find buttons containing 'login'"), matching by icon features, matching by location region, matching by element type, and combined matching. When exact matching fails, fuzzy matching and similarity calculation are enabled.

[0039] Specifically, the system maintains a rules engine containing a predefined decision rule library. For example, rules could include: "Click the 'OK' button if it's detected," "Go back and retry if an error message is detected," and "Call the CAPTCHA recognition service if a CAPTCHA input box is detected." For complex decision-making scenarios, the system supports decision models trained on historical data to learn the optimal strategy.

[0040] Then, based on the decision results, specific operation instructions are generated, including operation type (click, long press, swipe, text input, wait) and operation parameters (target coordinates, text content, swipe direction and distance, wait time, etc.). To simulate real user behavior, random offsets are added to the coordinates, Bézier curve trajectories are generated for swipes, and random intervals are added to the input.

[0041] Step 5: Execute simulated user operation commands through system input event injection; In practice, precise operation steps can be executed by directly injecting touch events through the Input subsystem or by combining with the Xposed framework for deep control. Before execution, the device status (screen on, network connection, application in foreground, etc.) is checked. After execution, a reasonable waiting time is added according to the operation type (2-5 seconds for network operations, 0.5-1 second for local UI responses). During the waiting period, screen changes are continuously monitored.

[0042] Specifically, the system prioritizes Input event injection (requires system privileges), sending touch events directly through the / dev / input / event device node for optimal performance. Alternative solutions include hooking InputManager via Xposed or simulating operations using an Accessibility Service. Click coordinates are randomly selected within element boundaries to avoid fixing the click center position. The swipe trajectory is generated using a cubic Bézier curve, ensuring speed consistent with human operation characteristics.

[0043] Step 6: After executing the operation command, capture and recognize the screen image again, verify whether the operation was successful based on the changes in the interface before and after the operation, and update the task execution status according to the verification result.

[0044] In practice, after the operation is completed, the screen image is captured and recognized again. The changes in the interface before and after the operation are compared to determine whether the operation was successful. For example, after clicking the login button, it checks whether homepage features or a login success message appear; after clicking the search button, it checks whether a list of search results appears.

[0045] If an operation fails or the result is not as expected, it will be handled according to the failure type and retry strategy. The strategies include: delayed retry (up to 3 times) if recognition fails; retrying after a period of time if the interface freezes; recognizing and closing unexpected pop-ups if they occur; retrying after a network timeout if the timeout is reached; and restarting the application if it crashes. All exceptions are logged in detail, including screenshots, recognition results, and error reasons.

[0046] Additionally, flow control and state transition steps can be set: update the task execution state machine, record the completion status of the current step and intermediate results. Determine the next step based on the task configuration: if it is sequential execution, proceed to the next step; if it is a conditional branch, select a path based on the judgment result; if it is a loop, check whether the exit condition is met. Repeat the above steps until all steps are completed or a termination condition is encountered.

[0047] Throughout the execution process, the system monitors task progress, recognition accuracy, operation success rate, and performance metrics (CPU / memory / GPU utilization, recognition time, operation response time, etc.) in real time. It detects anomalies (timeouts, consecutive failures, excessive resource consumption, etc.) and triggers alarms. Detailed data for each step of execution is collected, including timestamps, screenshots, recognition results, decision-making processes, operation instructions, and execution results.

[0048] Upon completion of the task, a detailed execution report is generated, including total time, time distribution for each step, success / failure statistics, key screenshots, exception logs, and performance data. The report is stored in JSON format and can be exported as a visual HTML page. The execution data is also archived for subsequent data analysis and model optimization.

[0049] It can also collect samples that fail to be identified or have low confidence during the execution process, manually label them, and then add them to the training set. New data is used regularly for incremental training or fine-tuning of the AI ​​model to improve recognition accuracy. Through reinforcement learning mechanisms, decision-making strategies are optimized based on the success / failure results of task execution. Online model updates are supported; new versions are distributed via an internal network server and automatically replace the old version after verification.

[0050] The AI-based screen recognition-based intelligent automated operation method for cloud phones provided in this embodiment uses computer vision technology to recognize screen content in real time, and combines deep learning models to perform scene understanding and intelligent decision-making, thereby achieving automated operation of cloud phone applications.

[0051] The method of this disclosure will be further described below with reference to a specific embodiment: Experimental environment: Based on Android 13 cloud phone system, configured with Qualcomm Snapdragon processor, and integrating the AI ​​automation system of this invention.

[0052] Test scope: A total of 23 applications of different types, including social networking, e-commerce, and games, were selected for testing, covering typical automated scenarios such as login authentication, content publishing, product search and purchase, and game hosting.

[0053] Test scale: More than 3,000 automated tasks were executed in total, a single application was repeatedly tested 100-200 times, and some scenarios underwent long-term stability testing for more than 8 hours.

[0054] Test results: 1. Functional Success Rate: The overall task completion success rate reached 87.5%, with an interface recognition accuracy rate of 92.3% and an operation execution success rate of 89.6%. This represents an improvement of 19.5 percentage points compared to traditional UI control-based automation solutions (with a success rate of approximately 68%).

[0055] 2. Performance metrics: Screen capture latency is less than 50ms, AI model inference time is 80-120ms for OCR, 100-150ms for object detection, 30-50ms for scene classification, operation response time is less than 100ms, system resource usage is 15-25% CPU and 200-300MB memory, which has virtually no impact on the operation of foreground applications.

[0056] 3. Compatibility: It can adapt well to different screen resolutions (1080p, 2K) and different application versions. It can still work normally without modifying the configuration after the application version is updated.

[0057] 4. Stability: The system runs continuously for more than 8 hours without crashes or memory leaks, and runs stably for more than 30 days without major failures.

[0058] Optimization results: Compared with the initial version of the project, after model optimization and algorithm improvement, the recognition accuracy increased from 78% to 92% (an increase of 14 percentage points), the inference speed increased by 2.3 times, memory usage decreased by 45%, and the overall task success rate increased from 65% to 87% (an increase of 22 percentage points).

[0059] It should be understood that the various parts of this disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0060] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A cloud phone intelligent automated operation method based on AI image recognition, characterized in that, include: Step 1: Receive the task execution request and the corresponding task configuration file, wherein the task configuration file defines an automated process with multiple steps. Step 2: In response to the task execution request, collect the current screen image of the cloud phone in real time based on system service permissions; Step 3: After preprocessing the current screen image, input it into the multimodal AI recognition engine for recognition and analysis, establish the structured description information of the current screen image, and form a list of interactive elements accordingly; Step 4: Based on the goal of the current execution step and the list of interactive elements, make intelligent decisions through a rule engine or decision model to determine the target element to be operated and generate the corresponding simulated user operation instructions. Step 5: Execute simulated user operation commands through system input event injection; Step 6: After executing the operation command, capture and recognize the screen image again, verify whether the operation was successful based on the changes in the interface before and after the operation, and update the task execution status according to the verification result.

2. The method according to claim 1, characterized in that, Step 2 specifically includes: Step 2.1: Access the screen frame buffer directly through the SurfaceFlinger mechanism in the Android system service to collect raw frame data; Step 2.2: Dynamically adjust the image capture frequency and capture area according to the current task type; Step 2.3: Perform format conversion and size normalization on the original frame data to obtain the current screen image.

3. The method according to claim 2, characterized in that, Step 3 specifically includes: Step 3.1: After preprocessing the current screen image, input it into the scene classification model to determine the current interface type; Step 3.2: Based on the current interface type, use the OCR engine, object detection model and icon matching algorithm to identify the text content and position, UI controls and bounding boxes, and specific icons in the image. Step 3.3: Integrate the recognition results of the OCR engine, object detection model and icon matching algorithm to generate structured description information containing element type, text content, coordinate position and confidence information, and form an interactive element list accordingly.

4. The method according to claim 3, characterized in that, Step 4 specifically includes: Step 4.1: Based on the target element features defined in the current step, locate the target element in the list of interactive elements using exact matching, fuzzy matching, or combined matching. Step 4.2: Based on the predefined decision rule base or the decision model trained on historical data, determine the type of operation and parameters to be performed in conjunction with the target elements; Step 4.3: Generate specific operation instructions based on the operation type and parameters, and add random offsets to the coordinates in the operation instructions and / or generate curves that simulate human operation for the sliding trajectory.

5. The method according to claim 1, characterized in that, Step 5 specifically includes: Step 5.1: Inject touch events directly into the / dev / input / event device node through the Android system's Input subsystem; Step 5.2: Before performing the operation, check the device screen status, network connection, and application foreground status; Step 5.3: After performing the operation, set a reasonable waiting time according to the operation type, and continuously monitor the changes on the screen during the waiting period.

6. The method according to claim 1, characterized in that, The steps of verifying the success of the operation based on the changes in the interface before and after the operation, and updating the task execution status according to the verification result, include: Step 6.1: If the operation is successful, the task status will be moved to the next sequential step, conditional branch, or loop step according to the definition in the task configuration file. Step 6.2: If the operation fails or the result does not meet expectations, retry or perform error recovery operation according to the predefined exception handling strategy. The exception handling strategy includes delaying retry, closing the unexpected pop-up window, performing a return operation, or restarting the application.

7. The method according to claim 1, characterized in that, Before step 1, the method further includes: The AI ​​automation controller is initialized when the Android system service starts with the system. Load a pre-set AI model file into memory, wherein the AI ​​model file includes a lightweight OCR model, an object detection model, and a scene classification model; Establish a connection channel with the screen capture service.

8. The method according to claim 1, characterized in that, After step 6, the method further includes: Collect screen images that have a confidence level below the threshold or have failed to be identified during task execution; After labeling the screen images, they are added to the training dataset, which is then used to incrementally train or fine-tune the model in the multimodal AI recognition engine. Based on the success and failure feedback of task execution, the strategy in the decision-making model is optimized through a reinforcement learning mechanism.

9. The method according to claim 1, characterized in that, After step 6, the method further includes: During the execution of the detection task, target data is monitored and collected in real time. The target data includes task progress, recognition accuracy, operation success rate, system resource consumption indicators, as well as timestamps, screenshots, recognition results and operation instructions for each step. After the task is completed, an execution report is generated based on the collected data, including total time consumption, step time distribution, success and failure statistics, and exception logs.

Images