Cloud mobile phone automatic operation method and device, electronic equipment and storage medium

By receiving user instructions, recognizing intent, generating operation plans, and adjusting in abnormal situations, the system solves the problem of low efficiency in dynamic planning and error handling in cloud phone automation operations, achieving efficient automation and cross-platform adaptation.

CN120892117APending Publication Date: 2025-11-04BEIJING ZERO ONE EVERYTHING INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202510796374.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-16
Publication Date
2025-11-04

AI Technical Summary

Technical Problem

Existing cloud phone automation technologies suffer from inefficiencies in dynamic planning and error handling, making it difficult to cope with diverse mobile operating system environments and dynamically changing operational needs.

Method used

The cloud phone automation operation method is adopted. It receives user instructions, recognizes user intent, calls the planning model to generate operation plans for multiple sub-tasks, establishes an execution loop and obtains the current screen information, executes actions through the mobile control protocol interface, and triggers dynamic plan adjustment or error self-handling in abnormal situations.

Benefits of technology

It significantly improves the efficiency and versatility of automated operations, enabling it to handle complex tasks and adapt to diverse mobile operating system environments, thereby enhancing operational reliability and adaptability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120892117A_ABST
    Figure CN120892117A_ABST
Patent Text Reader

Abstract

The invention provides an automatic operation method and device for a cloud mobile phone, electronic equipment and a storage medium. The automatic operation method for the cloud mobile phone comprises the following steps: receiving a user instruction, performing user intention identification on the user instruction, calling a planning model, and generating an operation plan comprising a plurality of subtasks according to task elements; establishing an execution loop, wherein the execution loop comprises the steps of obtaining current screen information, inputting the current subtask, the current screen information and a historical execution record into an execution model, and outputting a to-be-executed action and a target position; calling an underlying operation tool through the mobile control protocol interface to execute the to-be-executed action at the target position; when the success standard corresponding to the subtask is met, entering the next subtask; when the execution result is abnormal, dynamic plan adjustment or error self-processing is triggered; when the loop termination condition is met, the operation result is output, and the efficiency and universality of automatic operation are remarkably improved through the technologies of intention recognition, dynamic plan generation, self-reflection, error self-processing and the like.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and mobile device automatic operation, and particularly relates to a cloud mobile phone automatic operation method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the increasing complexity of mobile device operation tasks, the demand for automation operation is growing. Existing mobile device automation operation technologies, such as Mobile-Agent, use a single agent model combined with a visual perception scheme to identify icons on a mobile phone screenshot and perform action reasoning based on the understanding ability of a large language model, thereby achieving automatic operation of a mobile device according to user instructions. However, as the operation history grows, the reasoning ability and robustness of the single agent model in complex tasks are insufficient, and it is difficult to cope with diversified mobile operating system environments and dynamically changing operation requirements. Related technologies propose a multi-agent collaboration framework to handle complex operations. However, related technologies still have problems in dynamic planning adjustment, error self-processing, and the like, which affect the efficiency of automatic operation due to the lack of operation history reflection. SUMMARY

[0003] The present application provides a cloud mobile phone automatic operation method and device, electronic equipment and storage medium to solve the defects of traditional cloud mobile phone automatic operation methods in dynamic planning adjustment, error self-processing, and the like, which affect the efficiency of automatic operation.

[0004] The present application provides a cloud mobile phone automatic operation method, comprising:

[0005] receiving a user instruction, performing user intent recognition on the user instruction, and extracting task elements when recognizing a cloud mobile phone operation task;

[0006] calling a planning model, generating an operation plan containing multiple sub-tasks according to the task elements, and associating each sub-task with a preset success standard;

[0007] establishing an execution cycle to process each sub-task in turn, the execution cycle comprising obtaining current screen information, inputting the current sub-task, the current screen information and the historical execution record into an execution model, and outputting a to-be-executed action and a target position; and calling a bottom operation tool at the target position through a mobile control protocol interface to execute the to-be-executed action;

[0008] when the execution result meets the success standard corresponding to the current sub-task, entering the next sub-task;

[0009] when the execution result is abnormal, triggering dynamic plan adjustment or error self-processing;

[0010] when the loop termination condition is met, outputting an operation result.

[0011] According to the cloud mobile phone automatic operation method provided by the application, the user instruction is subjected to user intention recognition, which comprises:

[0012] The semantic of the user instruction is analyzed by a natural language processing model;

[0013] It is judged whether it belongs to a cloud mobile phone operation task based on a preset task classifier;

[0014] When it is identified as a cloud mobile phone operation task, the task elements are extracted, which include a task subject, an operation object and an expected result.

[0015] According to the cloud mobile phone automatic operation method provided by the application, the planning model is called to generate an operation plan containing multiple subtasks according to the task elements, which comprises:

[0016] The planning model adopts a tree decomposition strategy to decompose the task subject and the operation object into multiple hierarchical subtasks, establishes a time sequence dependency relationship between the subtasks, and sets success criteria for each subtask according to the expected result.

[0017] According to the cloud mobile phone automatic operation method provided by the application, the current screen information is obtained, which comprises:

[0018] The screenshot is obtained through the ADB protocol;

[0019] The interface structured data is extracted by using a visual recognition model.

[0020] According to the cloud mobile phone automatic operation method provided by the application, when the execution result is abnormal, the dynamic plan adjustment is triggered, which comprises:

[0021] When the number of consecutive failures exceeds a preset threshold, the subtask sequence is regenerated based on the execution model.

[0022] According to the cloud mobile phone automatic operation method provided by the application, when the execution result is abnormal, the error self-processing is triggered, which comprises:

[0023] When the click is invalid, the execution model automatically adjusts the target position;

[0024] When it is detected that the interface is not updated, the execution is delayed.

[0025] According to the cloud mobile phone automatic operation method provided by the application, the loop termination condition comprises:

[0026] All subtasks are marked as completed;

[0027] And / or,

[0028] The proportion of abnormal subtasks exceeds a preset threshold;

[0029] and / or,

[0030] The total execution time exceeds a preset time limit.

[0031] The application provides a cloud phone automatic operation device, comprising:

[0032] An identification module is configured to receive a user instruction, perform user intention identification on the user instruction, and extract a task element when the user instruction is identified as a cloud phone operation task.

[0033] A generation module is configured to call a planning model and generate an operation plan comprising a plurality of subtasks according to the task element, each subtask being associated with a preset success criterion.

[0034] A processing module is configured to establish an execution loop and sequentially process each subtask, the execution loop comprising obtaining current screen information, inputting the current subtask, the current screen information, and a historical execution record into an execution model, and outputting a to-be-executed action and a target position; calling a bottom operation tool through a mobile control protocol interface to execute the to-be-executed action at the target position; when an execution result meets a success criterion corresponding to the current subtask, entering a next subtask; and when the execution result is abnormal, triggering dynamic plan adjustment or error self-processing.

[0035] A termination module is configured to output an operation result when a loop termination condition is met.

[0036] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the cloud phone automatic operation method according to any one of the above when executing the program.

[0037] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the cloud phone automatic operation method according to any one of the above.

[0038] The cloud mobile phone automatic operation method, device, electronic equipment and storage medium provided by the application, through receiving a user instruction, performing user intention recognition on the user instruction, when recognizing a cloud mobile phone operation task, extracting task elements; calling a planning model, generating an operation plan containing multiple sub-tasks according to the task elements, each sub-task being associated with a preset success standard; establishing an execution cycle to process each sub-task in turn, the execution cycle including obtaining current screen information, inputting the current sub-task, the current screen information and the historical execution record into an execution model to output a to-be-executed action and a target position; calling a bottom operation tool through a mobile control protocol interface to execute the to-be-executed action at the target position; when the execution result meets the success standard corresponding to the current sub-task, entering the next sub-task; when the execution result is abnormal, triggering dynamic plan adjustment or error self-processing; when the loop termination condition is met, outputting the operation result, through intention recognition, dynamic plan generation, self-reflection and error self-processing, the efficiency and universality of automatic operation are significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0040] Figure 1 is one of the flowcharts of the cloud mobile phone automatic operation method provided by the application;

[0041] Figure 2 is the second flowchart of the cloud mobile phone automatic operation method provided by the application;

[0042] Figure 3 is one of the structural diagrams of the cloud mobile phone automatic operation device provided by the application;

[0043] Figure 4 is the architectural diagram of the cloud mobile phone automatic operation system driven by the MCP through the large model provided by the application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the application more clear, the technical scheme in the application will be described clearly and completely in the following combined with the drawings in the application. Obviously, the described embodiments are some embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0045] Figure 1 The flowchart of the cloud mobile phone automatic operation method provided by the embodiment of the present application is shown in Figure 1 The cloud mobile phone automatic operation method provided by the embodiment of the present application comprises the following steps:

[0046] Step 101, receiving a user instruction, performing user intention recognition on the user instruction, and extracting task elements when recognizing a cloud mobile phone operation task;

[0047] Step 102, calling a planning model, generating an operation plan containing multiple sub-tasks according to the task elements, and associating each sub-task with a preset success standard;

[0048] Step 103, establishing an execution cycle, sequentially processing each sub-task, the execution cycle comprising obtaining current screen information, inputting the current sub-task, the current screen information and the historical execution record into an execution model, and outputting a to-be-executed action and a target position; and calling a bottom operation tool at the target position through a mobile control protocol (MCP) interface to execute the to-be-executed action;

[0049] In the embodiment of the present application, a standardized mobile control protocol (MCP) tool calling interface is provided, allowing third-party developers to develop customized operation modules based on the protocol, forming a unified cloud mobile phone operation development ecosystem, supporting pluggable integration of tool plug-ins, opening an MCP tool set through an API, allowing other systems to directly reuse the bottom operation capability of the present solution, reducing the overall development cost of the industry, avoiding repeated implementation of basic operation functions, and decoupling the design of a large model and an MCP tool, allowing different AI models to call the same set of operation interfaces, maintaining operation consistency, independently deploying an MCP tool service, and adapting different manufacturers' cloud mobile phones to implement MCP through the same set of operation instructions, extending operation nodes as needed, and realizing distributed processing of computing resources and operation execution.

[0050] Step 104, when the execution result meets the success standard corresponding to the current sub-task, entering the next sub-task;

[0051] Step 105, when the execution result is abnormal, triggering dynamic plan adjustment or error self-processing;

[0052] For example: Help me post a photo I took today in WeChat Moments, with a mood text, initial planning: open WeChat → enter Moments → click publish → select photo → add text → publish, dynamic adjustment process: permission exception adjustment: find that the album permission is not turned on, insert the "turn on album permission" step.

[0053] Step 106, outputting an operation result when a loop termination condition is met.

[0054] In the embodiments of the present application, the cycle termination condition includes but is not limited to: all sub-tasks are marked as completed state; the proportion of abnormal sub-tasks exceeds a preset threshold; the total execution time exceeds a preset time limit, etc. The output result includes but is not limited to: task completion state mark, screen snapshot sequence of key operation node, operation instruction log containing timestamp, and scripted execution track that can be imported into test system, etc.

[0055] The conventional single-agent model combined with visual perception scheme identifies icons on the mobile phone screenshot and performs action reasoning based on the understanding ability of the large language model to realize automatic operation of the mobile device according to the user instruction. However, as the operation history grows, the single-agent model lacks reasoning ability and robustness in complex tasks, making it difficult to cope with diversified mobile operating system environments and dynamically changing operation requirements. Related technologies propose a multi-agent collaboration framework to handle complex operations. However, related technologies still have problems in dynamic planning adjustment, error self-processing, etc. due to the lack of operation history reflection, affecting the efficiency of automated operation.

[0056] The cloud phone automated operation method provided by the embodiments of the present application receives a user instruction, identifies the user intention of the user instruction, extracts task elements when identifying a cloud phone operation task, calls a planning model, generates an operation plan containing multiple sub-tasks according to the task elements, each sub-task is associated with a preset success standard, establishes an execution cycle to process each sub-task in turn, the execution cycle includes obtaining current screen information, inputting the current sub-task, current screen information and historical execution record into an execution model to output a to-be-executed action and a target position, calling a bottom operation tool at the target position through a mobile control protocol interface to execute the to-be-executed action, entering the next sub-task when the execution result meets the success standard of the current sub-task, triggering dynamic plan adjustment or error self-processing when the execution result is abnormal, and outputting an operation result when the cycle termination condition is met. Through the technologies of intention identification, dynamic plan generation, self-reflection and error self-processing, the efficiency and versatility of automated operation are significantly improved.

[0057] Based on any of the above embodiments, the user intention of the user instruction is identified, including:

[0058] Step 201, parsing the semantics of the user instruction through a natural language processing model;

[0059] Step 202, determining whether it belongs to a cloud phone operation task based on a preset task classifier;

[0060] Step 203, when identifying a cloud phone operation task, extracting task elements, the task elements including task subject, operation object and expected result.

[0061] In the embodiment of the application, the natural language processing model (such as BERT / GPT) is used to deeply analyze the instruction semantics, complex sentence patterns can be processed, and the semantic understanding accuracy is improved compared with the traditional keyword matching method.

[0062] According to any of the above embodiments, the calling planning model generates an operation plan containing multiple sub-tasks according to the task elements, which includes:

[0063] The planning model adopts a tree decomposition strategy to decompose the task subject and operation object into multiple hierarchical sub-tasks, establishes the time sequence dependency relationship between the sub-tasks, and sets success criteria for each sub-task according to the expected result.

[0064] In the embodiment of the application, the main task is hierarchically decomposed into atomic operations using a tree structure, which can be distributedly executed, thereby improving the reliability and adaptability of automatic operation.

[0065] According to any of the above embodiments, the obtaining of the current screen information includes:

[0066] The screenshot is obtained through the ADB (Android Debug Bridge) protocol;

[0067] The structured data of the interface is extracted using a visual recognition model.

[0068] The embodiments of the application combine general and special visual models to achieve accurate screen element recognition and coordinate positioning, eliminate the dependence on XML files or system metadata, and enhance cross-platform adaptability.

[0069] According to any of the above embodiments, when the execution result is abnormal, triggering dynamic plan adjustment includes:

[0070] When the number of consecutive failures exceeds a preset threshold, the sub-task sequence is regenerated based on the execution model.

[0071] The embodiments of the application realize context-aware task reconstruction through a dynamic plan adjustment mechanism, thereby improving the adaptive ability of the system: when it is detected that the number of consecutive failures exceeds a preset threshold (such as 3 times), the execution model dynamically reconstructs the sub-task sequence based on the operation history and the environment state (such as permission pop-up, network delay, etc.), only adjusts the abnormal task branch instead of restarting the whole process, and realizes millisecond-level fault self-healing.

[0072] According to any of the above embodiments, when the execution result is abnormal, triggering error self-processing includes:

[0073] When the click is invalid, the execution model automatically adjusts the target position;

[0074] When it is detected that the interface is not updated, the execution is delayed.

[0075] The embodiment of the application realizes intelligent real-time error correction through an error self-processing mechanism, thereby improving operation reliability: when a click failure is detected, the execution model automatically calculates the coordinate offset of the target position based on visual feedback to perform dynamic calibration; when the interface is not updated, the system dynamically adjusts the delay time through a heartbeat detection mechanism, effectively solving the misjudgment problem caused by network delay or rendering lag. The two-dimensional self-processing of spatial calibration and time optimization reduces the task interruption rate of the system in complex network environments and diverse device adaptation scenarios.

[0076] The embodiment of the application effectively handles operation errors through a dynamic planning adjustment and self-reflection mechanism, thereby improving task success rate.

[0077] Based on any of the above embodiments, the cloud phone automatic operation method provided by the embodiment of the application specifically includes the following steps.

[0078] (1) User intention recognition: receiving user input instructions, identifying user intentions based on a large model; if the intention is to operate the cloud phone, proceed to the subsequent steps; otherwise, prompt the user to modify or confirm the instructions through human intervention (Human-in-the-Loop).

[0079] (2) Plan generation: calling a planning model (Planning Model) responsible for overall task planning, generating a detailed operation plan (Plan) according to the user intention, including sub-tasks in stages (Stage1, Stage2,...).

[0080] (3) Execution cycle initialization: using the generated plan as the prompt word of the execution model (Execute Model) to initialize the execution cycle.

[0081] In the embodiment of the application, the planning model is gpt-4o or claude, and the execution model is a model with visual capabilities such as gpt-4o, claude3.7, etc.

[0082] (4) Screen information acquisition: acquiring cloud phone screenshots or structured data thereof as context input of the execution model for decision-making actions (Action).

[0083] (5) Action decision and execution: the execution model decides the next action based on the plan, screen information, and operation history, and executes it through the MCP calling corresponding tools; the action space includes opening App, clicking text, clicking icons, typing, scrolling up, scrolling down, returning to the previous page, exiting App, stopping, etc.

[0084] (6) Action Evaluation: Evaluate the execution result of the previous action, determine success, failure or invalid; if failure or invalid, record error type and trigger self-reflection. According to the current screenshot, ui information determines whether the previous action is successful;

[0085] (7) Dynamic Plan Adjustment: If the execution model detects that the plan is unreasonable (such as continuous failure or deviation from the target), call the planning model to regenerate the plan, try other solutions during task execution, and error self-processing: the execution model autonomously processes errors based on the common sense library, such as returning, retrying or adjusting parameters.

[0086] (8) Progress Update: After each action, generate a description of the current task progress and the next step plan.

[0087] (9) Termination Condition Check: Check if the termination condition is met, including task completion, continuous failure times reaching the threshold or operation steps reaching the maximum threshold, user actively stopping, if met, end the task and output the final result.

[0088] For example, (1) User Intention Recognition: Analyze the user's input instructions through a large model (such as GPT-4o), such as "send a message to contact A in WeChat". If it is identified as a cloud phone operation task, proceed to the next step; otherwise, prompt the user to modify the instructions through the interactive interface.

[0089] (2) Plan Generation: Call the planning model to generate the operation plan, for example:

[0090] Stage1: Open WeChat App.

[0091] Stage2: Navigate to the contact list and click on contact A.

[0092] Stage3: Input message content and send.

[0093] (3) Execution Loop Initialization: Use the plan as the execution model's prompt word to initialize the operation loop.

[0094] (4) Screen Information Acquisition: Obtain cloud phone screenshots through ADB, and identify interface elements (such as "WeChat" icon, contact A's text label) and their coordinates through ui-tars Qwen-VL-plus model.

[0095] (5) Action Decision and Execution: The execution model makes decisions based on the plan and screen information (such as "click WeChat icon"), and executes through MCP calling ADB keyboard or click tools.

[0096] (6) Action evaluation: The reflection model compares screenshots before and after operation to determine whether the action is successful. For example, if the "WeChat" icon is clicked but does not enter the WeChat interface, it is marked as a failure.

[0097] (7) Dynamic plan adjustment: If the action fails for 3 consecutive times, call the planning model to regenerate the plan, such as adjusting to "check if WeChat is installed". Error self-processing: If invalid operation is detected (such as clicking on a blank area), the model automatically retries or adjusts the click coordinates.

[0098] (8) Progress update: Generate progress description, such as "WeChat has been opened, next click on contact A".

[0099] (9) Termination condition check: If the task is completed (such as message sending success) or the maximum number of steps (such as 50 steps) is reached, end the task and generate a report.

[0100] The present application realizes cloud phone automation operation through large model driven MCP, combines multi-agent collaboration, visual perception and self-reflection mechanism, significantly improves operation efficiency and cross-platform adaptability. Through multi-agent collaboration (planning, decision-making, reflection), the decomposition and execution ability of complex tasks is significantly improved, overcoming the reasoning limitations of single agent. Combined with general and special visual models, accurate screen element recognition and coordinate positioning are realized, breaking the dependence on XML files or system metadata, and enhancing cross-platform adaptability. Dynamic plan adjustment and self-reflection mechanism effectively handle operation errors and improve task success rate. Through MCP protocol encapsulation tool, the operation calling process is simplified, and the execution efficiency is improved. The visual interface displays the operation process in real time, enhancing user interaction experience and debugging efficiency.

[0101] Compared with the prior art, the present application has significant advantages in dynamic plan adjustment, error self-processing and visual interaction, and is suitable for various cloud phone automation scenarios. Moreover, based on the DroidRun open source framework, through the technologies of intent recognition, dynamic plan generation, self-reflection and error self-processing, the efficiency and universality of automation operation are significantly improved.

[0102] The cloud phone automation operation device provided by the present application is described below. The cloud phone automation operation device described below can be correspondingly referred to the cloud phone automation operation method described above.

[0103] Figure 3 The schematic diagram of the cloud phone automation operation device provided by the present application is shown in Figure 3 The cloud phone automation operation device provided by the present application includes:

[0104] The recognition module 301 is configured to receive a user instruction, perform user intention recognition on the user instruction, and extract a task element when the user instruction is identified as a cloud mobile phone operation task.

[0105] The generation module 302 is configured to call a planning model and generate an operation plan containing a plurality of subtasks according to the task element, wherein each subtask is associated with a preset success criterion.

[0106] The processing module 303 is configured to establish an execution loop to sequentially process each subtask, wherein the execution loop comprises obtaining current screen information, inputting the current subtask, the current screen information and a historical execution record into an execution model to output a to-be-executed action and a target position, calling a bottom operation tool through a mobile control protocol interface to execute the to-be-executed action at the target position, entering a next subtask when an execution result meets a success criterion corresponding to the current subtask, and triggering dynamic plan adjustment or error self-processing when the execution result is abnormal.

[0107] The termination module 304 is configured to output an operation result when a loop termination condition is met.

[0108] The embodiment of the application also provides a cloud mobile phone automatic operation system driven by a large model and implemented through MCP, as shown in the figure. Figure 4 The system comprises the following modules.

[0109] The intention recognition module is configured to analyze a user instruction based on a large model and output an intention classification result.

[0110] The visual perception module is configured to analyze a screenshot in combination with GPT-4o and ui-tars Qwen-VL-plus and output structured elements and coordinates.

[0111] The planning agent module is configured to generate a staged plan and store the plan in a memory unit.

[0112] The decision agent module is configured to make a decision on an action based on a ReAct framework in combination with a plan and screen information and call an MCP interface.

[0113] The reflection agent module is configured to evaluate an action result and trigger plan adjustment or error processing.

[0114] The error processing module is configured to support return and retry operations in combination with a built-in common sense library.

[0115] The progress management module is configured to record operation history and generate a progress description.

[0116] The MCP interface module is configured to encapsulate an ADB tool through an MCP protocol and support action execution.

[0117] The visual interface module is configured to display a mobile phone screen, operation logs and tool calling details in real time.

[0118] In the embodiment of the application, the cloud mobile phone is connected through the ADB, and the USB debugging mode is started. The ADB keyboard is installed, and the default input method is set as the ADB Keyboard. The MCP server is configured, and the operation tool interface (such as clicking and inputting) is exposed.

[0119] The cloud mobile phone automatic operation device provided by the embodiment of the application receives a user instruction, identifies the user intention of the user instruction, extracts a task element when the user instruction is identified as a cloud mobile phone operation task, calls a planning model, generates an operation plan containing multiple subtasks according to the task element, each subtask is associated with a preset success standard, establishes an execution cycle, processes each subtask in turn, the execution cycle includes obtaining current screen information, inputting the current subtask, the current screen information and the historical execution record into an execution model, and outputting a to-be-executed action and a target position, calling a bottom operation tool in the target position through a mobile control protocol interface to execute the to-be-executed action, entering a next subtask when an execution result meets the success standard corresponding to the current subtask, triggering dynamic plan adjustment or error self-processing when the execution result is abnormal, and outputting an operation result when a loop termination condition is met.

[0120] The embodiment of the application further provides an electronic device, which can include a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus. The processor can call a logical instruction in the memory to execute a cloud mobile phone automatic operation method, which includes receiving a user instruction, identifying the user intention of the user instruction, extracting a task element when the user instruction is identified as a cloud mobile phone operation task, calling a planning model, generating an operation plan containing multiple subtasks according to the task element, each subtask being associated with a preset success standard, establishing an execution cycle, processing each subtask in turn, the execution cycle including obtaining current screen information, inputting the current subtask, the current screen information and the historical execution record into an execution model, and outputting a to-be-executed action and a target position, calling a bottom operation tool in the target position through a mobile control protocol interface to execute the to-be-executed action, entering a next subtask when an execution result meets the success standard corresponding to the current subtask, triggering dynamic plan adjustment or error self-processing when the execution result is abnormal, and outputting an operation result when a loop termination condition is met.

[0121] In another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements a cloud phone automatic operation method provided by each of the above methods, the method comprising: receiving a user instruction, performing user intention recognition on the user instruction, and extracting a task element when the user instruction is identified as a cloud phone operation task; calling a planning model to generate an operation plan comprising a plurality of subtasks according to the task element, each subtask being associated with a preset success criterion; establishing an execution loop to process each subtask in turn, the execution loop comprising obtaining current screen information, inputting the current subtask, the current screen information and a historical execution record into an execution model, and outputting a to-be-executed action and a target position; calling a bottom operation tool through a mobile control protocol interface to execute the to-be-executed action at the target position; when the execution result meets the success criterion corresponding to the current subtask, entering the next subtask; when the execution result is abnormal, triggering dynamic plan adjustment or error self-processing; and when a loop termination condition is met, outputting an operation result.

[0122] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0123] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for automating the operation of a cloud phone, characterized in that, include: Receive user instructions, perform user intent recognition on the user instructions, and extract task elements when the user instructions are recognized as cloud phone operation tasks. The planning model is invoked to generate an operation plan containing multiple sub-tasks based on the task elements, with each sub-task associated with a preset success criterion. An execution loop is established to process each subtask sequentially. The execution loop includes obtaining the current screen information, inputting the current subtask, the current screen information, and historical execution records into the execution model, and outputting the action to be executed and the target position. The underlying operation tool is invoked via the mobile control protocol interface to execute the action to be performed at the target location. When the execution result meets the success criteria of the current subtask, proceed to the next subtask; When the execution result is abnormal, dynamic plan adjustment or error self-handling is triggered; Output the operation result when the loop termination condition is met.

2. The automated operation method for cloud phones according to claim 1, characterized in that, The user intent recognition of the user instruction includes: The semantics of the user's instructions are parsed using a natural language processing model; Determine whether a task belongs to a cloud phone operation task based on a preset task classifier; When a task is identified as a cloud phone operation task, task elements are extracted, including the task subject, the operation object, and the expected result.

3. The automated operation method for cloud phones according to claim 2, characterized in that, The call planning model generates an operation plan containing multiple sub-tasks based on the task elements, including: The planning model employs a tree-like decomposition strategy to break down the task subject and operation object into multiple hierarchical subtasks, establishing temporal dependencies between subtasks; and sets success criteria for each subtask based on the expected results.

4. The automated operation method for cloud phones according to claim 1, characterized in that, The acquisition of current screen information includes: Capture screenshots using the ADB protocol; Use a visual recognition model to extract structured data from the interface.

5. The automated operation method for cloud phones according to claim 1, characterized in that, The triggering of dynamic plan adjustment when the execution result is abnormal includes: When the number of consecutive failures exceeds a preset threshold, the subtask sequence is regenerated based on the execution model.

6. The automated operation method for cloud phones according to claim 1, characterized in that, When the execution result is abnormal, the error self-handling mechanism is triggered, including: When the click fails, the execution model automatically adjusts the target position; Execution is delayed if the interface is not updated.

7. The automated operation method for cloud phones according to claim 1, characterized in that, The loop termination condition includes: All subtasks are marked as completed; And / or, The percentage of abnormal subtasks exceeds a preset threshold; And / or, The total execution time exceeded the preset time limit.

8. An automated operation device for cloud phones, characterized in that, include: The recognition module is used to receive user instructions, perform user intent recognition on the user instructions, and extract task elements when the user instructions are recognized as cloud phone operation tasks. The generation module is used to call the planning model and generate an operation plan containing multiple sub-tasks based on the task elements. Each sub-task is associated with a preset success criterion. The processing module is used to establish an execution loop to process each subtask in sequence. The execution loop includes obtaining the current screen information, inputting the current subtask, the current screen information and historical execution records into the execution model, and outputting the action to be executed and the target position. The underlying operation tool is invoked through the motion control protocol interface to execute the action to be performed at the target location; when the execution result meets the success criteria of the current subtask, the next subtask is entered; When the execution result is abnormal, dynamic plan adjustment or error self-handling is triggered; The termination module is used to output the operation result when the loop termination condition is met.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the cloud phone automated operation method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cloud phone automated operation method as described in any one of claims 1 to 7.

Citation Information

Cited By

  • AI SRE-oriented operation and maintenance right management and control method and system

    CN122226525A

  • Operation and maintenance access control methods and systems for AI SRE

    CN122226525B