Mobile phone automatic operation method, device and equipment and computer storage medium

Abstract

The invention relates to the technical field of large model application, in particular to an automatic mobile phone operation method, device and equipment based on a multi-modal agent and a computer storage medium. According to the automatic operation method of the mobile phone, the task disassembly and execution process is optimized by introducing the planning agent, the memory unit and the reflection agent, and the multi-agent cooperation architecture is beneficial to improving the task execution efficiency and reducing the processing time; a clarification agent is introduced, unnecessary interaction is greatly reduced through a probability prediction mode, a user can insert a decision-making process of an agent framework as little as possible, meanwhile, the agent can timely inquire the user when encountering problems which cannot be solved, and it is ensured that tasks can be smoothly executed; according to the invention, the user interaction performance of mobile phone operation is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of large model application technology, and in particular to a mobile phone automatic operation method, device, equipment and computer storage medium based on a multimodal intelligent agent. Background Art

[0002] With the rapid development of artificial intelligence technology, large-scale pre-trained models have achieved remarkable results in natural language processing, image recognition and many other fields. These models extract knowledge from massive data through deep learning technology and have powerful understanding and generation capabilities. However, the application of a single model in complex environments still has limitations.

[0003] To solve this problem, multi-agent systems came into being. Multi-agent systems involve multiple agents working together to complete tasks, and each agent can share information, allocate tasks, and make decisions in specific scenarios. This structure not only improves the flexibility and robustness of the system, but also enables agents to perform operations more effectively in dynamic and uncertain environments.

[0004] In terms of mobile phone operation, mobile phone operation programs based on the multi-agent framework have already performed very well. They perform well overall, but due to the long processing time and the inability to interact well with users, they do not perform well when processing long-process instructions, and when multiple choices are involved, they can only make random choices and cannot ask users in a timely manner. Summary of the invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem of poor user interaction in the prior art.

[0006] In order to solve the above technical problems, the present invention provides a mobile phone automatic operation method, comprising:

[0007] After receiving the user's command information, the screen perception information and interactive dialogue information of the mobile phone interface in the tth round are obtained;

[0008] Input the user instruction information, the summary of historical actions and historical action descriptions of the tth round, the screen perception information, the actions performed in the t-1th round, the action descriptions, and the action reflections into the big model, update the summary of historical actions and historical action descriptions of the tth round, and output the action plan of the tth round;

[0009] Input the user instruction information, the updated summary of historical actions and historical action descriptions of the tth round, the screen perception information and the action plan of the tth round into the big model, obtain the actions and action descriptions executed in the tth round from the action library, and call the action function to operate the mobile phone;

[0010] Determine whether the command task is completed. If not, proceed to the next step. If completed, end the process.

[0011] The user command information, the updated summary of historical actions and historical action descriptions of the tth round, the screen perception information of the tth round and the t+1th round, the action plan of the tth round, the actions performed in the tth round and the action descriptions are input into the big model, the action reflection of the tth round is output, and the subsequent work of this round and the next round is guided according to the action reflection of the tth round.

[0012] Preferably, the step of obtaining screen perception information of the mobile phone interface in the tth round includes:

[0013] Get the mobile phone screenshots of round t;

[0014] Performing image recognition and text recognition on the mobile phone screenshot to obtain multiple icons and corresponding texts under the icons;

[0015] The multiple icons and the corresponding texts under the icons are input into the large model to obtain the functional description of each icon, and analyze whether the keyboard of the mobile phone is awakened and whether each icon is clicked to obtain the perception information of the entire screen.

[0016] Preferably, the action library includes multiple basic operations and multiple complex operations, the basic operations include clicking, sliding, long pressing, typing, returning to the previous level, returning to the main interface, opening the program, stopping and communicating with the user, and the complex operations are composed of multiple basic operations encapsulated by uiautomator2.

[0017] Preferably, guiding the subsequent work of the current round and the next round based on the action reflection of the tth round includes:

[0018] If the action reflection is that the current action is correctly executed and the mobile phone interface changes, the next round of action planning, action decision-making and action reflection is performed;

[0019] If the action reflection is that the current action is executed correctly and the mobile phone interface has not changed, it will prompt that there is no change and suggest the user to adjust the click strategy or check the device status;

[0020] If the action reflection is that the current action is executed incorrectly, then return to re-plan the action or re-determine the action or skip the action;

[0021] If the action reflection shows that there is a problem with the plan of the current action, the problem is analyzed and the process returns to re-plan the action or re-decide the action.

[0022] Preferably, the updating of the summary of the historical actions and historical action descriptions of the tth round includes:

[0023] Get the summary of historical actions and historical action descriptions of the tth round, that is, the action sequence and corresponding action descriptions that have been performed before the tth round;

[0024] The action sequence and the corresponding action description are updated according to the action reflection of the t-1th round.

[0025] Preferably, after outputting the action plan for the tth round, the step further includes:

[0026] Determine whether clarification is needed:

[0027] If there are preset problems in the interactive dialogue information, the next round of action planning, action decision-making and action reflection will be started directly. The preset problems include the lack of corresponding parameters in the user instruction and the semantic ambiguity in the user instruction;

[0028] If a situation that requires a selection appears on the current interface, the user instruction information, the updated summary of historical actions and historical action descriptions for the tth round, the screen perception information, and the action plan for the tth round are input into the large model to generate multiple options, and the probability of each option is predicted. The options that meet the preset conditions are sent to the user, and the next round of action planning, action decision-making, and action reflection are started directly.

[0029] Preferably, the preset conditions include:

[0030] If the probability that only one option exists exceeds a first preset threshold, the option is sent to the user as an answer;

[0031] If the probability of two or more options existing exceeds a first preset threshold, options not lower than a second preset threshold are sent to the user for selection, wherein the sum of the first preset threshold and the second preset threshold is 1.

[0032] The present invention also provides a mobile phone automatic operation device, comprising:

[0033] The perception agent module is used to obtain the screen perception information and interactive dialogue information of the mobile phone interface in the tth round after receiving the user's command information;

[0034] A planning agent module, used to input the user instruction information, the summary of historical actions and historical action descriptions of the tth round, the screen perception information, the actions performed in the t-1th round, the action descriptions and the action reflections into the big model, update the summary of historical actions and historical action descriptions of the tth round and output the action plan of the tth round;

[0035] A decision agent module, used to input the user instruction information, the updated summary of historical actions and historical action descriptions of the tth round, the screen perception information and the action plan of the tth round into the big model, obtain the actions and action descriptions performed in the tth round from the action library, and call the action function to operate the mobile phone;

[0036] The loop judgment module is used to judge whether the current instruction task is completed. If not, it will continue to the next step. If it is completed, it will end the process;

[0037] The reflective agent module is used to input the user instruction information, the updated summary of the historical actions and historical action descriptions of the tth round, the screen perception information of the tth round and the t+1th round, the action plan of the tth round, the actions and action descriptions performed in the tth round into the large model, output the action reflection of the tth round, and guide the subsequent work of this round and the next round based on the action reflection of the tth round.

[0038] The present invention also provides a mobile phone automatic operation device, comprising:

[0039] Memory for storing computer programs;

[0040] The processor is used to implement the above-mentioned steps of the mobile phone automatic operation method when executing the computer program.

[0041] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned mobile phone automatic operation method are implemented.

[0042] The above technical solution of the present invention has the following advantages compared with the prior art:

[0043] The mobile phone automatic operation method described in the present invention optimizes the task decomposition and execution process by introducing a planning agent, a memory unit and a reflective agent. This multi-agent collaborative architecture helps to improve the efficiency of task execution and reduce processing time. The present invention also introduces a clarification agent and greatly reduces unnecessary interactions through a predictive probability model, allowing users to intervene in the decision-making process of the agent framework as little as possible. At the same time, when the agent encounters a problem that cannot be solved, it can promptly ask the user to ensure that the task can be executed smoothly. The present invention greatly improves the user interaction performance of mobile phone operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0045] Figure 1This is a flow chart of an implementation of a mobile phone automatic operation method provided by the present invention;

[0046] Figure 2 The present invention is a flowchart of a method for automatically operating a mobile phone provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The core of the present invention is to provide a method, device, equipment and computer storage medium for automatic operation of a mobile phone, which effectively improves the user interaction performance of mobile phone operation.

[0048] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0049] Please refer to Figure 1 , Figure 1 This is a flowchart of an automatic operation method for a mobile phone provided by the present invention; the specific operation steps are as follows:

[0050] S101: after receiving the user instruction information, obtaining the screen perception information and interactive dialogue information of the mobile phone interface in the tth round;

[0051] S102: Input the user instruction information, the summary of historical actions and historical action descriptions of the tth round, the screen perception information, the actions performed in the t-1th round, the action descriptions, and the action reflections into the big model, update the summary of historical actions and historical action descriptions of the tth round, and output the action plan of the tth round;

[0052] S103: input the user instruction information, the updated summary of historical actions and historical action descriptions of the tth round, the screen perception information and the action plan of the tth round into the big model, obtain the actions and action descriptions executed in the tth round from the action library, and call the action function to operate the mobile phone;

[0053] S104: Determine whether the current instruction task is completed. If not, proceed to step S105. If completed, end the task;

[0054] S105: Input the user instruction information, the updated summary of historical actions and historical action descriptions of the tth round, the screen perception information of the tth round and the t+1th round, the action plan of the tth round, the actions and action descriptions performed in the tth round into the big model, output the action reflection of the tth round, and guide the subsequent work of this round and the next round based on the action reflection of the tth round.

[0055] Based on the above embodiments, this embodiment describes step S101 in detail:

[0056] In a specific embodiment, obtaining screen perception information of the mobile phone interface in the tth round includes:

[0057] Get the screenshot of the mobile phone in round t; that is, the perception agent first takes a screenshot of the mobile phone and obtains a picture PI t

[0058] Perform image recognition and text recognition on the mobile phone screenshot to obtain multiple icons and corresponding texts under the icons; that is, analyze the mobile phone screenshot through image recognition and text recognition to obtain the corresponding icon IC t and the text A under the icon t

[0059] The multiple icons and the corresponding texts under the icons are input into the large model to obtain the functional description of each icon, and analyze whether the keyboard of the mobile phone is awakened and whether each icon is clicked to obtain the perception information of the entire screen. t and the text A t Pack it and transfer it to the big model, let the big model describe it, and get the functional description of each icon on the mobile phone screen. t At the same time, it will also analyze whether the phone's keyboard is awakened, whether the icon can be clicked, etc. The perception information of the entire screen is encapsulated into a perception packet PA t .

[0060] After obtaining these, if the current interface is with the user, the perceptual agent will also analyze and extract the instructions IN given by the current user t , or interactive dialogue DI t And store it.

[0061] DI t ,I,PA t =perceive(IC t ,A t ,D t )

[0062] Based on the above embodiments, this embodiment describes step S102 in detail:

[0063] The planning agent gets the screen information PA analyzed by the perception agent t , instruction information I and dialogue information DI t .

[0064] The user command information, the summary of the historical actions and historical action descriptions in the tth round, the screen perception information, the actions performed in the t-1th round, the action descriptions, and the action reflections are input into the big model; that is, a series of information is input into the big model in the form of prompt words, which includes the command information I, the summary of the historical actions and historical action descriptions in the tth round (the summary of the past actions SO t , the summary of past action descriptions SOD t ), the above two items will be included in the memory information module M t In the above example, the screen information perception package PA obtained by the perception agent is t , the last action performed AC t and the description of the action ACD t , and the return result R of the reflective agent t-1 .

[0065] Update the summary of the historical actions and historical action descriptions of the tth round and output the action plan of the tth round; that is, the big model will return a summary of all previous actions after thinking. t , and the plan for the next action PL t .

[0066] COM t ,PL t =plan(M t ,PA t ,AC t ,ACD t ,R t-1 )

[0067] Based on the above embodiments, the memory information module mainly stores the current instruction, the action sequence that has been performed for the current instruction, and the description of these actions. The action sequence will be affected by the reflective agent. If the current action is wrong, the current action will be deleted and replaced by the next action to be executed. After each instruction is completed, the impact of the current instruction on the mobile phone will also be recorded, which will have an impact and guiding role on the execution of subsequent instructions. Therefore, in a specific embodiment, the update of the summary of the historical actions and historical action descriptions of the tth round includes:

[0068] Get the summary of historical actions and historical action descriptions of the tth round, that is, the action sequence and corresponding action descriptions that have been performed before the tth round;

[0069] The action sequence and the corresponding action description are updated according to the action reflection of the t-1th round.

[0070] Based on the above embodiment, after outputting the action plan for the tth round, the method further includes:

[0071] like Figure 2 As shown, a clarification agent is introduced to judge whether clarification is needed:

[0072] The clarification agent will be awakened by the decision agent when necessary. There are two situations in which the decision agent will be awakened:

[0073] If there are preset problems in the interactive dialogue information, the next round of action planning, action decision-making and action reflection will be started directly. The preset problems include that the user instructions lack corresponding parameters and that the user instructions are semantically ambiguous; that is, first, when interacting with the user, there are some problems in the dialogue given by the user. The problems have the following two aspects: when the user needs to complete some instructions but lacks corresponding parameters; the user's instructions themselves are semantically ambiguous.

[0074] If a situation that requires a selection appears on the current interface, the user instruction information, the updated summary of historical actions and historical action descriptions for the tth round, the screen perception information, and the action plan for the tth round are input into the large model to generate multiple options, and the probability of each option is predicted. The options that meet the preset conditions are sent to the user, and the next round of action planning, action decision-making, and action reflection are started directly.

[0075] C t =clarify(I,OP t )

[0076] The preset conditions include:

[0077] If the probability that only one option exists exceeds a first preset threshold, the option is sent to the user as an answer;

[0078] If the probability of two or more options existing exceeds a first preset threshold, options not lower than a second preset threshold are sent to the user for selection, wherein the sum of the first preset threshold and the second preset threshold is 1.

[0079] In a specific embodiment, when multiple options OP appear, the user needs to make a choice. In this part, we use the parameter logprobs of the large model to output the prediction of the options, and use the softmax function to convert the obtained logarithmic probability into a normal probability. If the option with the highest probability exceeds 92.8%, then we think it is not necessary to bother the user to make a choice, and the agent can directly select the option as the final answer. However, if the option does not have an overwhelming advantage, we will list the options greater than 7.2% and give them to the user to choose.

[0080] Based on the above embodiments, this embodiment describes step S103 in detail:

[0081] The user instruction information, the updated summary of the historical actions and historical action descriptions of the tth round, the screen perception information and the action plan of the tth round are input into the big model; that is, the decision agent obtains the next step plan made by the planning agent, and inputs a series of information into the big model in the form of prompt words, which includes the instruction information I, the next step plan PL given by the planning agent t , the planning agent obtains the screen information PA analyzed by the perception agent t , the updated memory information M t .

[0082] Get the action and action description of the tth round from the action library, and call the action function to operate the mobile phone; that is, the large model will get the required action AC from its own action library based on the current situation information it has obtained t and a description of the action ACD t After getting the action, the corresponding action function will be called directly to operate the mobile phone.

[0083] In a specific embodiment, the action library includes multiple basic operations and multiple complex operations, the basic operations include clicking, sliding, long pressing, typing, returning to the previous level, returning to the main interface, opening the program, stopping and communicating with the user, and the complex operations are composed of multiple basic operations encapsulated by uiautomator2. That is, the decision-making agent has an action library AC, which contains the basic operations of the mobile phone and some complex operations of the corresponding software. The basic operations include clicking, sliding, long pressing, typing, returning to the previous level, returning to the main interface, opening the program, stopping, and communicating with the user. Since some operation processes are very long, if only basic operations are used and analyzed step by step, the accuracy will be greatly reduced, so we use uiautomator2 to encapsulate some complex operations to make some repeated multi-step operations more efficient. uiautomator2 is an architecture based on Android's visual-hierarchical structure to use XML files to operate the mobile phone. Since XML files are used instead of vision, this will make the click operation very accurate, especially in some very confusing interfaces, or when ads pop up, it is a good choice to use the visual-hierarchical structure directly. We have encapsulated some complex functions, including ordering meals, taking taxis, sending files, etc.

[0084] Based on the above embodiments, this embodiment describes step S104 in detail:

[0085] After rounds of operations, until the last step of this task, the planning agent makes a plan PL t+n , the entire task has been completed. At this time, the planning intelligent body will generate the historical operation process PL corresponding to the entire instruction i , and save iti . The output of the planning agent is transmitted to the decision-making agent, which will generate the action "stop".

[0086] Based on the above embodiments, this embodiment describes step S105 in detail:

[0087] The user instruction information, the updated summary of historical actions and historical action descriptions of the tth round, the screen perception information of the tth round and the t+1th round, the action plan of the tth round, the actions and action descriptions performed in the tth round are input into the large model; that is, after the decision-making agent completes the operation, the perception agent will be called again. Because the mobile phone interface will change after the action is executed, the perception agent is called again to perceive the current mobile phone screenshot and obtain the information PA on the current mobile phone screen t+1 ; At this time, the reflective agent will be called, and the reflective agent will get the previous plan PL t , memory M t , the previous action AC t and action description ACD t , and the screen information PA obtained by the perceptual agent twice before and after the action is executed t With PA t+1 ,After these are input into the large model in the form of prompt words, it will get reflection on the current action;

[0088] Output the action reflection of the tth round, and guide the subsequent work of this round and the next round according to the action reflection of the tth round.

[0089] R t =ref lect(M t ,PA t ,PA t+1 ,PL t ,AC t ,ACD t )

[0090] In a specific embodiment, guiding the subsequent work of the current round and the next round based on the action reflection of the tth round includes:

[0091] If the action reflection is that the current action is executed correctly and the mobile phone interface changes (this situation will not affect the subsequent work), the next round of action planning, action decision-making and action reflection is carried out;

[0092] If the action reflection is that the current action is executed correctly and the mobile phone interface has not changed (in this case, it is likely that the click is not accurate enough), then it will prompt that there is no change and suggest that the user adjust the click strategy or check the device status;

[0093] If the action reflection is that the current action is executed incorrectly (in this case, the reflection agent will give a prompt to return to the previous level), then return to re-plan the action or re-determine the action or skip the action;

[0094] If the action reflection shows that there is a problem with the plan of the current action (this is because there is a problem with the planning agent or the decision-making agent itself, the plan is wrong or the decision generates an incorrect action. In this case, the reflective agent will give a problem with the action and describe the problem and why it went wrong), then analyze the problem and return to re-plan the action or re-decide the action.

[0095] The present invention relates to a mobile phone automatic operation method based on a multimodal agent. The method uses five agents, namely, a perception agent, a planning agent, a decision agent, a clarification agent and a feedback agent, to form a multi-agent architecture. Together with a memory module, the overall framework is formed. Among them, 1) the perception agent uses image recognition and text recognition to perceive the mobile phone interface, and describes it with a large model. 2) The memory module records the decisions made by the agent, and can also record the changes in mobile phone information after the instructions are completed. 3) The clarification agent clarifies the fuzzy instructions and performs probability analysis on the multi-option tasks to ensure that as few users as possible seek help. 4) The planning agent summarizes the decisions and actions made, and plans the next step. 5) The decision agent makes decisions on the generated plan, combines the action library, and takes corresponding actions. 6) The reflection agent reflects on the decisions made. If there is an error, the problem is described and fed back to the planning agent.

[0096] The present invention uses a multi-agent framework, which helps the large model process complex screen information on the mobile phone and make correct decisions. The present invention introduces a clarification agent into the multi-agent, and greatly reduces unnecessary interactions through the prediction probability model, allowing users to intervene in the decision-making process of the agent framework as little as possible, while also allowing the agent to promptly ask the user when encountering a desperate situation to ensure that the task can be successfully executed. The present invention adds skills encapsulated using uiautomator2 to the skill library that the agent can use, so that some multi-step operations can be executed faster and more accurately, improving efficiency and accuracy. The present invention uses multimodal data as an interface for multi-agents to obtain external information, so that the agent can better understand the mobile phone information.

[0097] Based on the above embodiments, this embodiment describes the present invention in detail with a complete process experiment:

[0098] (1) Deployment phase: The specific deployment is as follows:

[0099] 1-1) The Ocr recognition model uses ConvNextViT-document image recognition model uses GroundingDINO, the text description model uses GPT-4O, and the large model used by the intelligent agent is GPT-4O;

[0100] 1-2) When using on an Android phone, Android ADB, or Android Debug Bridge, is required to perform specific actions.

[0101] 1-3) Install related dependencies.

[0102] 1-4) Add some additional hints to guide the whole framework to be more suitable for the required scene, which will be stored in the memory information module M t middle

[0103] 1-5) Enter the relevant api-key and URL

[0104] (2) Implementation stage: The specific implementation is as follows

[0105] 2-1) Issue instructions to the WeChat logged in on the framework mobile phone on WeChat, or directly enter instructions I into the interactive interface.

[0106] 2-2) After receiving the instruction, the framework starts running. First, the perception agent takes a screenshot of the mobile phone screen. t , and then pass the screenshot to ConvNextViT-document and GroundingDINO to obtain the recognized icons and text information, as well as the location information ICt and At of these icons and texts.

[0107] 2-3) This information is IC t With A t Pack and send to gpt-4o, describe these icons, and get the description of their functions D t

[0108] 2-4) Get the current mobile phone keyboard information K t

[0109] 2-5) Identify text information, obtain user instructions, and save user instructions I

[0110] 2-6) Finally, we get the perception packet PA t

[0111] DIt,I,PAt=perceive(ICt,At,Dt)

[0112] 2-7) The screen information PA t And the user instruction I is input to the planning agent

[0113] 2-8) Since this is the first round, the planning agent only performs planning without summarizing. It only needs to input screen information, user instructions, and additional information. User instructions are included in the memory information M t middle.

[0114] PL t =plan(M t ,PA t-1 )

[0115] 2-8) The planning agent gives feedback. First, it needs to indicate whether it needs to wake up the clarification agent clari_flag

[0116] 2-9) If the clarification agent clari_flag == True needs to be awakened in 2-8), the content PL given by the planning agent will be t Clarifying Agent

[0117] 2-10) The problem PL given by the planning agent t The instruction I is given to the clarification agent, which determines what kind of question the instruction given by the user belongs to.

[0118] 2-11) After judging, ask a clarification question clari_ques and give feedback to the user

[0119] 2-12) After the user gives the next step instruction I', jump to 3-1)

[0120] 2-13) If the clarification agent clari_flag == False does not need to be awakened in 2-8), the content PL given by the planning agent is t Hand it over to the decision agent

[0121] 2-14) Input a series of parameters to the decision agent, including the plan PL given by the planning agent t , screen information PA t , including the additional information M in the memory information module t Since this is the first execution, no other parameters related to historical information are required.

[0122] 2-15) After obtaining the output of the decision-making agent, it is disassembled to obtain the next action to be performed and a description of the action.

[0123] ACD t ,AC t =decide(M t ,PA t ,PL t )

[0124] 2-16) Execute action AC t

[0125] 2-17) The perception agent perceives and passes the new round of screen information to the feedback agent

[0126] DI t+1 ,I,PA t+1 =perceive(IC t+1 ,A t+1 ,D t+1 )

[0127] 2-18) Input the screen information twice before and after the action into the feedback agent, plus additional information. Since this is the first round, there is no historical information.

[0128] R t =ref lect(M t ,PA t ,PA t+1 ,PL t ,AC t ,ACD t )

[0129] 2-19) The feedback agent gives feedback that the current action is executed incorrectly. In this case, the reflective agent will give a prompt to return to the previous level to guide the next work; after the current action is executed, the mobile phone interface does not change. In this case, it is likely that the click is not accurate enough. The reflective agent will give a prompt without change to guide the next work; there is a problem with the planning of the current action. In this case, it is because there is a problem with the planning agent or the decision-making agent itself, the plan is wrong or the decision generates an incorrect action. In this case, the reflective agent will give a problem with the action and describe the problem and why it went wrong, so as to guide the next work.

[0130] 2-20) If the feedback result in 2-19) is that the previous step went smoothly, Rt.err_flag == True, then proceed to 3-1)

[0131] 2-21) If the feedback result in 2-19) is that there was a problem in the previous step Rt.err_flag == False, then record the feedback information (3)

[0133] 3-1) The second round begins, first using the result PA of the last perception agent t+1 , this is because no operation has occurred and the screen information has not changed.

[0134] 3-2) PA the screen information t+1 , the last feedback result R t, historical actions, summary of historical actions, user historical chat information, historical information composed of commands M t+1 Input to the planning agent.

[0135] 3-3) The planning agent summarizes the actions that have been taken before, as well as the descriptions of the actions, and summarizes which step has been completed and what the next plan is. This will be affected by the feedback agent in the previous step. If there is a problem in the previous step, this step will provide a solution around the problem in the previous step.

[0136] COM t+1 ,PL t+1 =plan(M t+1 ,PA t+1 ,AC t+1 ,ACD t+1 ,R t )

[0137] 3-4) If a selection is required on the current interface, the planning agent will wake up the clarification agent clari_flag == True

[0138] 3-5) PA the screen information t+1 , instruction I, current plan PL t+1 And historical chat records M t+1 Enter the clarification agent, which will generate four options based on the content on the screen, and use the logprobs parameter of GPT to give the probabilities Prob of the four options and calculate the size of the probabilities. If two or more are greater than 0.928, each option greater than 0.072 will be sent to the user, and then a new round will be started.

[0139] C t =clarify(I,OP t )

[0140] 3-6) If there is no need to select clari_flag == False, then directly set the plan PL t+1 Input to the decision agent, and also input screen information PA t+1 , additional information recorded in the memory information, historical action information, historical action description M t+1 .

[0141] 3-7) The decision-making agent gives the corresponding decision

[0142] ACD t+1 ,AC t+1 =decide(M t+1 ,PA t+1 ,PL t+1 )

[0143] 3-8) After obtaining the output of the decision-making agent, disassemble it to obtain the next action to be performed and the description of the action ACD t+1 ,AC t+1

[0144] 3-9) Execute action AC t+1

[0145] 3-10) The perception agent perceives and passes the new round of screen information to the feedback agent

[0146] DI t+2 ,I,PA t+2 =perceive(IC t+2 ,A t+2 ,D t+2 )

[0147] 3-11) Input the screen information before and after the action into the feedback agent, plus additional information, historical operations, and historical operation descriptions.

[0148] R +1 =ref lect(Mt +1 ,PAt +1 ,PAt +2 ,PLt +1 ,ACt + 1,ACDt + 1)

[0149] The feedback agent gives feedback that the current action is executed incorrectly. In this case, the reflective agent will give a prompt to return to the previous level to guide the following work; after the current action is executed, the mobile phone interface does not change. In this case, it is likely that the click is not accurate enough. The reflective agent will give a prompt without change to guide the following work; there is a problem with the planning of the current action. In this case, it is because there is a problem with the planning agent or the decision-making agent itself, the plan is wrong or the decision generates an incorrect action. In this case, the reflective agent will give a problem with the action and describe the problem, describe why the error occurred, and guide the following work.

[0150] 3-12) After a series of operations, until the last step of this task t+n (4)

[0152] 4-1) In the last round, the planning agent makes a plan PL t+n , the entire task has been completed. At this time, the planning intelligent body will generate the historical operation flow PL corresponding to the entire instruction i , and save it i .

[0153] 4-2) The output of the planning agent is transmitted to the decision agent, which will generate the action "stop"

[0154] ACD t+n ,AC t+n =decide(M t+n ,PA t+n ,PL t+n )

[0155] ACstop t+n ==Stop

[0156] 4-3) The task corresponding to this instruction I is completed. At this time, the perceptual intelligent body will resume monitoring the user. If the user sends a new instruction I', the entire program will continue to execute and start a new round.

[0157] The embodiment of the present invention also provides a mobile phone automatic operation device; the specific device may include:

[0158] The perception agent module is used to obtain the screen perception information and interactive dialogue information of the mobile phone interface in the tth round after receiving the user's command information;

[0159] A planning agent module, used to input the user instruction information, the summary of historical actions and historical action descriptions of the tth round, the screen perception information, the actions performed in the t-1th round, the action descriptions and the action reflections into the big model, update the summary of historical actions and historical action descriptions of the tth round and output the action plan of the tth round;

[0160] A decision agent module, used to input the user instruction information, the updated summary of historical actions and historical action descriptions of the tth round, the screen perception information and the action plan of the tth round into the big model, obtain the actions and action descriptions performed in the tth round from the action library, and call the action function to operate the mobile phone;

[0161] The loop judgment module is used to judge whether the current instruction task is completed. If not, it will continue to the next step. If it is completed, it will end the task;

[0162] The reflective agent module is used to input the user instruction information, the updated summary of the historical actions and historical action descriptions of the tth round, the screen perception information of the tth round and the t+1th round, the action plan of the tth round, the actions and action descriptions performed in the tth round into the large model, output the action reflection of the tth round, and guide the subsequent work of this round and the next round based on the action reflection of the tth round.

[0163] The mobile phone automatic operation device of this embodiment is used to implement the aforementioned mobile phone automatic operation method. Therefore, the specific implementation method of the mobile phone automatic operation device can be seen in the implementation method part of the aforementioned mobile phone automatic operation method. For example, the perception intelligent agent module, the planning intelligent agent module, the decision-making intelligent agent module, the loop judgment module, and the reflection intelligent agent module are respectively used to implement steps S101, S102, S103, S104 and S105 in the above method. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part, which will not be repeated here.

[0164] Based on the above embodiments, the mobile phone automatic operation device further includes:

[0165] Clarification agent module, used to determine whether clarification is needed:

[0166] If there are preset problems in the interactive dialogue information, the next round of action planning, action decision-making and action reflection will be started directly. The preset problems include the lack of corresponding parameters in the user instruction and the semantic ambiguity in the user instruction;

[0167] If a selection is required on the current interface, the user instruction information, the updated summary of the historical actions and historical action descriptions of the tth round, the screen perception information, and the action plan of the tth round are input into the big model to generate multiple options, and the probability of each option is predicted. The options that meet the preset conditions are sent to the user, and the next round of action planning, action decision-making, and action reflection are started directly. The preset conditions include:

[0168] If the probability that only one option exists exceeds a first preset threshold, the option is sent to the user as an answer;

[0169] If the probability of two or more options existing exceeds a first preset threshold, options not lower than a second preset threshold are sent to the user for selection, wherein the sum of the first preset threshold and the second preset threshold is 1.

[0170] A specific embodiment of the present invention further provides a mobile phone automatic operation device, comprising: a memory for storing a computer program; a processor for implementing the steps of the above-mentioned mobile phone automatic operation method when executing the computer program.

[0171] A specific embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for automatic operation of a mobile phone are implemented.

[0172] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0173] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0176] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A method for automatic operation of a mobile phone, characterized in that: include: After receiving the user's command information, the screen perception information and interactive dialogue information of the mobile phone interface in the tth round are obtained; Input the user instruction information, the summary of historical actions and historical action descriptions of the tth round, the screen perception information, the actions performed in the t-1th round, the action descriptions, and the action reflections into the big model, update the summary of historical actions and historical action descriptions of the tth round, and output the action plan of the tth round; Input the user instruction information, the updated summary of historical actions and historical action descriptions of the tth round, the screen perception information and the action plan of the tth round into the big model, obtain the actions and action descriptions executed in the tth round from the action library, and call the action function to operate the mobile phone; Determine whether the current instruction task is completed. If not, proceed to the next step. If completed, end the process. The user command information, the updated summary of historical actions and historical action descriptions of the tth round, the screen perception information of the tth round and the t+1th round, the action plan of the tth round, the actions performed in the tth round and the action descriptions are input into the big model, the action reflection of the tth round is output, and the subsequent work of this round and the next round is guided according to the action reflection of the tth round.

2. The method for automatic operation of a mobile phone according to claim 1, characterized in that: After outputting the action plan of the tth round, the following further comprises: Determine whether clarification is needed: If there are preset problems in the interactive dialogue information, the next round of action planning, action decision-making and action reflection will be started directly. The preset problems include the lack of corresponding parameters in the user instruction and the semantic ambiguity in the user instruction; If a situation that requires a selection appears on the current interface, the user instruction information, the updated summary of historical actions and historical action descriptions for the tth round, the screen perception information, and the action plan for the tth round are input into the large model to generate multiple options, and the probability of each option is predicted. The options that meet the preset conditions are sent to the user, and the next round of action planning, action decision-making, and action reflection are started directly.

3. The mobile phone automatic operation method according to claim 2, characterized in that: The preset conditions include: If the probability that only one option exists exceeds a first preset threshold, the option is sent to the user as the answer; If the probability of two or more options existing exceeds a first preset threshold, options not lower than a second preset threshold are sent to the user for selection, wherein the sum of the first preset threshold and the second preset threshold is 1.

4. The method for automatically operating a mobile phone according to claim 1, characterized in that: The obtaining of screen perception information of the mobile phone interface in round t includes: Get the mobile phone screenshots of round t; Performing image recognition and text recognition on the mobile phone screenshot to obtain multiple icons and corresponding texts under the icons; The multiple icons and the corresponding texts under the icons are input into the large model to obtain the functional description of each icon, and analyze whether the keyboard of the mobile phone is awakened and whether each icon is clicked to obtain the perception information of the entire screen.

5. The mobile phone automatic operation method according to claim 1, characterized in that: The action library includes multiple basic operations and multiple complex operations. The basic operations include clicking, sliding, long pressing, typing, returning to the previous level, returning to the main interface, opening the program, stopping and communicating with the user. The complex operations are composed of multiple basic operations encapsulated by uiautomator2.

6. The method for automatically operating a mobile phone according to claim 1, characterized in that: The following steps of guiding the subsequent work of this round and the next round based on the action reflection of the tth round include: If the action reflection is that the current action is correctly executed and the mobile phone interface changes, the next round of action planning, action decision-making and action reflection is performed; If the action reflection is that the current action is executed correctly and the mobile phone interface has not changed, it will prompt that there is no change and suggest the user to adjust the click strategy or check the device status; If the action reflection is that the current action is executed incorrectly, then return to re-plan the action or re-determine the action or skip the action; If the action reflection shows that there is a problem with the plan of the current action, the problem is analyzed and the process returns to re-plan the action or re-decide the action.

7. The method for automatically operating a mobile phone according to claim 1, characterized in that: The updating of the summary of the historical actions and historical action descriptions of the tth round includes: Get the summary of historical actions and historical action descriptions of the tth round, that is, the action sequence and corresponding action descriptions that have been performed before the tth round; The action sequence and the corresponding action description are updated according to the action reflection of the t-1th round.

8. A mobile phone automatic operation device, characterized in that: include: The perception agent module is used to obtain the screen perception information and interactive dialogue information of the mobile phone interface in the tth round after receiving the user's command information; A planning agent module, used to input the user instruction information, the summary of historical actions and historical action descriptions of the tth round, the screen perception information, the actions performed in the t-1th round, the action descriptions and the action reflections into the big model, update the summary of historical actions and historical action descriptions of the tth round and output the action plan of the tth round; A decision agent module, used to input the user instruction information, the updated summary of historical actions and historical action descriptions of the tth round, the screen perception information and the action plan of the tth round into the big model, obtain the actions and action descriptions performed in the tth round from the action library, and call the action function to operate the mobile phone; The loop judgment module is used to judge whether the current instruction task is completed. If not, it will continue to the next step. If it is completed, it will end the process; The reflective agent module is used to input the user instruction information, the updated summary of the historical actions and historical action descriptions of the tth round, the screen perception information of the tth round and the t+1th round, the action plan of the tth round, the actions and action descriptions performed in the tth round into the large model, output the action reflection of the tth round, and guide the subsequent work of this round and the next round based on the action reflection of the tth round.

9. A mobile phone automatic operation device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of a mobile phone automatic operation method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the mobile phone automatic operation method as claimed in any one of claims 1 to 7 are implemented.

Images