Task processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a task processing method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the steps that a task needing to be completed by the electronic equipment is determined, screen information of a touch screen of the electronic equipment is obtained, historical action information of the electronic equipment is obtained, the historical action information is determined based on actions executed by the electronic equipment for the task, and the task, the screen information and the historical action information are input into a large language model; and obtaining a candidate action output by the large language model, and controlling the electronic equipment to execute the candidate action so as to enable the electronic equipment to complete the task. According to the method, the tasks related to the user interface can be decomposed into the actions through the large language model, then the electronic equipment is intelligently controlled to execute the decomposed actions to complete the tasks needing to be completed, and the working difficulty and workload of a user can be reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a task processing method, device, electronic device and storage medium. Background Art

[0002] With the development of science and technology, electronic devices are used more and more widely and have more and more functions. They have become one of the necessities in people's daily life. At present, electronic devices are generally integrated with touch screens, so that users can control the electronic devices through the graphical user interface (GUI) displayed on the touch screen. However, when controlling the electronic device to process some tasks, the user usually needs to control the electronic device through the GUI many times, which makes the user's work difficult and the workload is large. Summary of the invention

[0003] In view of the above problems, the present application proposes a task processing method, device, electronic device and storage medium to solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a task processing method, the method comprising: determining a task that an electronic device needs to complete; obtaining screen information of a touch screen of the electronic device, and obtaining historical action information of the electronic device, wherein the historical action information is determined based on actions that the electronic device has performed for the task; inputting the task, the screen information, and the historical action information into a large language model to obtain candidate actions output by the large language model; and controlling the electronic device to execute the candidate actions so that the electronic device completes the task.

[0005] In a second aspect, an embodiment of the present application provides a task processing device, comprising: a task determination module, used to determine a task that an electronic device needs to complete; an information acquisition module, used to obtain screen information of a touch screen of the electronic device, and to obtain historical action information of the electronic device, wherein the historical action information is determined based on the actions that the electronic device has performed for the task; a candidate action acquisition module, used to input the task, the screen information, and the historical action information into a large language model to obtain candidate actions output by the large language model; and an action execution module, used to control the electronic device to execute the candidate action so that the electronic device completes the task.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, and the program code can be called by a processor to execute the above method.

[0008] The task processing method, device, electronic device and storage medium provided in the embodiments of the present application determine the task that the electronic device needs to complete, obtain screen information of the touch screen of the electronic device, and obtain historical action information of the electronic device, wherein the historical action information is determined based on the action that the electronic device has performed for the task, input the task, screen information and historical action information into a large language model, obtain candidate actions output by the large language model, and control the electronic device to execute the candidate actions so that the electronic device completes the task, thereby decomposing the tasks related to the user interface into actions through the large language model, and then intelligently controlling the electronic device to execute the decomposed actions to complete the task to be completed, which can reduce the user's work difficulty and workload. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A schematic diagram showing a process flow of a task processing method provided by an embodiment of the present application is shown;

[0011] Figure 2 A schematic diagram showing a process flow of performing task processing by a program provided in an embodiment of the present application is shown;

[0012] Figure 3 A schematic diagram showing a process flow of a program provided in an embodiment of the present application performing task processing based on a large language model;

[0013] Figure 4 A schematic diagram showing a process flow of a task processing method provided by an embodiment of the present application is shown;

[0014] Figure 5 A schematic diagram showing a process flow of a task processing method provided by an embodiment of the present application is shown;

[0015] Figure 6 A schematic diagram showing a process flow of a task processing method provided by an embodiment of the present application is shown;

[0016] Figure 7 A schematic diagram showing a process flow of a task processing method provided by an embodiment of the present application is shown;

[0017] Figure 8A schematic diagram showing an interaction process provided by an embodiment of the present application is shown;

[0018] Fig. 9 A module block diagram of a task processing device provided by an embodiment of the present application is shown;

[0019] Fig.10 A block diagram of an electronic device according to an embodiment of the present application for executing a task processing method according to an embodiment of the present application is shown;

[0020] Fig.11 A storage unit for storing or carrying program codes for implementing the task processing method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0022] Among them, intelligent voice assistant applications have grown rapidly in recent years and have become a very core application form in the field of electronic devices. However, current intelligent voice assistant applications have at least the following problems:

[0023] First, in terms of technical capabilities, one of the core capabilities of intelligent voice assistants is the ability to understand user intentions. Existing intelligent voice assistants generally use intent classification models or rules to identify user intentions. The disadvantages of this method are that the generalization performance is not good enough and it is not smart enough. If the user changes the wording, the user's intentions often cannot be correctly identified.

[0024] Second, in terms of interaction form, because the intelligent voice assistant cannot understand complex user intentions and cannot break down user goals into tasks, its interaction form is "process-driven", that is, users are required to guide the intelligent voice assistant step by step to achieve its interaction purpose.

[0025] Third, in terms of multimodal understanding, intelligent voice assistants can only process text information and still face limitations in dialogue interaction combined with GUI. They are unable to understand the user's UI interface and complete some UI interaction actions on behalf of the user.

[0026] In response to the above problems, the inventors have found through research that the Large Language Model (LLM) is much better than the small model in understanding user intentions and reasoning. Therefore, LLM can be used to understand user intentions and handle more complex intentions; LLM is used to decompose the user's complex tasks, and the interaction form adopts a "goal-driven" interaction. The user only needs to give a goal, and the goal can be decomposed and completed step by step; it supports interaction with the GUI, can understand the user's current interaction interface and help the user complete the interaction of the user interface, rather than just being a chat robot. In summary, the embodiments of the present application propose a task processing method, device, electronic device, and storage medium, which can decompose tasks related to the user interface into actions through a large language model, and then intelligently control the electronic device to execute the decomposed actions to complete the required tasks, which can reduce the user's work difficulty and workload. Among them, the specific task processing method is described in detail in the subsequent embodiments.

[0027] See also Figure 1 , Figure 1 The flowchart of the task processing method provided by an embodiment of the present application is shown. The method is used to decompose the tasks related to the user interface into actions through a large language model, and then intelligently control the electronic device to perform the decomposed actions to complete the required tasks, which can reduce the user's work difficulty and workload. In a specific embodiment, the task processing method is applied to Fig. 9 The task processing device 200 and the electronic device 100 ( Fig.10 ). The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device used in this embodiment may include a smart phone, a tablet computer, a wearable electronic device, etc., which is not limited here. Figure 1 The process shown is described in detail. The task processing method may specifically include the following steps:

[0028] Step S110: Determine the task that the electronic device needs to complete.

[0029] Optionally, the embodiment of the present application can be applied to a program (Agent) installed in an electronic device, such as Figure 2 As shown in the figure, an agent can be understood as a program that can think and interact with the environment. Given a task, the agent will think about how to solve it, make a plan, take a series of actions on the environment, receive feedback from the environment, and repeat these steps until the task is completed.

[0030] like Figure 3As shown, the same is true for the Agent based on LLM, because LLM is very good at reasoning tasks. Therefore, this embodiment can use LLM to help the Agent think about the actions that need to be taken in the current environment state. Figure 3 In the framework, Task refers to tasks published in natural language, such as “Please send a WeChat message to Xiao Zhang about the meeting at 5 pm today”; Large Language Model (LLM): large language models such as gpt-4; Environment: screen information of electronic devices; Tools: barrier-free services obtain screen information and help users complete actions such as clicking; Action: actions that can be performed by the Agent, including clicking, text input, screen sliding, returning, task completion, etc.; Observation: screen information of the current electronic device, which can be a screenshot of the interface of the electronic device or barrier-free access to view tree information.

[0031] Among them, the electronic device may include a touch screen, which is used to display information input by the user, information provided to the user, and various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, numbers, videos, and any combination thereof. In one example, the touch screen can be a liquid crystal display (Liquid Crystal Display, LCD) or an organic light-emitting diode (Organic Light-Emitting Diode, OLED), which is not limited here.

[0032] In this embodiment, the tasks that the electronic device needs to complete can be determined, wherein the tasks that the electronic device needs to complete can refer to tasks that need to be completed through a graphical user interface on the touch screen of the electronic device. For example, the tasks that the electronic device needs to complete may include "send a WeChat message to Xiao Zhang about a meeting at 5 o'clock today", "book a flight to Shenzhen tomorrow morning", "take a car home at 6 o'clock this afternoon", etc., without limitation here.

[0033] In some embodiments, the task may be input into the electronic device by the user in an interactive form. For example, the task may be input into the electronic device by the user in a voice manner, the task may be input into the electronic device by the user in a text manner, the task may be input into the electronic device by the user in a touch operation manner, etc., which are not limited here. Optionally, the information input by the user in an interactive form may be obtained, and the information may be determined as a task to be completed; or, the information input by the user in an interactive form may be obtained, and fields may be extracted from the information to determine the task to be completed, etc., which are not limited here.

[0034] In some embodiments, the task may be generated when a preset condition is met. For example, if the current time meets the preset time, the task that the electronic device needs to complete is automatically generated; if the current location of the electronic device meets the preset location, the task that the electronic device needs to complete is automatically generated; if the current posture of the electronic device meets the preset posture, the task that the electronic device needs to complete is automatically generated, etc., which are not limited here.

[0035] Step S120: acquiring screen information of the touch screen of the electronic device, and acquiring historical action information of the electronic device, wherein the historical action information is determined based on actions that have been performed by the electronic device for the task.

[0036] In this embodiment, when it is determined that the electronic device needs to complete a task, screen information of the touch screen of the electronic device can be acquired in response to the task.

[0037] In some implementations, screen information of the touch screen of an electronic device can be obtained through an accessibility service. Among them, the accessibility service is a mechanism provided by operating systems such as Android, which aims to help users with visual, hearing, and motor impairments to better use electronic devices. These services can access and operate device interface elements such as text, buttons, pictures, etc. Specifically, through the accessibility service, at least the following operations can be completed:

[0038] Reading screen content: Accessibility services can access all text and interface elements displayed on the touch screen. This can be achieved by traversing the AccessibilityNodeInfo tree, which represents all interactive elements on the current touch screen.

[0039] Simulating clicks and gestures: Accessibility services can simulate screen clicks, long presses, swipes, and other gestures to allow applications to automatically interact with elements on a touch screen.

[0040] Get window state: Accessibility services can know which app or window is currently active to determine which app the user is interacting with.

[0041] Additional functionality: Accessibility services can also perform other actions, such as changing the device volume, interacting with notifications, simulating keyboard input, etc., to further expand the purpose and scope of accessibility services.

[0042] In some implementations, screen information of a touch screen of an electronic device may be obtained through a programming interface (API). The touch screen generally provides a programming interface that allows developers to access and operate interface elements through code. For example, some touch screen frameworks or libraries provide APIs for reading and modifying interface content. Therefore, these APIs may be used to obtain screen information of the touch screen, such as the properties, status, and content of the user interface on the touch screen, which are not limited here.

[0043] In some implementations, screen information of a touch screen of an electronic device may be obtained through software tools. Among them, some software tools may be used to analyze the interface content of the touch screen, for example, UI automated testing tools, screenshot tools, UI element identification tools, etc. These tools may be used to capture screen information of the touch screen, such as the properties, status, content, etc. of the user interface on the touch screen, which are not limited here.

[0044] In some implementations, the screen information of the touch screen of the electronic device can be obtained through system logs and debugging information. In some cases, the operating system or firmware of the touch screen may provide system logs or debugging information, which may contain clues or data about the content of the screen information of the touch screen. Therefore, the screen information of the touch screen, such as the properties, status, content, etc. of the user interface on the touch screen, can be obtained by analyzing these system logs or debugging information, which is not limited here.

[0045] In this embodiment, when it is determined that the electronic device needs to complete a task, historical action information of the electronic device can be obtained in response to the task, wherein the historical action information is determined based on the action that the electronic device has performed for the task. Optionally, if the electronic device has performed an action for the task, the historical action information can be determined based on the action that the electronic device has performed for the task; if the electronic device does not have a performed action for the task, the historical action information of the electronic device can be empty.

[0046] As an example, if the task that the electronic device needs to complete corresponds to 10 actions, and the electronic device has performed 3 actions for the task, then the historical action information of the electronic device can be determined based on the 3 actions that have been performed; if the task that the electronic device needs to complete corresponds to 10 actions, and the electronic device has not yet started to perform actions for the task, then the historical action information of the electronic device is empty.

[0047] As an implementable method, if the electronic device has executed actions for the task, it can be determined whether the length of the historical action information composed of all executed actions exceeds the input length limit of the LLM. If the length of the historical action information composed of all executed actions does not exceed the input length limit of the LLM, the historical action information can be composed of all executed actions, that is, at this time, the historical action information of the electronic device obtained can be composed of all executed actions; if the length of the historical action information composed of all executed actions exceeds the input length limit of the LLM, the historical action information can be composed of some of the executed actions among all the executed actions to avoid the historical action information exceeding the input length limit of the LLM, that is, at this time, the historical action information of the electronic device obtained can be composed of some of the executed actions, and optionally, the partially executed actions can be actions with a later execution time or execution order among all the executed actions.

[0048] In some implementations, the actions that the electronic device has performed for the task may be cached in the memory of the electronic device and form historical action information, based on which the historical action information of the electronic device for the task may be read from the memory of the electronic device.

[0049] As an example, the actions performed by the electronic device for the task may include the following forms:

[0050] 1.{"action_type":"click","id":"<element_id> "}, which means clicking a certain control requires the control's id information.

[0051] 2.{"action_type":"type","id":"<element_id> ","text":"<input_text> "}, indicating that text input is performed on controls such as EditText, which requires the control id information and the text information to be entered.

[0052] 3.{"action_type":"scroll","id":"<element_id> ","direction":"<UP|DOWN|LEFT|RIGHT> "}, indicating sliding the screen up, down, left, or right.

[0053] 4.{"action_type":"navigate_home"}, means pressing the home button to enter the desktop.

[0054] 5.{"action_type":"navigate_back"}, means pressing the back key to return.

[0055] 6. {"action_type": "status_complete"}, a status action indicating that the task is completed and the task can be terminated.

[0056] 7. {"action_type": "status_impossible"}, a status action indicating that the task cannot be completed and the task needs to be terminated.

[0057] As an example, if the task that the electronic device needs to complete is "Help me send a WeChat message to Zhang Xiao with the content 'There will be a meeting at 5 pm this afternoon'", the historical action information of the electronic device can include the following forms:

[0058] {"step_idx": 1, "action_description": click[WeChat]}

[0059] {"step_idx": 2, "action_description": click[Search]}

[0060] {"step_idx": 3, "action_description": type[Zhang Xiao] in [Search]}

[0061] {"step_idx": 4, "action_description": click[Zhang Xiao]}

[0062] {"step_idx": 5, "action_description": type[There will be a meeting at 5 pm this afternoon] in [null]}

[0063] {"step_idx": 6, "action_description": click[Send]}

[0064] Step S130: Input the task, the screen information, and the historical action information into the large language model to obtain candidate actions output by the large language model.

[0065] In this embodiment, when the task, screen information and historical action information are obtained, the task, screen information and historical action information can be input into a large language model to predict the next action required to complete the task, and obtain the candidate action output by the large language model. The large language model is obtained through machine learning. Specifically, a training data set is first collected, wherein the attributes or characteristics of one type of data in the training data set are distinguished from another type of data, and then the neural network is trained and modeled according to a preset algorithm using the collected training data set, thereby summarizing the rules based on the training data set to obtain a large language model. Optionally, the training data set of the diffusion model can be a plurality of tasks that need to be completed by the electronic device.

[0066] It is understandable that the large language model can be pre-trained and stored locally in the electronic device. Based on this, when the electronic device obtains tasks, screen information, and historical action information, it can directly call the large language model locally. For example, it can directly send instructions to the large language model to instruct the large language model to read the task, screen information, and historical action information in the target storage area, or the electronic device can directly input the task, screen information, and historical action information into the large language model stored locally, thereby effectively avoiding the influence of network factors that reduce the speed of inputting tasks, screen information, and historical action information into the large language model, so as to improve the speed of the large language model to obtain multiple tasks, screen information, and historical action information, and improve the user experience.

[0067] In addition, the large language model can also be pre-trained and stored in a server that is connected to the electronic device. Based on this, when the electronic device obtains tasks, screen information, and historical action information, it can send instructions to the large language model stored in the server through the network to instruct the large language model to read the tasks, screen information, and historical action information through the network, or the electronic device can send the tasks, screen information, and historical action information to the large language model stored in the server through the network, thereby reducing the storage space occupied by the electronic device and reducing the impact on the normal operation of the electronic device by storing the large language model in the server.

[0068] In some implementations, when the task, screen information, and historical action information are acquired, the task, screen information, and historical action information may be respectively input into a large language model to obtain candidate actions output by the large language model.

[0069] In some implementations, when the task, screen information, and historical action information are acquired, the task, screen information, and historical action information may be integrated and input into a large language model to obtain candidate actions output by the large language model.

[0070] Step S140: controlling the electronic device to execute the candidate action so that the electronic device completes the task.

[0071] In this embodiment, when a candidate action is obtained from the output of a large language model, the electronic device can be controlled to execute the candidate action so that the electronic device can complete the task to be completed, thereby enabling the GUI application to have the ability to interact with AI in a unified and non-invasive manner, giving the user an intelligent experience. As an example, if the candidate action is "send", the electronic device can be controlled to execute the "send" action; if the candidate action is "search", the electronic device can be controlled to execute the "search" action, which is not limited here.

[0072] In some implementations, when candidate actions are obtained from the output of a large language model, the candidate actions can be executed through an accessibility service so that the electronic device completes the task that needs to be completed.

[0073] In some embodiments, after the electronic device is controlled to perform the candidate action, the historical action information may be updated based on the candidate action. Optionally, the candidate action may be recorded in the historical action information to form new historical action information, thereby facilitating the subsequent large language model to more accurately predict the next candidate action.

[0074] In some implementations, after controlling the electronic device to perform the candidate action, it may be determined whether the electronic device has completed the task. If it is determined that the electronic device has completed the task, no other operations may be performed; if it is determined that the electronic device has not completed the task, the steps of obtaining screen information of the touch screen of the electronic device to controlling the electronic device to perform the candidate action may be repeated until it is determined that the electronic device has completed the task, thereby enabling the user to only need to give one task, and then the task can be decomposed and completed step by step.

[0075] As an implementable manner, after controlling the electronic device to execute the candidate action, it can be determined whether the candidate action is the last action to complete the task. If it is determined that the candidate action is the last action to complete the task, it can be indicated that the electronic device has completed the task; if it is determined that the candidate action is not the last action to complete the task, it can be determined that the electronic device has not completed the task.

[0076] As another feasible method, after controlling the electronic device to execute the candidate action, the historical action information can be updated based on the candidate action to obtain updated historical action information, and then, based on the task, the updated historical action information and the screen information, it is determined whether the task is completed.

[0077] A task processing method provided by an embodiment of the present application determines a task that an electronic device needs to complete, obtains screen information of a touch screen of the electronic device, and obtains historical action information of the electronic device, wherein the historical action information is determined based on the action that the electronic device has performed for the task, inputs the task, screen information, and historical action information into a large language model, obtains candidate actions output by the large language model, and controls the electronic device to execute the candidate actions so that the electronic device completes the task, thereby being able to decompose tasks related to the user interface into actions through the large language model, and then intelligently control the electronic device to execute the decomposed actions to complete the task to be completed, thereby reducing the user's work difficulty and workload.

[0078] See also Figure 4 , Figure 4 The following is a flow chart of a task processing method provided by an embodiment of the present application. Figure 4 The process shown is described in detail, and the task processing method may specifically include the following steps:

[0079] Step S210: Determine the task that the electronic device needs to complete.

[0080] For the detailed description of step S210, please refer to step S110, which will not be repeated here.

[0081] Step S220: Acquire a view hierarchy structure of user interface elements on the touch screen of the electronic device.

[0082] Among them, the screen information needs to be transmitted to LLM for auxiliary decision-making. There are two difficult problems that need to be solved:

[0083] 1. Screen information is multimodal, including text, images, and structural hierarchical information, while LLM can only accept text input.

[0084] 2. It is not feasible to directly input the hierarchical data of the screen’s view tree into LLM because it contains too much information, such as detailed properties of each UI element, which may exceed the input length limit of LLM and also interfere with LLM’s reasoning.

[0085] In order to address these challenges, this embodiment proposes to apply the screen UI to the LLM by using the screen information conversion technology.

[0086] In this embodiment, a view hierarchy structure (view tree) of user interface elements on the touch screen of the electronic device may be obtained.

[0087] In some implementations, a view hierarchy of user interface elements on a touch screen may be obtained through an accessibility service.

[0088] In some implementations, the root view of the current Activity may be obtained, and then the entire view tree may be traversed recursively or iteratively to obtain a view hierarchy of user interface elements on the touch screen.

[0089] Step S230: converting the view hierarchy into a text modal format to obtain the screen information.

[0090] In this embodiment, when the view hierarchy of the user interface elements on the touch screen is obtained, the view hierarchy can be converted into a text mode to obtain screen information. It can be understood that at this time the screen information is in text mode and can meet the input requirements of LLM.

[0091] In some embodiments, when the view hierarchy of the user interface elements on the touch screen is obtained, the view hierarchy can be converted into the form of HyperText Markup Language (HTML) to obtain the screen information. It is understandable that the screen information in the form of HTML has at least the following advantages. First, it can reduce the use of tokens: HTML is a relatively efficient way to represent the view tree structure. By hiding invisible elements and other means, the use of tokens can be effectively reduced to avoid exceeding the limitations of the LLM context. Second, it is easy to understand: when LLM is trained, most of the training data is obtained by crawling the web. A large part of it is expressed in HTML, so LLM can understand HTML very well and obtain good reasoning effects.

[0092] As an implementable method, this embodiment designs an algorithm that can use depth-first search traversal to convert the view hierarchy of the Android UI into HTML syntax, thereby achieving the purpose of converting the screen information into a text mode and streamlining the screen UI information to improve the accuracy of LLM decision-making. Of course, other traversal algorithms can also be used for traversal, which is not limited here.

[0093] The view nodes of the view hierarchy may be converted according to the following corresponding relationship.

[0094] a. Type mapping:

[0095] TextView → p

[0096] Button→button (all Button-related classes, such as Button, ImageButton)

[0097] ImageView → img (including icons and images)

[0098] EditText → input

[0099] Others such as LinearLayout converted to div

[0100] b. Property list

[0101] class: the type of Android object, such as TextView, Button

[0102] text: user-visible text information in the element

[0103] resource_id: The ID string of the referenced resource

[0104] content_desc: Describes the accessibility attributes of the element, such as alt-text

[0105] c. Element ID

[0106] In order to match the original UI element, an ID attribute is assigned to each HTML element, and the ID value is the hashcode of the View, so that the same ID can be maintained when the View object does not change. That is, the unique identifier hashcode of the traversed view in the hash table can be obtained, and the unique identifier of the view in the hash table can be used as the unique identifier of the corresponding HTML element.

[0107] In summary, using HTML to represent screen information can help LLM better understand the current interface: As mentioned above, most LLMs currently only accept text input, so how to make LLM understand the current screen is a difficult point. Considering that LLM is trained with a large amount of HTML data crawled from web pages during training, LLM has a better understanding of HTML. Based on this, this embodiment uses the method of converting the current GUI view tree into HTML, so that the entire reasoning result has a good effect.

[0108] Step S240: Acquire historical action information of the electronic device, wherein the historical action information is determined based on actions that the electronic device has performed for the task.

[0109] Step S250: inputting the task, the screen information and the historical action information into a large language model to obtain candidate actions output by the large language model.

[0110] Step S260: Control the electronic device to execute the candidate action so that the electronic device completes the task.

[0111] For the detailed description of step S240 to step S260, please refer to step S120 to step S140, which will not be repeated here.

[0112] The task processing method provided by an embodiment of the present application determines the task that the electronic device needs to complete, obtains the view hierarchy of the user interface elements on the touch screen of the electronic device, converts the view hierarchy into a text mode, obtains screen information, obtains historical action information of the electronic device, wherein the historical action information is determined based on the action that the electronic device has performed for the task, inputs the task, screen information and historical action information into a large language model, obtains candidate actions output by the large language model, and controls the electronic device to execute the candidate actions so that the electronic device completes the task. Compared to Figure 1 The task processing method shown in this embodiment also uses HTML to represent the screen information, so that the large language model can better understand the current interface, thereby making the entire reasoning result have a better effect.

[0113] See also Figure 5 , Figure 5 The following is a flow chart of a task processing method provided by an embodiment of the present application. Figure 5 The process shown is described in detail, and the task processing method may specifically include the following steps:

[0114] Step S310: Determine the task that the electronic device needs to complete.

[0115] Step S320: obtaining screen information of the touch screen of the electronic device, and obtaining historical action information of the electronic device, wherein the historical action information is determined based on actions that have been performed by the electronic device for the task.

[0116] For the detailed description of step S310 to step S320, please refer to step S110 to step S120, which will not be repeated here.

[0117] Step S330: Assemble the task, the screen information and the historical action information through an instruction template to obtain a target instruction.

[0118] In some implementations, the electronic device may be pre-set and store instruction templates, and when tasks, screen information, and historical action information are obtained, the tasks, screen information, and historical action information may be assembled using the instruction templates to obtain target instructions.

[0119] Among them, the instruction template is a structured framework that defines how to combine different information elements (such as task description, screen element identifier, action parameters, etc.) to form a complete instruction. The instruction template can contain placeholders or specific format requirements to indicate where to insert tasks, screen information, and historical action information.

[0120] Optionally, the instruction template is used to determine the next candidate action given a user instruction (task) based on the screen information and historical action information. Examples of selectable candidate actions: %s; historical action information: %s; screen information: %s; task: %s. Please carefully analyze the user's requirements and process them. The processing needs to be carried out in the following steps: Step 1: ...; Step 2: ...; Step 3: ... Use the following format for derivation: Step 1: <Step 1 derivation>; Step 2: <Step 2 derivation>; Step 3: <Step 3 derivation>.

[0121] As an implementable approach, the instruction template can be used as a framework to insert task information, screen information, and historical action information into corresponding placeholders or format requirements through operations such as string concatenation, data formatting, and parameter replacement. The assembled target instruction should be a clear, accurate, and executable command that describes how to perform a user-specified task in the current screen context.

[0122] Step S340: input the target instruction into the large language model to obtain the candidate action output by the large language model.

[0123] In this embodiment, when a target instruction is obtained, the target instruction can be input into a large language model to obtain candidate actions output by the large language model.

[0124] Step S350: controlling the electronic device to execute the candidate action so that the electronic device completes the task.

[0125] For the detailed description of step S350, please refer to step S140, which will not be repeated here.

[0126] An embodiment of the present application provides a task processing method, which determines the task that an electronic device needs to complete, obtains screen information of a touch screen of the electronic device, and obtains historical action information of the electronic device, wherein the historical action information is determined based on the action that the electronic device has performed for the task, assembles the task, screen information, and historical action information through an instruction template, obtains a target instruction, inputs the target instruction into a large language model, obtains a candidate action output by the large language model, and controls the electronic device to execute the candidate action so that the electronic device completes the task. Compared to Figure 1The task processing method shown in the figure, this embodiment also uses the instruction template to assemble the task, screen information and historical action information and then inputs them into the large language model to improve the processing effect of the large language model.

[0127] See also Figure 6 , Figure 6 The following is a flow chart of a task processing method provided by an embodiment of the present application. Figure 6 The process shown is described in detail, and the task processing method may specifically include the following steps:

[0128] Step S410: Determine the task that the electronic device needs to complete.

[0129] For the detailed description of step S510, please refer to step S110, which will not be repeated here.

[0130] Step S420: Determine the complexity level corresponding to the task.

[0131] Optionally, in addition to executing the task processing method proposed in the embodiment of the present application, the electronic device can also have built-in action scripts (Action Script) for some high-frequency applications, such as WeChat-related operations. However, the generalization ability of executing actions through scripts is poor. When the operation path changes (such as the pop-up box such as permission confirmation when the relevant function is used for the first time), the action script cannot be well compatible. When the third-party application is revised, the action script also needs to be updated. By comparison, it is found that the advantage of the action script is fast execution speed, but the disadvantage is narrow coverage and weak fault tolerance; the advantage of the task processing method proposed in the embodiment of the present application is wide coverage and strong fault tolerance, but the disadvantage is slow execution speed.

[0132] Based on the above analysis, this embodiment can be set in a single task, the part with a relatively clear path can be executed using action scripts to improve the response speed, and the task processing method can be used in the link with complex actions to meet compatibility.

[0133] Therefore, in this embodiment, the complexity corresponding to the task can be obtained.

[0134] In some implementations, the number of steps required to complete a task may be determined, and the complexity of the task may be determined based on the number of steps required to complete the task. The number of steps required to complete a task is positively correlated with the complexity of the task, that is, the greater the number of steps required to complete a task, the higher the complexity of the task; the smaller the number of steps required to complete a task, the lower the complexity of the task.

[0135] As an implementable method, multiple complexity levels may be preset, such as level one complexity level, level two complexity level, and level three complexity level. When the number of steps required to complete a task is in the range of 0-5, it corresponds to level one complexity level, when the number of steps required to complete a task is in the range of 6-10, it corresponds to level two complexity level, and when the number of steps required to complete a task is more than 10, it corresponds to level three complexity level. Based on this, when determining the number of steps required to complete a task, the complexity level corresponding to the task may be determined from level one complexity level, level two complexity level, and level three complexity level based on the correspondence between the preset complexity level and the number of steps required to complete the task.

[0136] Step S430: If the complexity reaches a complexity threshold, screen information of the touch screen of the electronic device is obtained, and historical action information of the electronic device is obtained, wherein the historical action information is determined based on actions that have been performed by the electronic device for the task.

[0137] In some embodiments, the electronic device may pre-set and store a complexity threshold. The complexity threshold is used as a basis for determining the complexity of the task. Therefore, in this embodiment, when the complexity of the task is obtained, the complexity of the task may be compared with the complexity threshold to determine whether the complexity of the task reaches the complexity threshold. Optionally, the complexity threshold may be, for example, the second complexity.

[0138] Among them, if it is determined that the complexity reaches the complexity threshold, it can be considered that the task is relatively complex, the action script is difficult to cover, and it is difficult to complete the task. Therefore, the task processing method proposed in the embodiment of the present application can be adopted for execution. Therefore, the screen information of the touch screen of the electronic device can be obtained, and the historical action information of the electronic device can be obtained to participate in the prediction of the next candidate action.

[0139] If it is determined that the complexity does not reach the complexity threshold, it can be considered that the task is relatively clear and simple, and the action script can cover and complete the task. Therefore, the action script can be used to execute the task to improve the response speed.

[0140] Step S440: inputting the task, the screen information and the historical action information into a large language model to obtain candidate actions output by the large language model.

[0141] Step S450: controlling the electronic device to execute the candidate action so that the electronic device completes the task.

[0142] For the detailed description of step S440 to step S450, please refer to step S130 to step S140, which will not be repeated here.

[0143] The task processing method provided by an embodiment of the present application determines the task that the electronic device needs to complete, determines the complexity of the task, and if the complexity reaches a complexity threshold, obtains the screen information of the touch screen of the electronic device and obtains the historical action information of the electronic device, wherein the historical action information is determined based on the action that the electronic device has performed for the task, inputs the task, screen information and historical action information into the large language model, obtains the candidate action output by the large language model, and controls the electronic device to execute the candidate action so that the electronic device completes the task. Compared to Figure 1 The task processing method shown in the figure, this embodiment also adopts the task processing method to process when the complexity of the task reaches the complexity threshold, so as to improve the compatibility of the electronic device.

[0144] See also Figure 7 , Figure 7 The following is a flow chart of a task processing method provided by an embodiment of the present application. Figure 7 The process shown is described in detail, and the task processing method may specifically include the following steps:

[0145] Step S510: Determine the task that the electronic device needs to complete.

[0146] For the detailed description of step S510, please refer to step S110, which will not be repeated here.

[0147] Step S520: Determine the usage frequency of the application corresponding to the task.

[0148] Optionally, in addition to executing the task processing method proposed in the embodiment of the present application, the electronic device can also have built-in action scripts (Action Script) for some high-frequency applications, such as WeChat-related operations. However, the generalization ability of executing actions through scripts is poor. When the operation path changes (such as the pop-up box such as permission confirmation when the relevant function is used for the first time), the action script cannot be well compatible. When the third-party application is revised, the action script also needs to be updated. By comparison, it is found that the advantage of the action script is fast execution speed, but the disadvantage is narrow coverage and weak fault tolerance; the advantage of the task processing method proposed in the embodiment of the present application is wide coverage and strong fault tolerance, but the disadvantage is slow execution speed.

[0149] Based on the above analysis, this embodiment takes into account the limited coverage of action scripts and can only perform targeted optimization for high-frequency applications, but does not cover medium-frequency applications and long-tail low-frequency applications. If the action script can be combined with the task processing method provided in the embodiment of this application, a good experience can be achieved in the interaction of high, medium and low-frequency applications.

[0150] Therefore, in this embodiment, the usage frequency of the application corresponding to the task can be obtained.

[0151] In some implementations, a list of applications recently used by a user may be obtained through the getRecentTasks interface. The list is arranged in reverse order of the order in which the applications were used. By comparing task lists obtained at different time points, the frequency of use of the application corresponding to the task may be calculated.

[0152] In some implementations, the list of currently running applications can be obtained through the getRunningTasks interface. By frequently obtaining this list and comparing two consecutive results, it is possible to determine which applications are frequently used, thereby calculating the usage frequency of the application corresponding to the task.

[0153] Of course, in this embodiment, the usage frequency of the application corresponding to the task may be determined in other ways, which will not be described in detail here.

[0154] As an implementable method, the number of times the application corresponding to the task has been used in the past preset time period can be determined, and the usage frequency of the application corresponding to the task can be determined based on the number of times the application corresponding to the task has been used in the past preset time period. It is assumed that the usage frequency is divided into three types: low, medium, and high, and when the number of times the application has been used in the past preset time period is in the range of 0-5, it corresponds to a low usage frequency; when the number of times the application has been used in the past preset time period is in the range of 6-10, it corresponds to a high usage frequency; when the number of times the application has been used in the past preset time period is more than 10, it corresponds to a high usage frequency. Based on this, when determining the number of times the application corresponding to the task has been used in the past preset time period, the usage frequency of the application corresponding to the task can be determined from low usage frequency, medium usage frequency, and high usage frequency based on the correspondence between the preset number of uses and usage frequency.

[0155] Step S530: If the usage frequency does not reach the usage frequency threshold, screen information of the touch screen of the electronic device is obtained, and historical action information of the electronic device is obtained, wherein the historical action information is determined based on the actions that the electronic device has performed for the task.

[0156] In some embodiments, the electronic device may pre-set and store a usage frequency threshold. The usage frequency threshold is used as a basis for determining the usage frequency of the application corresponding to the task. Therefore, in this embodiment, when the usage frequency of the application corresponding to the task is obtained, the usage frequency of the application corresponding to the task may be compared with the usage frequency threshold to determine whether the usage frequency of the application corresponding to the task reaches the usage frequency threshold. Optionally, the usage frequency threshold may be, for example, the usage frequency mentioned above.

[0157] Among them, if it is determined that the usage frequency does not reach the usage frequency threshold, it can be considered that the application corresponding to the task is a medium-frequency application or a long-tail frequency application, and the action script is difficult to cover, and it is difficult to complete the task. Therefore, the task processing method proposed in the embodiment of the present application can be adopted for execution. Therefore, the screen information of the touch screen of the electronic device can be obtained, and the historical action information of the electronic device can be obtained to participate in the prediction of the next candidate action.

[0158] If it is determined that the usage frequency reaches the usage frequency threshold, it can be considered that the application corresponding to the task is a high-frequency application, and the action script can cover and complete the task. Therefore, the action script can be used to execute the task to improve the response speed.

[0159] Step S540: input the task, the screen information and the historical action information into a large language model to obtain candidate actions output by the large language model.

[0160] Step S550: Control the electronic device to execute the candidate action, so that the electronic device completes the task.

[0161] For the detailed description of step S540 to step S550, please refer to step S130 to step S140, which will not be repeated here.

[0162] The task processing method provided by an embodiment of the present application determines the task that the electronic device needs to complete, determines the usage frequency of the application corresponding to the task, and if the usage frequency does not reach the usage frequency threshold, obtains the screen information of the touch screen of the electronic device and obtains the historical action information of the electronic device, wherein the historical action information is determined based on the action that the electronic device has performed for the task, inputs the task, screen information and historical action information into the large language model, obtains the candidate action output by the large language model, and controls the electronic device to execute the candidate action so that the electronic device completes the task. Compared to Figure 1 The task processing method shown in the figure, this embodiment also adopts the task processing method to process when the usage frequency of the application corresponding to the task does not reach the usage frequency threshold, so as to improve the coverage of the electronic device.

[0163] See also Figure 8 , Figure 8 FIG. 1 is a schematic diagram showing the interaction process provided by an embodiment of the present application. Figure 8 As shown, the interaction process may include the following steps:

[0164] Step 1: The user issues a task instruction, such as "Please send a WeChat message to Xiao Zhang saying there is a meeting at 5 pm today."

[0165] Step 2: The agent obtains screen information (view tree) through the accessibility service.

[0166] Step 3: Agent converts the screen information (view tree) into HTML.

[0167] Step 4: The agent reads the historical action information of the current task from the memory.

[0168] Step 5: The agent assembles the user instructions, screen information, and historical action information into the final prompt through the prompt template.

[0169] Step 6: Agent sends prompt to LLM.

[0170] Step 7: LLM outputs the next candidate action.

[0171] Step 8: The agent executes the candidate action through the accessibility service.

[0172] Step 9: The agent records the executed actions into the historical action information.

[0173] Step 10: The Agent determines whether the task is completed. If not, it repeats the process after step 2.

[0174] See also Fig. 9 , Fig. 9 The module block diagram of the task processing device provided by an embodiment of the present application is shown below. Fig. 9 The task processing device 200 may include: a task determination module 210, an information acquisition module 220, a candidate action acquisition module 230 and an action execution module 240, wherein:

[0175] The task determination module 210 is used to determine the task that the electronic device needs to complete.

[0176] The information acquisition module 220 is used to acquire screen information of the touch screen of the electronic device and acquire historical action information of the electronic device, wherein the historical action information is determined based on actions that the electronic device has performed for the task.

[0177] Furthermore, the information acquisition module 220 includes: a view hierarchy acquisition submodule and a screen information acquisition submodule, wherein:

[0178] The view hierarchy structure acquisition submodule is used to acquire the view hierarchy structure of the user interface elements on the touch screen of the electronic device.

[0179] The screen information acquisition submodule is used to convert the view hierarchy into a text modal form to obtain the screen information.

[0180] Furthermore, the screen information acquisition submodule includes: a screen information acquisition unit, wherein:

[0181] The screen information acquisition unit is used to convert the view hierarchy into a form of hypertext markup language HTML to obtain the screen information.

[0182] Furthermore, the screen information acquisition unit includes: a screen information acquisition subunit, wherein:

[0183] The screen information acquisition subunit is used to traverse the view hierarchy and create corresponding HTML elements for the traversed views to obtain the screen information.

[0184] Furthermore, the screen information acquisition unit includes: an identifier assignment subunit, wherein:

[0185] The identifier assigning subunit is used to obtain the unique identifier of the traversed view in the hash table, and use the unique identifier of the view in the hash table as the unique identifier of the corresponding HTML element.

[0186] Furthermore, the information acquisition module 220 includes: a complexity determination submodule and a first information acquisition submodule, wherein:

[0187] The complexity determination submodule is used to determine the complexity corresponding to the task.

[0188] The first information acquisition submodule is used to acquire screen information of the touch screen of the electronic device and acquire historical action information of the electronic device if the complexity reaches a complexity threshold.

[0189] Furthermore, the information acquisition module 220 includes: a usage frequency determination submodule and a second information acquisition submodule, wherein:

[0190] The usage frequency determination submodule is used to determine the usage frequency of the application corresponding to the task.

[0191] The second information acquisition submodule is used to acquire screen information of the touch screen of the electronic device and acquire historical action information of the electronic device if the usage frequency does not reach the usage frequency threshold.

[0192] The candidate action obtaining module 230 is used to input the task, the screen information and the historical action information into a large language model to obtain the candidate actions output by the large language model.

[0193] Furthermore, the candidate action obtaining module 230 includes: a target instruction obtaining submodule and a candidate action obtaining submodule, wherein:

[0194] The target instruction acquisition submodule is used to assemble the task, the screen information and the historical action information through an instruction template to obtain a target instruction.

[0195] The candidate action obtaining submodule is used to input the target instruction into the large language model to obtain the candidate action output by the large language model.

[0196] The action execution module 240 is used to control the electronic device to execute the candidate action so that the electronic device completes the task.

[0197] Furthermore, the task processing device 200 further includes: a historical action information updating module, wherein:

[0198] A historical action information updating module is used to update the historical action information based on the candidate action.

[0199] Furthermore, the task processing device 200 further includes: a repeating module, wherein:

[0200] A repeating module is used to repeat the steps of acquiring screen information of the touch screen of the electronic device to controlling the electronic device to execute the candidate action if it is determined that the electronic device has not completed the task, until the electronic device completes the task.

[0201] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0202] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0203] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0204] See also Fig.10 , which shows a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 may be an electronic device capable of running applications, such as a smart phone, a tablet computer, an e-book, etc. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0205] Among them, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 110 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0206] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing functions (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0207] See also Fig.11 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 300 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0208] The computer readable storage medium 300 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer readable storage medium 300 has storage space for program code 310 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 310 can be compressed, for example, in an appropriate form.

[0209] In summary, the task processing method, device, electronic device and storage medium provided in the embodiments of the present application determine the task that the electronic device needs to complete, obtain screen information of the touch screen of the electronic device, and obtain historical action information of the electronic device, wherein the historical action information is determined based on the action that the electronic device has performed for the task, inputs the task, screen information and historical action information into a large language model, obtains candidate actions output by the large language model, and controls the electronic device to execute the candidate actions so that the electronic device completes the task, thereby being able to decompose tasks related to the user interface into actions through the large language model, and then intelligently control the electronic device to execute the decomposed actions to complete the task to be completed, which can reduce the user's work difficulty and workload.

[0210] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A task processing method, characterized in that: The method comprises: Determine the tasks that the electronic device needs to perform; Acquire screen information of a touch screen of the electronic device, and acquire historical action information of the electronic device, wherein the historical action information is determined based on actions that the electronic device has performed for the task; Inputting the task, the screen information, and the historical action information into a large language model to obtain candidate actions output by the large language model; The electronic device is controlled to execute the candidate action so that the electronic device completes the task.

2. The method according to claim 1, characterized in that The acquiring screen information of the touch screen of the electronic device includes: Obtaining a view hierarchy of user interface elements on a touch screen of the electronic device; The view hierarchy is converted into a text modal form to obtain the screen information.

3. The method according to claim 2, characterized in that The converting the view hierarchy structure into a text modal form to obtain the screen information includes: The view hierarchy structure is converted into a form of hypertext markup language HTML to obtain the screen information.

4. The method according to claim 3, characterized in that The converting the view hierarchy into a form of a hypertext markup language HTML to obtain the screen information includes: The view hierarchy is traversed, and corresponding HTML elements are created for the traversed views to obtain the screen information.

5. The method according to claim 4, characterized in that The method further comprises: Get the unique identifier of the traversed view in the hash table, and use the unique identifier of the view in the hash table as the unique identifier of the corresponding HTML element.

6. The method according to any one of claims 1 to 5, characterized in that: The step of inputting the task, the screen information, and the historical action information into a large language model to obtain candidate actions output by the large language model includes: Assembling the task, the screen information and the historical action information through an instruction template to obtain a target instruction; The target instruction is input into the large language model to obtain the candidate action output by the large language model.

7. The method according to any one of claims 1 to 5, characterized in that: After controlling the electronic device to execute the candidate action, the method further includes: The historical action information is updated based on the candidate action.

8. The method according to any one of claims 1 to 5, characterized in that: After controlling the electronic device to execute the candidate action, the method further includes: If it is determined that the electronic device has not completed the task, the steps of acquiring screen information of the touch screen of the electronic device to controlling the electronic device to execute the candidate action are repeated until the electronic device completes the task.

9. The method according to any one of claims 1 to 5, characterized in that: The acquiring screen information of the touch screen of the electronic device and acquiring historical action information of the electronic device includes: Determine the complexity level of the task; If the complexity reaches a complexity threshold, screen information of the touch screen of the electronic device is obtained, and historical action information of the electronic device is obtained.

10. The method according to any one of claims 1 to 5, characterized in that: The acquiring screen information of the touch screen of the electronic device and acquiring historical action information of the electronic device includes: Determine the usage frequency of the application corresponding to the task; If the usage frequency does not reach the usage frequency threshold, screen information of the touch screen of the electronic device is obtained, and historical action information of the electronic device is obtained.

11. A task processing device, characterized in that: The device comprises: A task determination module, used to determine the task that the electronic device needs to complete; an information acquisition module, configured to acquire screen information of a touch screen of the electronic device and acquire historical action information of the electronic device, wherein the historical action information is determined based on actions that the electronic device has performed for the task; A candidate action obtaining module, used for inputting the task, the screen information and the historical action information into a large language model to obtain candidate actions output by the large language model; The action execution module is used to control the electronic device to execute the candidate action so that the electronic device completes the task.

12. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 10.

Images