Display method, electronic device, readable storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and provides a display method, electronic equipment, a readable storage medium and a program product. According to the method, the electronic equipment executes related operation on a first interface element through an intelligent agent, and the intelligent agent is obtained through reinforcement learning training based on a reward function. The reward function is based on the difference between the prediction format and the real format of the response result output by the intelligent agent in the training process, and the prediction operation type and the real operation type, the prediction operation content and the real operation content of the interface element output by the intelligent agent in the training process. According to the method and the device, the intelligent agent is subjected to reinforcement learning training based on the reward function according to the difference between the predicted position and the real position, so that the recognition accuracy of the intelligent agent on the interface elements can be improved, and the problem that the intelligent agent cannot execute a task or an execution result is wrong when executing the task is avoided.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a display method, electronic device, readable storage medium, and program product. Background Technology

[0002] A graphical user interface agent (GUI agent) is an intelligent agent that understands the application interface by "observing" images (such as screenshots) and simulates human clicks and inputs to perform complex tasks (such as ordering takeout, booking hotels, checking information, and buying tickets).

[0003] However, when the GUI agent executes a task, the GUI agent may misunderstand the application interface, which may cause the task to fail to execute or the task execution result to be incorrect. Summary of the Invention

[0004] Some embodiments of this application provide a display method, electronic device, readable storage medium, and program product for solving the problem of a GUI agent failing to perform a task or encountering an error during task execution.

[0005] The present application is described below from multiple aspects, and the embodiments and beneficial effects of these aspects can be referred to each other.

[0006] In a first aspect, this application provides a display method applied to an electronic device, on which an intelligent agent is deployed. The method includes: displaying a first application interface; receiving a first user instruction, the first user instruction being used to instruct the execution of a first operation on a first interface element on the first application interface; inputting the first user instruction and a first image of the first application interface to the intelligent agent, obtaining a first response result output by the intelligent agent, the first response result including a first position of the first interface element, a first operation type, and a first operation content; and executing the first operation on the first interface element by the intelligent agent based on the first position, the first operation type, and the first operation content; wherein the intelligent agent is obtained by reinforcement learning training based on a reward function, and the reward function is based on a first reward function, a second reward function, and a third reward function. The sum of the functions determines the reward function; the first reward function is determined based on the difference between the predicted format of the response output by the agent during training and the actual format of the response, the response including the predicted position of the interface element, the predicted operation type, and the predicted operation content; the second reward function is determined based on the first distance, the difference between the predicted operation type of the interface element output by the agent during training and the actual operation type of the interface element, and the difference between the predicted operation content of the interface element output by the agent during training and the actual operation content of the interface element, wherein the first distance is the distance between the predicted position of the interface element output by the agent during training and the actual position of the interface element; the third reward function is determined based on the second reward function and the first distance.

[0007] In the above display method, an agent performs related operations on the first interface element. This agent is trained based on a reward function. The agent's ability to perform tasks is improved through training with this reward function, avoiding problems such as the agent being unable to complete tasks or producing incorrect task execution results. This is because the reward function is determined by the sum of a first reward function, a second reward function, and a third reward function. The first reward function is determined based on the difference between the predicted format of the agent's output response and the actual format of the response. Training the agent based on the first reward function ensures the accuracy of the agent's output format. The second reward function is determined based on the first distance between the predicted position and the actual position of the interface element, the difference between the predicted operation type and the actual operation type of the interface element, and the difference between the predicted operation content and the actual operation content of the interface element. Training the agent based on the second reward function improves the accuracy of the agent in recognizing the position, operation type, and operation content of interface elements. The third reward function is determined based on the second reward function and the first distance between the predicted position of the interface element and the actual position of the interface element. Training the agent based on the third reward function is equivalent to strengthening the agent's position recognition training, which can further improve the accuracy of the agent in recognizing the position of interface elements.

[0008] After the agent completes reinforcement learning training, the trained agent identifies the first position, first operation type, and first operation content of the first interface element on the first application interface based on the first user command and the first image of the first application interface. Then, based on the first position, first operation type, and first operation content of the first interface element, the agent performs the first operation on the first interface element on the first application interface. This helps the user to automatically perform tasks without manual operation, thus improving the user experience.

[0009] For example, the first reward function can be the format reward function mentioned below, the second reward function can be the exact reward function mentioned below, the third reward function can be the distance reward function mentioned below, and the agent can be the GUI agent mentioned below. The reward function can be determined based on the following formula (1), formula (2), or formula (3).

[0010] In some embodiments, the first reward function is determined based on the difference between the predicted format of the first response result output by the agent during training and the true format of the first response result, including: if the predicted format and the true format are the same, the first reward function takes a first value, which is greater than 0; or, if the predicted format and the true format are different, the first reward function takes a second value, which is less than 0.

[0011] During reinforcement learning training of the agent, the first reward function is greater than 0 when the predicted format of the agent's output response is the same as the true format; conversely, the first reward function is less than 0 when the predicted format of the agent's output response is different from the true format. That is, during training, the agent is positively rewarded when its output response is in the correct format, and negatively penalized when its output response is in the wrong format. This ensures that the agent outputs a correctly formatted response, thereby guaranteeing that the agent performs the correct operation based on the response.

[0012] In some embodiments, the absolute value of the first value is equal to the absolute value of the second value.

[0013] Regarding the training of the agent's accuracy in recognizing the format of interface elements, the absolute values ​​of the first and second values ​​are equal, which ensures the consistency of positive rewards and negative penalties, thereby guaranteeing the training effect of the agent in recognizing the format of interface elements.

[0014] In some embodiments, the second reward function is determined based on the first distance, the difference between the predicted operation type of the interface element output by the agent during training and the actual operation type of the interface element, and the difference between the predicted operation content of the interface element output by the agent during training and the actual operation content of the interface element. Specifically, if the predicted operation type and the actual operation type are the same, the first distance is less than or equal to a distance threshold, and the similarity between the predicted operation content and the actual operation content is greater than or equal to a similarity threshold, then the second reward function takes a third value, which is greater than 0. Alternatively, if the predicted operation type and the actual operation type are different, or the first distance is greater than a distance threshold, or the similarity between the predicted operation content and the actual operation content of the interface element is less than or equal to a similarity threshold, then the second reward function takes a fourth value, which is less than 0.

[0015] During reinforcement learning training of the agent, the second reward function is greater than 0 when the predicted operation type of the interface element output by the agent is the same as the actual operation type of the interface element, the first distance between the predicted position of the interface element output by the agent and the actual position of the interface element is less than or equal to a distance threshold, and the similarity between the predicted operation content of the interface element output by the agent and the actual operation content of the interface element is greater than or equal to a similarity threshold; otherwise, the second reward function is less than 0. That is, during training, when the agent correctly identifies the operation type, position, and operation content of the interface element, it is positively rewarded; when the agent incorrectly identifies the operation type, position, or operation content of the interface element, it is negatively penalized. This improves the accuracy of the agent in recognizing the operation type, position, and operation content of interface elements.

[0016] In some embodiments, the absolute values ​​of the third value and the fourth value are equal.

[0017] Regarding the training of the agent's accuracy in recognizing the operation type, position, and content of interface elements, the absolute values ​​of the third and fourth values ​​are equal, which ensures the consistency of positive rewards and negative penalties, thereby guaranteeing the training effect of the agent in recognizing the operation type, position, and content of interface elements.

[0018] In some embodiments, the third reward function is determined based on the second reward function and the first distance, including: if the value of the second reward function is greater than 0, then the value of the third reward function is the fifth value, the fifth value is less than 0, and the absolute value of the fifth value is positively correlated with the first distance; or, if the value of the second reward function is less than 0, then the value of the third reward function is 0.

[0019] During reinforcement learning training of the agent, when the value of the second reward function is greater than 0, it indicates that the agent has correctly identified the operation type, position, and content of the interface element. In this case, the third reward function is less than 0. Furthermore, the larger the first distance between the predicted position of the interface element output by the agent and its actual position, the larger the absolute value of the third reward function; conversely, the smaller the first distance, the smaller the absolute value of the third reward function. That is, during training, when the agent correctly identifies the operation type, position, and content of the interface element, it is penalized based on the first distance between the predicted position and its actual position. The larger the first distance, the greater the penalty; conversely, the smaller the first distance, the smaller the penalty. This improves the accuracy of the agent's position recognition of interface elements.

[0020] In some embodiments, the distance threshold is the radius of a preset circle centered on the actual position of the interface element; the fifth value is determined based on the ratio between the first distance and the radius of the preset circle.

[0021] For example, when the value of the second reward function is greater than 0, the fifth value can be calculated using the following formula (11).

[0022] The radius of the preset circle is related to the image size of the application interface input to the intelligent agent and the operation type of the interface elements.

[0023] For example, if the operation type of the interface element on the application interface is a click operation, a long press operation, or a input operation, then the value of the preset circle radius can be a first ratio of the smaller of the width and height of the application interface image. If the operation type of the interface element on the application interface is a swipe operation or a drag operation, then the value of the preset circle radius can be a second ratio of the smaller of the width and height of the application interface image. The first ratio is greater than the second ratio.

[0024] It's understandable that the operational space on an application interface is limited. When the operation type of an interface element on the application interface is a swipe or drag operation, setting the radius of the preset circle at a smaller ratio, that is, setting a smaller distance threshold, can provide more effective operational space for swipe and drag operations, ensuring that the agent can smoothly perform swipe or drag operations on the interface element.

[0025] In some embodiments, the first operation type includes a click operation, a long press operation, a swipe operation, an input operation, or a drag operation; the first operation content includes the number of clicks, the duration of the long press, the swipe direction, the input text, or the drag direction.

[0026] In some embodiments, the agent is trained on a first training set, which is obtained by deleting at least one training sample from a second training set; wherein, after any one of the at least one training sample is input into the agent, the distance between any one of the multiple predicted positions of the interface element output by the agent and the actual position of the interface element is less than or equal to a distance threshold, or the distance between any one of the multiple predicted positions of the interface element output by the agent and the actual position of the interface element is greater than the distance threshold.

[0027] It can be understood that if the distance between the predicted position and the actual position of an interface element is less than or equal to the distance threshold, it means that the position of the interface element recognized by the intelligent agent is correct. Conversely, if the distance between the predicted position and the actual position of an interface element is greater than the distance threshold, it means that the position of the interface element recognized by the intelligent agent is incorrect.

[0028] During the training of the agent, at least one training sample in the second training set that the agent correctly or incorrectly identifies multiple positions is deleted, and only the training samples that the agent partially or partially identifies multiple positions are obtained to form the first training set. Then, the agent is trained using reinforcement learning on the first training set, which can improve the reinforcement learning training effect of the agent.

[0029] In some embodiments, the reward function is the reward function in a Markov process model, which is established based on user instructions and an application interface as the interaction environment. For example, the Markov process model can be the model shown in the following formula (12).

[0030] Training agents using reinforcement learning based on Markov process models can improve the training effect of agents, thereby enhancing their ability to perform tasks.

[0031] Secondly, this application provides an electronic device, including: a memory for storing instructions; and a processor, which, when executing the instructions in the memory, causes the electronic device to perform the method provided in the first aspect of this application. The beneficial effects achievable in the second aspect can be found in the beneficial effects of the method provided in any embodiment of the first aspect, and will not be repeated here.

[0032] Thirdly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method provided in any embodiment of the first aspect. The beneficial effects achievable through this third aspect can be referenced to the beneficial effects of the method provided in any embodiment of the first aspect, and will not be repeated here.

[0033] Fourthly, this application provides a computer program product including computer program code. When the computer program code is run on a computer, it causes the computer to implement the method provided in any embodiment of the first aspect. The beneficial effects achievable in this fourth aspect can be found in the beneficial effects of the method provided in any embodiment of the first aspect, and will not be repeated here.

[0034] Fifthly, this application provides a chip including a processor for reading and executing a computer program stored in a memory to perform the method provided in any embodiment of the first aspect. The beneficial effects achievable through the fifth aspect can be found in the beneficial effects of the method provided in any embodiment of the first aspect, and will not be repeated here. Attached Figure Description

[0035] Figure 1 According to some embodiments of this application, a schematic diagram of agent reinforcement learning is shown;

[0036] Figure 2According to some embodiments of this application, a schematic diagram of the input and output of a GUI agent is shown;

[0037] Figures 3A to 3C According to some embodiments of this application, an exemplary application scenario diagram is shown;

[0038] Figure 4 According to some embodiments of this application, another exemplary application scenario diagram is shown;

[0039] Figures 5A to 5D According to some embodiments of this application, another exemplary application scenario diagram is shown;

[0040] Figure 6 According to some embodiments of this application, another exemplary application scenario diagram is shown;

[0041] Figures 7A to 7C According to some embodiments of this application, a schematic diagram of the reinforcement learning training process of a GUI agent is shown;

[0042] Figure 8 According to some embodiments of this application, a schematic diagram of a reinforcement learning training process is shown;

[0043] Figure 9 According to some embodiments of this application, a flowchart of a display method is shown;

[0044] Figure 10 According to some embodiments of this application, a schematic diagram of the structure of an electronic device is shown. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are all within the protection scope of this application.

[0046] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0047] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0048] To facilitate understanding, the categories involved in this application will be introduced below.

[0049] (1) Reinforcement learning

[0050] Reinforcement learning (RL), also known as reward learning, evaluation learning, or reinforcement learning, is used to describe and solve the problem of how an agent learns strategies to maximize rewards or achieve specific goals during its interaction with the environment.

[0051] Reinforcement learning is a method of learning where an agent learns through trial and error. Rewards obtained through interactions with the environment guide behavior, with the goal of maximizing the agent's reward. Reinforcement learning does not require a training dataset. In reinforcement learning, the reinforcement signals (i.e., rewards) provided by the environment evaluate the quality of an action, rather than telling the reinforcement learning system how to produce the correct action. Because the external environment provides limited information, the agent must learn through its own experience. In this way, the agent learns by taking action... Knowledge is gained from an environment that is evaluated (i.e. rewarded), and action plans are improved to adapt to the environment.

[0052] like Figure 1 As shown, reinforcement learning can include five elements: agent, environment, state, action, and reward. The agent's input is its state, and its output is its action.

[0053] The training process of reinforcement learning can be as follows: the agent interacts with the environment multiple times, obtaining the action, state, and reward for each interaction. These multiple sets of (action, state, reward) are then used as training data to train the agent once. This process is repeated for the next round of training until the convergence condition is met, at which point training ends.

[0054] The process of obtaining the action, state, and reward of an interaction is as follows: Figure 1As shown, the current state s(t) of the environment is input to the agent, and the action a(t) output by the agent is obtained. The reward r(t) for this interaction is calculated based on the relevant performance indicators of the environment under the action a(t). Thus, the action a(t), action a(t), and reward r(t) for this interaction are obtained. The action a(t), action a(t), and reward r(t) of this interaction are recorded for subsequent training of the agent. Furthermore, the next state s(t+1) of the environment under the action a(t) can also be recorded to enable the next interaction between the agent and the environment. Here, t represents time.

[0055] (2) Intelligent agent

[0056] An intelligent agent is an entity capable of thinking and interacting with its environment. For example, an intelligent agent can be a computer system or part of a computer system in a specific environment. Based on its own perception of the environment, an intelligent agent can autonomously complete set goals within its environment, following existing instructions or through self-learning.

[0057] (3) Markov decision process (MDP) model (or finite Markov decision process model)

[0058] The MDP model is a model used for reinforcement learning, specifically a mathematical model for analytical decision problems based on discrete-time stochastic control. An MDP model includes an agent and the environment in which the agent interacts.

[0059] In the MDP model, the agent observes the state of the environment, makes decisions (also known as actions) based on the current state of the environment, interacts with the environment to obtain the next state and reward, and then performs reinforcement learning based on the obtained reward.

[0060] The method provided in this application can be applied to any electronic device deployed with a GUI agent, including but not limited to mobile stations (MS) and mobile terminals (MT). For example, the electronic device can be a mobile phone, smart TV, wearable device, tablet computer, desktop computer, laptop computer, virtual reality (VR) device, augmented reality (AR) device, terminal in industrial control, terminal in self-driving, terminal in remote medical surgery, terminal in smart grid, terminal in transportation safety, terminal in smart city, terminal in smart home, etc. This application does not limit the specific form of the electronic device.

[0061] Figure 2 According to some embodiments of this application, a schematic diagram of the input and output of a GUI agent is shown.

[0062] like Figure 2 As shown, when the GUI agent on an electronic device interacts with user commands and application interfaces as the interaction environment, the electronic device can input user commands and images of the application interface as input data into the GUI agent and obtain the response results output by the GUI agent. The response results can include information such as the operation type, operation type, and position of the interface elements on the application interface.

[0063] Operation types include, but are not limited to, click, long press, swipe, input, and drag. Operation content includes, but is not limited to, the number of clicks, the duration of the long press, the direction of the swipe, the text entered, and the direction of the drag.

[0064] In some embodiments, in scenarios where a GUI agent on an electronic device interacts with the application interface currently displayed on the electronic device in response to user commands, the electronic device can input the received user command and an image (such as a screenshot) of the currently displayed application interface into the GUI agent. The GUI agent performs semantic recognition on the user command and the image of the application interface to determine the operation type, operation content, and location of the interface element that needs to be operated on the application interface. Then, based on the operation type, operation content, and location of the interface element, the GUI agent executes the operation indicated by the user command on the interface element.

[0065] It's understandable that if the GUI agent accurately identifies the element information, its task execution success rate will be high. Conversely, if the GUI agent's accuracy in identifying the element information is low, its task execution success rate will be low.

[0066] The following section uses a mobile phone as an example and YOYO as the GUI agent to introduce the application scenarios of GUI agents in detail.

[0067] Figures 3A to 3C An exemplary application scenario diagram is shown according to some embodiments of this application.

[0068] like Figures 3A to 3C As shown, mobile phone 100 displays the price interface U1 of the group-buying application. Price interface U1 displays the prices of three different packages for a superior double room at Hotel A on January 24th. Mobile phone 100 responds to the user's wake-up operation, such as entering a voice wake-up phrase (e.g., "Hello, YOYO") or triggering a shortcut key to wake up YOYO (e.g., a custom long press of the power button to wake up YOYO), waking up YOYO and displaying YOYO's chat window C1.

[0069] After receiving a voice command from a user stating, "I want to book a superior double room at Hotel A on January 24th, including two breakfasts, for 560 yuan," the phone 100 can input this voice command and a screenshot of the price interface U1 into YOYO. YOYO then identifies the booking control K1 on the price interface U1, along with its element information, such as the operation type (click), the action (one click), and its location. YOYO then clicks the booking control K1 on the price interface U1, opening the payment interface U2 of the group-buying app. Finally, YOYO identifies and clicks the quick payment control K2 on the payment interface U2, completing the hotel booking. This reduces user actions and makes hotel booking more convenient.

[0070] However, in some cases, YOYO identifies incorrect element information in the booking control K1, causing YOYO to be unable to complete the hotel booking task. For example, ... Figure 4 As shown, YOYO recognizes that the reservation control K1 is at position a. Since position a is outside the touch range of control K1, YOYO cannot click on the reservation control K1, thus preventing YOYO from completing the task of booking a hotel.

[0071] Figures 5A to 5DAccording to some embodiments of this application, another exemplary application scenario diagram is shown.

[0072] like Figures 5A to 5D As shown, in response to YOYO being woken up, phone 100 displays YOYO's chat window C1. When phone 100 receives a user voice command stating, "I want to know the detailed opening hours and reservation information for attraction B," phone 100 can input the user's voice command and a screenshot of desktop 10 into YOYO. Then, YOYO, based on the user's voice command and the screenshot of desktop 10, identifies the icon 11 of the group-buying application that needs to be operated on desktop 10, as well as the element information of icon 11, such as the operation type of icon 11 being a click, the operation content being a single click, and the location of icon 11. Then, based on the identified element information of icon 11, YOYO clicks the group-buying application icon 11, opening the group-buying application's search interface U3.

[0073] Next, YOYO identifies the search bar 12 on the search interface U3 and its element information, such as the operation type of search bar 12 being text input, the operation content being the text "attraction B" to be entered, and its location. Then, based on the identified element information of search bar 12, YOYO inputs the text "attraction B" into search bar 12.

[0074] Then, YOYO identifies the search control K3 within the search bar 12 and its element information, such as the operation type of search control K3 being a click, the operation content being one click, and the location of search control K3. Based on the identified element information of search control K3, YOYO clicks on search control K3. The group-buying app responds to YOYO's search for attraction B, searches for attraction B, and displays the search results interface U4 for attraction B. The search results interface U4 displays detailed opening hours and reservation information for attraction B. In this way, YOYO can automatically help users find information about attraction B without requiring manual operation, improving the user experience.

[0075] However, in some cases, YOYO identifies incorrect element information in the reservation control K1, causing YOYO to be unable to complete the task of querying attraction B. For example, ... Figure 6 As shown, YOYO recognizes that the search control K3 is located at position b. Since position b is outside the touch range of the search control K3, YOYO cannot click on the search control K3, thus preventing YOYO from completing the task of searching for attraction B.

[0076] In view of this, embodiments of this application provide a display method that uses a GUI agent to perform related operations on interface elements. This GUI agent is trained using reinforcement learning based on a reward function. The reinforcement learning training improves the accuracy of the GUI agent's recognition of interface elements, thereby ensuring that the GUI agent can successfully complete the tasks instructed by the user.

[0077] In some embodiments, the reward function is determined based on the sum of a format reward function (as the first reward function), a precise reward function (as the third reward function), and a distance reward function (as determined by the third reward function).

[0078] In some embodiments, the formula for the reward function is shown in formula (1) below:

[0079] (1)

[0080] in, For the reward function, For the formatted reward function, For a precise reward function, This is the distance reward function.

[0081] In other embodiments, the formula for the reward function may be in other forms, such as the following formulas (2) and (3), etc., without limitation.

[0082] (2)

[0083] in, For other types of reward functions, such as preset fixed rewards, there are no restrictions.

[0084] (3)

[0085] in, , and The weighting coefficient is greater than 0 and less than 1.

[0086] The following describes the format reward function. Precise reward function and distance reward function Let me introduce it.

[0087] 1. Formatted reward function

[0088] In some embodiments, the format reward function It is determined based on the difference between the predicted format of the response output by the GUI agent during reinforcement learning training and the actual format of the interface element. In other words, the predicted format is the format of the response that the GUI agent predicts to output based on the input user command and the image of the application interface, while the actual format is the actual format of the GUI agent's response corresponding to the user command and the image of the application interface.

[0089] For example, if the predicted format of the response output by the GUI agent is the same as the true format of the response, then the format reward function... The value of is the first value, where the first value is greater than 0. Alternatively, if the predicted format of the response output by the GUI agent differs from the true format of the response, then the format reward function... The value is the second value, where the second value is less than 0.

[0090] It is understandable that the first value can be any empirical value greater than 0, such as +1, +2, +3, etc., without restriction. The second value can be any empirical value less than 0, such as -1, -2, -3, etc., without restriction.

[0091] In some embodiments, the absolute value of the first value is equal to the absolute value of the second value. For example, if the first value is +1, then the second value can be -1.

[0092] It can be understood that during reinforcement learning training, when the predicted format of the response output by the GUI agent is the same as the actual format of the response, the first reward function takes a value greater than 0, indicating a positive reward for the GUI agent. When the predicted format of the response output by the GUI agent is different from the actual format of the response, the first reward function takes a value less than 0, indicating a negative penalty for the GUI agent. In this reinforcement learning training process, the absolute values ​​of the first and second values ​​are equal, ensuring consistency between positive rewards and negative penalties, thereby improving the training effect of the GUI agent. It can be understood that the response refers to the predicted result output by the GUI agent in response to input data (such as user commands and images from the application interface). The format of the response refers to the format of the predicted result output by the GUI agent, such as a fast inference format and a pre-planned guided inference format. Fast inference formats can be in the form of ActionType(coordinate:(x,y),content), where ActionType represents the basic operation type (such as click, input, etc.), coordinate represents the position coordinates of the operation (x, y), and content represents the content of the operation (such as the input text or scroll direction). Planning-oriented inference formats can include... <think> ...< / think> and <answer> ...< / answer> The format of these two pairs of Hypertext Markup Language (HTML) tags. Among them, <think> ...< / think> Tags represent the reasoning chain, analysis steps, and logical thinking of the GUI agent. <answer> ...< / answer> The tagged blog contains answers obtained through GUI agent reasoning.

[0093] 2. Precise reward function

[0094] In some embodiments, the exact reward function It is determined based on the first distance between the predicted position of the interface element output by the GUI agent during reinforcement learning training and the actual position of the interface element, the difference between the predicted operation type of the interface element output by the GUI agent during reinforcement learning training and the actual operation type of the interface element, and the difference between the predicted operation content of the interface element output by the GUI agent during reinforcement learning training and the actual operation content of the interface element.

[0095] This can be understood as follows: Predicted position is the location of the UI element predicted by the GUI agent, while the actual position is the actual location of the UI element. Predicted operation type is the operation type of the UI element predicted by the GUI agent, while the actual operation type is the actual operation type of the UI element. Predicted operation type is the operation content of the UI element predicted by the GUI agent, while the actual operation content is the actual operation content of the UI element.

[0096] In some embodiments, the exact reward function is defined when the GUI agent simultaneously satisfies the following first, second, and third conditions during reinforcement learning training. The value is the third value; otherwise, it is the fourth value, where the third value is greater than 0 and the fourth value is less than 0.

[0097] It is understandable that the third value can be any empirical value greater than 0, such as +1, +2, +3, etc., without any restriction. The fourth value can be any empirical value less than 0, such as -1, -2, -3, etc., without any restriction.

[0098] In some embodiments, the absolute value of the third value is equal to the absolute value of the fourth value. For example, if the third value is +2, then the fourth value can be -2.

[0099] It is understandable that in this reinforcement learning training process, the absolute value of the third value is equal to the absolute value of the fourth value, which can ensure the consistency between positive rewards and negative penalties, thereby improving the reinforcement learning training of the GUI agent and achieving better training results.

[0100] The first, second, and third conditions are described below:

[0101] (1) First condition

[0102] During reinforcement learning training, the predicted operation type of the UI element output by the GUI agent is the same as the actual operation type of that UI element.

[0103] (2) Second condition

[0104] During reinforcement learning training, the predicted position of the interface element output by the GUI agent is less than or equal to a distance threshold.

[0105] In some embodiments, the distance threshold can be the radius of a preset circle centered on the actual position of the interface element. Alternatively, it can be any other value. It can be understood that the actual position of a UI element can be the location of its center point, or it can be the location of other points within the touch area of ​​the UI element; there are no restrictions on this.

[0106] In some embodiments, the radius of the preset circle is... The size is related to the size of the image (such as a screenshot) input to the application interface of the GUI agent during reinforcement learning training, as well as the operation type of the interface elements.

[0107] For example, if the operation type of the interface element on the application interface is click operation, long press operation, or input operation, then the radius of the preset circle is... The value can be a first ratio of the smaller of the width and height of the image on the application interface. If the operation type of the interface element on the application interface is a swipe or drag operation, then the radius of the preset circle is... The value of can be a second ratio of the smaller of the width and height of the image of the application interface. The first ratio is greater than the second ratio.

[0108] It is understood that the first and second percentages can be any empirical values, and there are no restrictions on them. For example, the first percentage can be 14%, 15%, or 16%, and the second percentage can be 7.5%, 8%, or 8.5%, etc.

[0109] It's understandable that the effective space for input from the application interface to the GUI agent for swipe and drag operations is limited. When the operation type of an interface element is a swipe or drag operation, the radius of the preset circle is set to a smaller scale. In other words, setting a smaller distance threshold can provide more effective operating space for swipe and drag operations, ensuring that the GUI agent can smoothly perform swipe or drag operations on the interface element.

[0110] (3) Third condition

[0111] During reinforcement learning training, the similarity between the predicted operation content of the interface element output by the GUI agent and the actual operation content of the interface element is greater than or equal to the similarity threshold.

[0112] In some embodiments, if the predicted operation content is the number of clicks, the duration of a long press, the direction of a swipe, or the direction of a drag, the similarity threshold can be 1, indicating that the predicted operation content of the interface element output by the GUI agent during the reinforcement learning training process is the same as the actual operation content of the interface element.

[0113] In other embodiments, if the predicted operation content is text, the similarity threshold can be an empirical value greater than 0 and less than or equal to 1. For example, the similarity threshold can be 0.5, 0.6, 0.7, etc., and there is no limitation thereto.

[0114] For example, Table 1 below shows the precise reward conditions corresponding to different action types (operation types).

[0115] Table 1

[0116]

[0117] It is understandable that the actions performed by the GUI agent in response to user commands to execute corresponding tasks can be operations on interface elements on the application interface, such as click actions, long press actions, swipe actions, text input actions, and drag actions as shown in Table 1 above, as well as other actions, such as opening the application, closing the application, returning actions, completing actions, desktop actions (actions performed on the desktop), completing actions, waiting actions, entering actions, user takeover actions (actions that terminate the execution of the task and prompt the user to take over), screenshot actions, and long screenshot actions as shown in Table 1 above.

[0118] As shown in Table 1 above, a click action can be executed by the GUI agent calling the `tap(x,y)` function. A long press action can be executed by the GUI agent calling the `long_press(x,y)` function. A swipe action can be executed by the GUI agent calling the `scroll(x,y,dir)` function. A text input action can be executed by the GUI agent calling the `text(x,y,input)` function. A drag action can be executed by the GUI agent calling the `drag(x1,y)` function. 1, The actions performed by the function (x2, y2) are as follows: Return action: This can be an action performed by the GUI agent calling the `navigate_back()` function. Desktop action: This can be an action performed by the GUI agent calling the `navigate_home()` function. Completion action: This can be an action performed by the GUI agent calling the `finish()` function. Waiting action: This can be an action performed by the GUI agent calling the `wait()` function. Enter action: This can be an action performed by the GUI agent calling the `enter()` function. User takeover action: This can be an action performed by the GUI agent calling the `takeover()` function. Screenshot action: This can be an action performed by the GUI agent calling the `screenshot()` function. Long screenshot action: This can be an action performed by the GUI agent calling the `long_screenshot()` function.

[0119] As shown in Table 1 above, different actions have different reward logics and precise reward conditions, as detailed below:

[0120] (1) Click action and long press action

[0121] During reinforcement learning training, for interface elements on the application interface whose action type is click and long press, the GUI agent can be precisely rewarded when the precise reward condition corresponding to the following formula (4) is met.

[0122] (4)

[0123] in, The coordinates representing the predicted position of the UI element. The coordinates representing the actual position of the UI element (such as the position of the center point of the UI element). H represents the distance between the predicted position of an interface element and its actual position, where H represents the minimum value of the width and height of the application interface image.

[0124] (2) Sliding motion

[0125] During reinforcement learning training, for interface elements on the application interface whose action type is sliding, when the precise reward conditions corresponding to the above formula (4) and the following formula (5) are met, the GUI agent can be precisely rewarded.

[0126] (5)

[0127] in, Indicates the predicted sliding direction. Indicates the actual sliding direction.

[0128] It can be understood that formula (5) means that the sliding direction of the interface element predicted by the GUI agent is the same as the actual sliding direction of the interface element.

[0129] (3) Text input action

[0130] During reinforcement learning training, for interface elements on the application interface whose action type is text input, when the precise reward conditions corresponding to the above formula (4) and the following formula (6) are met, the GUI agent can be precisely rewarded.

[0131] (6)

[0132] in, This indicates that the GUI agent predicts the input text. This indicates the actual text that the GUI agent needs to input. This represents the similarity between the text predicted by the GUI agent and the actual text that the GUI agent needs to input.

[0133] (4) Dragging action

[0134] During reinforcement learning training, for interface elements on the application interface whose action type is text input, the GUI agent can be accurately rewarded when the following formulas (7) and (8) are satisfied.

[0135] (7)

[0136] in, The coordinates of the predicted position of the drag start point of the interface element. This represents the coordinates of the actual starting point of the dragging of this interface element. This represents the distance between the predicted position of the drag start point of an interface element and the actual position of the drag start point of that interface element.

[0137] (8)

[0138] in, The coordinates of the predicted position of the drag endpoint of the interface element. This represents the coordinates of the actual position of the drag endpoint of the interface element. This represents the distance between the predicted position of the drag endpoint of an interface element and the actual position of the drag endpoint of that interface element.

[0139] (5) Open / close application actions

[0140] During reinforcement learning training, for actions such as opening or closing an application, the GUI agent can be precisely rewarded when the following formulas (9) and (10) are satisfied.

[0141] (9)

[0142] in, Indicates the prediction name of the application. This indicates the actual name of the application.

[0143] It can be understood that formula (9) means that the name of the application predicted by the GUI agent to be opened or closed is consistent with the name of the application that actually needs to be opened or closed.

[0144] (10)

[0145] in, This indicates that the application is open. This indicates that the application is closed. The symbol " / " represents a real action, and " / " represents "or".

[0146] It can be understood that formula (10) represents the predicted action of the GUI agent, such as opening or closing the application, which is the same as the actual action that the GUI agent needs to perform.

[0147] Specifically, during reinforcement learning training, if the GUI agent predicts that the action of opening or closing an application needs to be performed is consistent with the actual action that the GUI agent needs to perform, and the name of the application that the GUI agent predicts needs to open or close is consistent with the name of the application that the GUI agent actually needs to open or close, then the GUI agent can be accurately rewarded.

[0148] (6) Other actions

[0149] During reinforcement learning training, for actions such as returning, going to the desktop, completing, waiting, pressing Enter, user takeover, taking a screenshot, and taking a long screenshot, if the GUI agent's predicted action matches the actual action, the GUI agent can be accurately rewarded. However, if the GUI agent's predicted action does not match the actual action, the GUI agent will not be accurately rewarded.

[0150] 3. Distance reward function

[0151] In some embodiments, the distance reward function Based on the precise reward function The distance is determined by the first distance mentioned above.

[0152] For example, if the exact reward function If the value of is greater than 0, then the distance to the reward function The value is the fifth value, where the fifth value is less than 0 and the absolute value of the fifth value is positively correlated with the first distance. Alternatively, if the exact reward function... If the value of is less than 0, then the distance to the reward function The value of is 0.

[0153] In some embodiments, the fifth value is based on the aforementioned first distance and the radius of the aforementioned preset circle. The ratio between them is determined.

[0154] For example, distance reward function The formula is shown in formula (11) below:

[0155] (11)

[0156] in, For The radius of a preset circle with center at the center.

[0157] It is understandable that the predicted position of interface elements is... The actual position of the interface element The first distance between .

[0158] From formula (11), it can be seen that when the exact reward function... At that time, distance reward function When the exact reward function At that time, distance reward function .

[0159] When the predicted position of a UI element output by the GUI agent matches the actual position of that UI element, the distance reward function... This indicates that the GUI agent will not be penalized.

[0160] When the predicted position of an interface element output by the GUI agent is inconsistent with the actual position of that interface element, the distance reward function... This indicates a penalty applied to the GUI agent. Furthermore, the greater the distance between the predicted and actual positions (i.e., the greater the prediction deviation), the greater the penalty; conversely, the smaller the distance, the smaller the prediction deviation, and the smaller the penalty. In this way, the GUI agent can be appropriately penalized based on its deviation in recognizing the positions of interface elements, thereby improving the accuracy of its element recognition.

[0161] For example, for the reward function R shown in formula (1) above, during the reinforcement learning training process of the GUI agent, if the predicted format of the response output by the GUI agent is the same as the true format, then the format reward function... ;otherwise, If the GUI agent satisfies the first, second, and third conditions simultaneously during reinforcement learning training, then the exact reward function... Otherwise, precise reward function If the exact reward function Then the distance reward function If the exact reward function Then the distance reward function .

[0162] The following is an exemplary description of the reinforcement learning training process for a GUI agent.

[0163] Figures 7A to 7CAccording to some embodiments of this application, a schematic diagram of a reinforcement learning training process for a GUI agent is shown.

[0164] like Figures 7A to 7C As shown, mobile phone 100 displays the product list interface U5 of a food delivery app. Mobile phone 100 responds to the user's long press of the power button to wake up YOYO and displays YOYO's dialog window C1. Mobile phone 100 responds to a user's voice command, "I want to view and select other sorting methods through the comprehensive sorting button to better browse product information," and inputs the user's voice command and a screenshot M1 of the product list interface U5 to YOYO. YOYO responds to the user's voice command and the screenshot M1 of the product list interface U5, and outputs the response result shown in the dialog bar E2. The format of this response result is a planning-oriented reasoning format, including YOYO's reasoning process and reasoning results, specifically including YOYO's observations, plans, and responses regarding the product list interface U5.

[0165] The observations from U5 regarding the product list interface output by YOYO are as follows: The screenshot shows a product list interface of a mobile shopping or food delivery app, with a search box and filter buttons at the top. A prominent "Comprehensive Sort" button is located in the middle of the interface, next to a drop-down arrow suggesting that more sorting options can be expanded. Multiple product entries are displayed below the product list, each containing information such as store name, rating, distance, and price. The user's task is to click the "Comprehensive Sort" button to view and select other sorting methods for better browsing of product information. YOYO's output is: tap (583, 178), indicating that a click operation is performed at coordinates (583, 178).

[0166] During the aforementioned reinforcement training process, the predicted operation type for the comprehensive sorting button K4 on the product list interface U5 output by YOYO was "click," the predicted operation content was "click once," and the predicted position coordinates were (583, 178). Furthermore, as... Figure 7C As shown, the first distance L between the predicted position and the actual position of the comprehensive sorting button K4 is less than or equal to the distance threshold. Since the predicted format of the response output by YOYO is the same as the true format, the format reward function... Because the predicted operation type and the actual operation type of the comprehensive sorting button K4 output by YOYO are the same, the predicted operation content is the same as the actual operation content, and the first distance L between the predicted position and the actual position is less than or equal to the distance threshold. Precise reward function Due to the precise reward function Distance reward function .

[0167] In some embodiments, the reward function represented by any one of the above formulas (1) to (3) It can be the reward function in an MDP model built with user commands and application interface as the interaction environment. That is, the process of the GUI agent performing tasks is constructed into an MDP model, and then the GUI agent is trained by reinforcement learning based on the MDP model.

[0168] The following section introduces the MDP model, which uses user commands and application interfaces as the interaction environment.

[0169] In some embodiments, the MDP model can be expressed as the following formula (12).

[0170] (12)

[0171] in, This represents the MDP model. This represents the state space, which consists of user commands and images of the application interface. This represents the action space, which is the set of actions for the GUI agent. Represents the state transition function. Represents the initial state distribution. This represents the finite time step for each task. The state transition function is also included. The probability is defined as the probability that the application interface will transition from its current state to the next state after the GUIagent performs an action on the application interface. This probability is within the range [insert range here]. Within the range. Reward function It is a rule-based reward function that indicates the effectiveness and accuracy of an action in completing a task.

[0172] Understandable, for single-step tasks , This represents the state space of the application interface image and single-step user commands. For multi-step tasks, H > 1. This represents the application interface's visuals and the state space for multi-step user commands. A single-step task is a task that requires only one operation, and a single-step user command is a user instruction that instructs to perform an operation once. A multi-step task is a task that requires multiple operations, and a multi-step user command is a user instruction that instructs to perform multiple operations.

[0173] It is understandable that at each time step The GUI agent observes a spatial state. And select actions based on the learned strategies. Then receive a rule-based reward. During reinforcement learning training, the goal of GUIagent is to find an efficient policy. In order to maximize expected return In other words, during the reinforcement learning process of the GUI agent, when the reward... Training ends when the maximum value is reached, resulting in a GUI agent trained through reinforcement learning.

[0174] In some embodiments, when training a GUI agent using reinforcement learning, the training data of the GUI agent can be filtered to improve the training effect of the GUI agent.

[0175] The following section introduces the methods for filtering training data for the GUI agent.

[0176] During reinforcement learning training of a GUI agent, a training sample (such as user commands and images of the application interface used for reinforcement learning training) is input into the GUI agent. The GUI agent can perform multiple inferences and output multiple prediction results. If all the prediction results output by the GUI agent are correct, it means that the GUI agent has already achieved a good recognition effect on the training sample, and the training sample cannot further improve the GUI agent's recognition effect, thus having low value for reinforcement learning training. If all the prediction results output by the GUI agent are incorrect, it means that the GUI agent cannot recognize the training sample at all, and cannot improve the GUI agent's recognition effect, also having low value for reinforcement learning training. If some of the prediction results output by the GUI agent are correct and some are incorrect, it means that the GUI agent can recognize the training sample, but the recognition is rather vague, and the training sample has some training value for the GUI agent.

[0177] It can be understood that the training samples for the GUI agent can be divided into the following three categories:

[0178] (1) First type of training samples: samples whose multiple prediction results are all correct after being input into the GUI agent.

[0179] (2) Second type of training samples: Samples in which all prediction results output by the GUI agent are wrong after being input into the GUI agent.

[0180] (3) Third type of training samples: samples whose prediction results are partially correct and partially incorrect after being input into the GUI agent.

[0181] Since the training value of the first and second types of training samples is low, while the training value of the third type of training samples is high, in some embodiments, when training the GUI agent using reinforcement learning, the first and second types of training samples can be deleted, while the third type of training samples can be retained. The GUI agent can then be trained using the third type of training samples to obtain a GUI agent with better training performance.

[0182] For example, Figure 8 According to some embodiments of this application, a schematic diagram of a reinforcement learning training process is shown.

[0183] like Figure 8 As shown, the training samples are Figure 7B In the product list interface U5, image M1 and the user command "I want to view other sorting methods through the comprehensive sorting button to better browse product information" are used. The label (i.e., the range of correct answers) corresponding to this training sample is: tap(460<x<700, 150<y<220), indicating that the position where the horizontal coordinate is within the range of 460<x<700 and the vertical coordinate is within the range of 150<y<220 is clicked. When this training sample is input into the GUI agent for training, if the GUI agent outputs the first and second sets of prediction results, the training sample is deleted. If the GUI agent outputs the third set of prediction results, the training sample is retained.

[0184] The specific reasons are as follows:

[0185] The first set of predictions includes tap(500,200), tap(520,180), and tap(520,210). The coordinates of these three predictions are all within the coordinate range corresponding to the label tap(460<x<700,150<y<220), and they are all correct predictions. This indicates that the training samples in this training belong to the first type of training samples and have no training value, so they should be deleted.

[0186] The second set of predictions includes tap(450,140), tap(440,230), and tap(430,240). The coordinates of these three predictions are not within the coordinate range corresponding to the label tap(460<x<700,150<y<220), and are therefore incorrect predictions. This indicates that the training samples used in this training belong to the second type of training samples and have no training value, and should be deleted.

[0187] The third set of predictions includes tap(510,200), tap(720,180), and tap(515,205). Among them, tap(510,200) and tap(515,205) are within the coordinate range corresponding to the label tap(460<x<700,150<y<220), which are correct predictions. However, tap(720,180) is not within the coordinate range corresponding to the label tap(460<x<700,150<y<220), which are incorrect predictions. This indicates that the training samples in this training belong to the third type of training samples, have certain training value, and should be retained.

[0188] In some embodiments, during the reinforcement learning training of the GUI agent, the sampling temperature of the GUI agent can be set to control the number of prediction results output by the GUI agent after each training sample is input to the GUI agent. For example, the GUI agent can be controlled to output 6 prediction results. Then, based on the prediction results output by the GUI agent, the training samples of the GUI agent are subjected to the aforementioned double filtering, and only the third type of training samples are included for training.

[0189] It can be understood that sampling temperature is a parameter used to control the randomness or determinism of the probability distribution of the GUI agent output. The number of prediction results output by the GUI agent can be adjusted by adjusting the sampling temperature of the GUI agent.

[0190] The following will continue to combine Figure 8 The process of reinforcement learning training is introduced.

[0191] like Figure 8 As shown, the models in the reinforcement learning training process include a pre-trained reference model and a reinforcement learning policy model. The pre-trained reference model is the pre-trained model, such as a pre-trained GUI agent. The reinforcement learning policy model is the model obtained by training the pre-trained model using reinforcement learning, for example, the GUI agent obtained by training the pre-trained GUI agent using reinforcement learning. A constraint relationship is established between the pre-trained reference model and the reinforcement learning policy model through Kullback-Leibler divergence (KL divergence) to avoid excessive differences between the output of the reinforcement learning policy model and the output of the pre-trained reference model, thus ensuring training stability.

[0192] During reinforcement learning training, the effectiveness and accuracy of the reinforcement learning policy model's actions (such as clicks and swipes) in the GUI environment are evaluated in real time by a dedicated spatial capability-enhanced rule-based reward function (as described above). This function generates a scalar reward signal that precisely quantifies the quality of the action (e.g., the proximity of the click location to the target area), thereby optimizing the policy gradient of the reinforcement learning policy model. It can be understood that by preserving the capabilities of the pre-trained reference model through KL divergence while optimizing the policy gradient of the reinforcement learning policy model algorithm, the training crashes can be prevented.

[0193] In some embodiments, static or dynamic filtering can be used to perform dual filtering on the training samples, filtering out the first and second types of training samples and retaining the third type. Since the third type of training samples are those that the GUI agent cannot accurately identify, using the third type of training samples for reinforcement learning training of the GUI agent can prevent the GUI agent from converging prematurely during reinforcement learning training, thereby enabling the GUI agent to achieve better reinforcement learning training results.

[0194] Static filtering refers to filtering out the first and second types of training samples before training begins. Dynamic filtering refers to filtering out the first and second types of training samples during training.

[0195] The following introduces two training methods for guided reinforcement policy optimization (GRPO).

[0196] (1) GRPO training method based on token-level importance sampling

[0197] During reinforcement learning training, the GUI agent generates a set of responses based on each state space S (a state space composed of user commands and images of the application interface). Then each reply is processed by a rule-based reward function. An evaluation is conducted. Therefore, the GRPO optimization strategy based on token-level importance sampling is adopted. The target is defined as shown in the following formulas (13) and (14):

[0198] (13)

[0199] (14)

[0200] in, , , Representation strategy The optimization objective function, This represents a spatial state s and a sequence of actions (or responses). Expectations, s S represents sampling a state from the state distribution S (such as a screenshot of a mobile phone screen). This indicates that the old policy is applied to the state space S. Generate G different action sequences (or responses), where G is the number of trajectories sampled each time (e.g., G=6, meaning 6 different outputs are generated for the same input). This represents the complete action sequence or response for the i-th sample. Indicates the length of the action sequence. This represents the l-th token in the i-th action sequence. This indicates that a prefix has been generated in spatial state S. Strategies under these conditions Generate token The probability, This indicates that a prefix has been generated in spatial state S. old strategy under the condition Generate token The probability, The dominant function represents the i-th trajectory. This indicates that the clipping range exceeds the parameter. This represents the weighting coefficient of the KL divergence regularization term. Indicates the KL divergence. This represents the reward for the i-th sample, where i and l are both integers. For clipping functions, This indicates taking the average. This indicates the standard deviation.

[0201] It is understood that sampling in this embodiment refers to the process by which the GUI agent, during reinforcement learning training, generates specific operation instructions (i.e., action sequences) with a certain degree of randomness based on the currently learned knowledge (policy). An action sequence refers to a series of operations or instructions executed sequentially by the GUI agent to complete a certain task.

[0202] It is understandable that the constraints in formula (14) are... and This represents the dynamic filtering process. This indicates that the GUI agent predicted the correct data, while This indicates that the GUI agent predicted incorrect data.

[0203] For example, in Figure 7C In the scenario shown, "tap(583,178)" is a response, while From 1 to This represents each token, for example, ['tap','(','583','178',')']. This indicates the token currently generated by the model. The probability, for example, the probability of "tap". This indicates that the token was generated before the model was last updated. The probability of the updated probability is the ratio of the probability before and after the update, which is called the importance sampling coefficient. In formula (13), the importance sampling coefficient is calculated based on the token level, ensuring the accuracy of importance sampling. Additionally, the dominance function is calculated... At that time, the mean and standard deviation represent the reward group, respectively. The mean and standard deviation of the KL divergence are also considered. Furthermore, the KL divergence ensures stable learning during the reinforcement learning process.

[0204] In other words, the GRPO training method based on token-level importance sampling splits the GUI agent's response into multiple tokens for sampling training, which is a fine-grained training method.

[0205] (2) GRPO training method based on importance sampling of response-level (entire response level)

[0206] The GRPO training method based on response-level importance sampling expands the token-level importance sampling dimension to the response-level dimension, where the optimization strategy... The target is defined as shown in the following formula (15):

[0207] (15)

[0208] in, This represents the overall probability of generating a response, for example, generating "tap(583,178)". This represents the probability of generating a certain response before the model's last update, for example, "tap(500,180)". Therefore, importance sampling... It is based on response-level calculation, which is different from the token-level method mentioned above. This approach can make reinforcement learning training more stable and eliminate the impact of some special tokens on the overall system.

[0209] In other words, the GRPO training method based on response-level importance sampling samples the responses of the GUI agent as a whole for training, which is a coarse-grained training method.

[0210] The display method provided in the embodiments of this application will be described below with reference to the flowchart.

[0211] picture 10. According to some embodiments of this application, a flowchart of a display method is shown.

[0212] like Figure 9 As shown, the method includes:

[0213] S101: The electronic device displays the first application interface.

[0214] For example, such as Figure 3A As shown, mobile phone 100 displays the price interface U1 of the group buying application (as an example of the first application interface).

[0215] S102: The electronic device receives a first user instruction, which is used to instruct the execution of a first operation on a first interface element on a first application interface.

[0216] It is understandable that the first interface element can be an icon, button, input box, slider, switch, or drop-down list, etc. The first user command can be a voice command, text command, gesture command, or operation command entered by the user. The first operation can be a click operation, long press operation, swipe operation, input operation, or drag operation, etc.

[0217] S103: The electronic device inputs the first user instruction and the first image of the first application interface to the intelligent agent and obtains the first response result output by the intelligent agent. The first response result includes the first position of the first interface element, the first operation type, and the first operation content.

[0218] After receiving the first user instruction, the electronic device can acquire the first image of the first application interface, and then input the first user instruction and the first image into the intelligent agent to obtain the first response result output by the intelligent agent.

[0219] It is understood that the intelligent agent in the embodiments of this application can be the aforementioned GUI agent, specifically, for example, the aforementioned YOYO.

[0220] In some embodiments, the electronic device can obtain the first image of the first application interface by taking a screenshot of the first application interface, or the electronic device can obtain the first image by other means, such as by screen recording.

[0221] For example, such as Figure 3B As shown, after receiving a user voice command containing the content "I want to book a superior king room at Hotel A on January 24th, including 2 breakfasts, for a price of 560 yuan" (as an example of the first user command), mobile phone 100 inputs the user voice command and a screenshot of the price interface U1 (as an example of the first image) to YOYO. Then, YOYO outputs the first response result based on the user voice command and the screenshot of the price interface U1. The first response result includes the booking control K1 on the price interface U1 that needs to be operated (as an example of the first interface element) and the element information of the booking control K1, such as the operation type of the booking control K1 being a click, the operation content being one click, and the location of the booking control K1.

[0222] S104: The electronic device performs a first operation on a first interface element through an intelligent agent based on a first location, a first operation type, and a first operation content, wherein the intelligent agent is obtained by reinforcement learning training based on a reward function.

[0223] The intelligent agent can perform a first operation on the first interface element based on the first position, the first operation type, and the first operation content of the first interface element.

[0224] For example, such as Figures 3A to 3C As shown, YOYO, based on the identified operation type, operation type and location of the reservation control K1, clicks on the reservation control K1 on the price interface U1 (as an example of the first operation) to open the payment interface U2 of the group buying application.

[0225] It is understood that the reward function is determined based on the sum of the first reward function, the second reward function and the third reward function. For the specific calculation method, please refer to the relevant introduction of the above formulas (1), (2) and (3), which will not be repeated here.

[0226] It is understandable that the first reward function can be the aforementioned formatted reward function, the second reward function can be the aforementioned precise reward function, and the third reward function can be the aforementioned distance reward function.

[0227] In some embodiments, the first reward function is determined based on the difference between the predicted format of the response output by the agent during training and the true format of the response. The response includes the predicted location of interface elements, the predicted operation type, and the predicted operation content.

[0228] For example, if the predicted format and the true format are the same, the first reward function takes the first value, which is greater than 0; or, if the predicted format and the true format are different, the first reward function takes the second value, which is less than 0. The absolute value of the first value is equal to the absolute value of the second value.

[0229] In some embodiments, the second reward function is determined based on the first distance, the difference between the predicted operation type of the interface element output by the agent during training and the actual operation type of the interface element, and the difference between the predicted operation content of the interface element output by the agent during training and the actual operation content of the interface element. The first distance is the distance between the predicted position of the interface element output by the agent during training and the actual position of the interface element. The third reward function is determined based on the second reward function and the first distance.

[0230] For example, if the predicted operation type is the same as the actual operation type, the first distance is less than or equal to the distance threshold, and the similarity between the predicted operation content and the actual operation content is greater than or equal to the similarity threshold, then the second reward function takes the third value, which is greater than 0. Alternatively, if the predicted operation type is different from the actual operation type, or the first distance is greater than the distance threshold, or the similarity between the predicted operation content and the actual operation content of the interface element is less than or equal to the similarity threshold, then the second reward function takes the fourth value, which is less than 0. The absolute values ​​of the third and fourth values ​​are equal.

[0231] In some embodiments, the third reward function is determined based on the second reward function and the first distance.

[0232] For example, if the value of the second reward function is greater than 0, then the value of the third reward function is the fifth value, which is less than 0 and its absolute value is positively correlated with the first distance; or, if the value of the second reward function is less than 0, then the value of the third reward function is 0. The distance threshold is the radius of a preset circle centered on the actual position of the interface element. The fifth value is determined based on the ratio between the first distance and the radius of the preset circle.

[0233] In some embodiments, the agent is trained on a first training set, which is obtained by deleting at least one training sample from a second training set. Specifically, when any one of the at least one training sample is input into the agent, the distance between any one of the multiple predicted positions of the interface element output by the agent and the actual position of the interface element is less than or equal to a distance threshold; or, the distance between any one of the multiple predicted positions of the interface element output by the agent and the actual position of the interface element is greater than the distance threshold. The specific process can be referred to the aforementioned description of the filtering method for training data of the GUI agent, and will not be repeated here.

[0234] It is understood that the first training set may be a training set that only includes the aforementioned third type of training samples, and the second training set may be a training set that includes the aforementioned first type of training samples, second type of training samples, and third type of training samples.

[0235] In some embodiments, the reward function is the reward function in a Markov process model, which is established based on user instructions and application interfaces as the interaction environment. For details, please refer to the relevant introduction of the MDP model in the aforementioned formula (12), which will not be repeated here.

[0236] In this embodiment, an intelligent agent performs related operations on a first interface element. This intelligent agent is trained based on the aforementioned reward function. Training the intelligent agent based on this reward function improves its ability to perform tasks and avoids problems such as the agent being unable to complete tasks or producing incorrect task execution results. After completing reinforcement learning training, the trained intelligent agent identifies the first position, first operation type, and first operation content of the first interface element on the first application interface based on the first user command and the first image of the first application interface. Then, based on the first position, first operation type, and first operation content of the first interface element, it performs the first operation on the first interface element on the first application interface. This helps the user automatically perform tasks without manual operation, improving the user experience.

[0237] Figure 10 According to some embodiments of this application, a schematic diagram of the structure of an electronic device is shown.

[0238] like Figure 10 As shown, the electronic device 1000 includes a processor 110, a mobile communication module 120, a wireless communication module 130, a power supply module 140, a screen 150, an interface module 160, a camera 170, a memory 180, and a sensor module 190, etc. The electronic device 1000 can be the aforementioned mobile phone 100, or other terminal devices deployed with intelligent agents (such as GUI agents).

[0239] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 1000. In other embodiments of this application, the electronic device 1000 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0240] Processor 110 may include one or more processing units, such as processing modules or circuits of a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), microprocessor (MCU), artificial intelligence (AI) processor, or field programmable gate array (FPGA). Different processing units may be independent devices or integrated into one or more processors. Processor 110 may include storage units for storing instructions and data. In some embodiments, the storage unit in processor 110 is a memory 180.

[0241] In some embodiments, the processor 110 is an AI processor. The aforementioned intelligent agent can be deployed in the processor 110. When the screen 150 of the electronic device 1000 displays the first application interface, the electronic device 1000 receives the aforementioned first user instruction. The electronic device 1000 inputs the first user instruction and the first image of the first application interface to the intelligent agent deployed on the processor 110, obtains the first element information of the first interface element output by the intelligent agent, and then performs a first operation on the first interface element on the first application interface based on the first element information.

[0242] This application also provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the methods provided in the foregoing embodiments.

[0243] This application also provides a computer program product, including computer program code, which, when run on a computer, causes the computer to execute the methods provided in the foregoing embodiments.

[0244] This application provides a chip including a processor for reading and executing a computer program stored in a memory to perform the methods provided in the foregoing embodiments.

[0245] The embodiments disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0246] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor, a microcontroller, an application-specific integrated circuit, or a microprocessor.

[0247] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0248] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on or on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, optical discs, read-only memory, magneto-optical disks, random access memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0249] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0250] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0251] It should be noted that in the examples and description of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the aforementioned element.

[0252] Although this application has been illustrated and described with reference to certain embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made thereto without departing from the scope of this application.

Claims

1. A display method, characterized in that, Applied to an electronic device on which an intelligent agent is deployed, the method includes: Display the first application interface; Upon receiving a first user instruction, the first user instruction is used to instruct the execution of a first interface element on the first application interface; The first user instruction and the first image of the first application interface are input to the intelligent agent to obtain the first response result output by the intelligent agent. The first response result includes the first position of the first interface element, the first operation type and the first operation content. The intelligent agent performs the first operation on the first interface element based on the first location, the first operation type, and the first operation content. The agent is obtained through reinforcement learning training based on a reward function, which is determined by the sum of a first reward function, a second reward function, and a third reward function. The first reward function is determined based on the difference between the predicted format of the response result output by the agent during training and the actual format of the response result. The response result includes the predicted position of the interface element, the predicted operation type, and the predicted operation content. The second reward function is determined based on the first distance, the difference between the predicted operation type of the interface element output by the agent during training and the actual operation type of the interface element, and the difference between the predicted operation content of the interface element output by the agent during training and the actual operation content of the interface element. The first distance is the distance between the predicted position of the interface element output by the agent during training and the actual position of the interface element. The third reward function is determined based on the second reward function and the first distance.

2. The method according to claim 1, characterized in that, The first reward function is determined based on the difference between the predicted format of the response results output by the agent during training and the true format of the response results, and includes: If the predicted format and the true format are the same, then the first reward function takes a first value, which is greater than 0; or, If the predicted format is different from the true format, then the first reward function takes the second value, which is less than 0.

3. The method according to claim 2, characterized in that, The absolute value of the first value is equal to the absolute value of the second value.

4. The method according to any one of claims 2 to 3, characterized in that, The second reward function is determined based on the first distance, the difference between the predicted operation type of the interface element output by the agent during training and the actual operation type of the interface element, and the difference between the predicted operation content of the interface element output by the agent during training and the actual operation content of the interface element, including: If the predicted operation type and the actual operation type are the same, the first distance is less than or equal to a distance threshold, and the similarity between the predicted operation content and the actual operation content is greater than or equal to a similarity threshold, then the value of the second reward function is a third value, which is greater than 0; or, If the predicted operation type is different from the actual operation type, or if the first distance is greater than a distance threshold, or if the similarity between the predicted operation content and the actual operation content of the interface element is less than or equal to a similarity threshold, then the second reward function takes a fourth value, which is less than 0.

5. The method according to claim 4, characterized in that, The absolute value of the third value is equal to the absolute value of the fourth value.

6. The method according to any one of claims 2 to 5, characterized in that, The third reward function is determined based on the second reward function and the first distance, and includes: If the value of the second reward function is greater than 0, then the value of the third reward function is a fifth value, which is less than 0 and whose absolute value is positively correlated with the first distance; or, If the value of the second reward function is less than 0, then the value of the third reward function is 0.

7. The method according to claim 6, characterized in that, The distance threshold is the radius of a preset circle centered on the actual position of the interface element; The fifth value is determined based on the ratio between the first distance and the radius of the preset circle.

8. The method according to any one of claims 1 to 7, characterized in that, The first operation type includes click operation, long press operation, swipe operation, input operation or drag operation; The first operation includes the number of clicks, the duration of the long press, the direction of the swipe, the input text, or the direction of the drag.

9. The method according to any one of claims 1 to 8, characterized in that, The agent is trained based on a first training set, which is obtained by deleting at least one training sample from a second training set. Wherein, after any one of the at least one training sample is input into the agent, the distance between any one of the multiple predicted positions of the interface element output by the agent and the actual position of the interface element is less than or equal to a distance threshold, or the distance between any one of the multiple predicted positions of the interface element output by the agent and the actual position of the interface element is greater than the distance threshold.

10. The method according to any one of claims 1 to 9, characterized in that, The reward function is the reward function in a Markov process model, which is established using user commands and application interfaces as the interaction environment.

11. An electronic device, characterized in that, include: Memory, used to store instructions; A processor, when executing instructions in the memory, causes the electronic device to perform the method of any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 10.

13. A computer program product, characterized in that, It includes computer program code that, when run on a computer, causes the computer to perform the method of any one of claims 1 to 10.

Images