Interaction method and device, electronic equipment and storage medium

Abstract

The invention provides an interaction method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence, and the method comprises the steps: obtaining an interaction task, a first interaction control of a first GUI associated with the interaction task, an executable first action list, and a first picture displaying the first GUI; and calling an intelligent agent integrated with a backtracking mechanism, determining an execution action under at least one time step according to the interaction task, the first picture, the first interaction control and the first action list, and executing the execution action so as to complete the interaction task. Therefore, the task completion quality can be improved, repeated operation or wrong operation caused by the fact that the action deviates from the task target is avoided, the task execution process is smoother, unnecessary resource waste is reduced, and then the task completion efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an interaction method, device, electronic device, and storage medium. Background Art

[0002] Rapid advances in large language models (LLMs) and vision-language models (VLMs) have provided a solid foundation for developing intelligent agents capable of interacting within the graphical user interface (GUI) environment of mobile terminals. These VLM-based agents can effectively control mobile terminals by parsing GUI pages, generating corresponding operational instructions, and executing them.

[0003] However, relevant technologies still face significant challenges when handling complex, multi-step tasks. Numerous research efforts have explored various strategies to improve agents' task handling capabilities, such as optimizing decision-making processes using Monte Carlo tree search algorithms or formulating high-level plans before task execution and continuously reflecting on and refining these plans during execution. However, these approaches largely focus on improving the performance of the VLM itself, with less consideration given to ensuring that each action closely aligns with the ultimate task objective. This can lead to discrepancies during execution, compromising the quality and efficiency of task completion. Summary of the Invention

[0004] The present application aims to solve one of the technical problems in the related art at least to a certain extent.

[0005] To this end, the present application proposes an interactive method, device, electronic device and storage medium to improve the quality of task completion and avoid repeated operations or erroneous operations caused by actions deviating from the task objectives, so as to make the task execution process smoother, reduce unnecessary waste of resources, and thus improve the efficiency of task completion.

[0006] In one aspect, an embodiment of the present application provides an interaction method, including:

[0007] Acquire an interactive task, a first interactive control of a first GUI associated with the interactive task, a list of executable first actions, and a first image showing the first GUI;

[0008] Call an intelligent agent integrated with a backtracking mechanism to determine and execute an execution action in at least one time step based on the interaction task, the first picture, the first interaction control and the first action list to complete the interaction task.

[0009] Another embodiment of the present application provides an interactive device, including:

[0010] an acquisition module, configured to acquire an interactive task, a first interactive control of a first GUI associated with the interactive task, a first executable action list, and a first image showing the first GUI;

[0011] A calling module is used to call an intelligent agent integrated with a backtracking mechanism, determine and execute an execution action in at least one time step based on the interaction task, the first picture, the first interaction control and the first action list to complete the interaction task.

[0012] In another aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the interaction method as described in the aforementioned aspect is implemented.

[0013] In yet another aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the interaction method as described in the aforementioned aspect is implemented.

[0014] In another aspect, an embodiment of the present application provides a computer program product having a computer program stored thereon, which implements the interaction method as described in the aforementioned aspect when the program is executed by a processor.

[0015] The interactive method, device, electronic device and storage medium proposed in this application, by calling an intelligent agent integrated with a backtracking mechanism, determines the execution action based on multimodal input data (including interactive tasks, interactive controls and executable action lists of GUI pages associated with interactive tasks, and pictures of GUI pages), compared with the traditional method that focuses on improving VLM performance and pays less attention to the fit between actions and the final task goals, can more accurately make each action closely revolve around the final task goal, reduce the situation where the execution stage does not meet expectations, thereby improving the quality of task completion, and avoid repeated operations or erroneous operations caused by actions deviating from the task goals, making the task execution process smoother, reducing unnecessary waste of resources, and thus improving the efficiency of task completion. In addition, when processing complex multi-step tasks, through the backtracking mechanism, the determined actions can be dynamically adjusted and optimized during the execution process, which helps to deal with various complex situations and uncertainties that may arise during the task execution process, so that the intelligent agent can better adapt to and complete complex multi-step interactive tasks and improve the user's interactive experience.

[0016] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0018] Figure 1 A flowchart of the first interaction method provided in an embodiment of the present application;

[0019] Figure 2 A flowchart of the second interaction method provided in an embodiment of the present application;

[0020] Figure 3 A flowchart of the third interaction method provided in an embodiment of the present application;

[0021] Figure 4 A flowchart of the fourth interaction method provided in an embodiment of the present application;

[0022] Figure 5 A flowchart of the fifth interaction method provided in an embodiment of the present application;

[0023] Figure 6 Schematic diagram of the working mechanism of BacktrackAgent provided in this application and Agent in related technologies;

[0024] Figure 7 Schematic diagram of the reasoning process of BacktrackAgent at the tth time step provided in an embodiment of the present application;

[0025] Figure 8 Schematic diagram of the interaction process between the BacktrackAgent provided in the embodiment of the present application and the Agent in the related technology and the GUI page;

[0026] Figure 9 A flowchart of a method for training an intelligent agent provided in an embodiment of the present application;

[0027] Figure 10 A schematic diagram of the structure of an interactive device provided in an embodiment of the present application;

[0028] Figure 11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0030] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0031] It should also be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0032] In related technologies, intelligent agents that can interact in a GUI environment can understand the requirements of complex tasks and complete them by interacting with the GUI page multiple times. However, these complex tasks may require more than a dozen or even dozens of actions to complete, and any error in any of the actions may lead to task failure. In addition, reinforcement learning methods can be used to sample some positive and negative examples to encourage the agent to generate positive actions that can complete the task and avoid negative actions. However, the performance of this method depends on the quality of the sampled data and whether it is sufficient to cover all error scenarios. They slightly improve the task completion rate, but it is still difficult to recover from the error state.

[0033] As an example, the reasoning process of an agent that can interact in a GUI environment mainly includes the following steps:

[0034] 1. The agent observes the task content, historical action list, GUI page in the current terminal environment, and executable action space of the given task.

[0035] 2. The agent generates execution actions based on the background information in step 1.

[0036] 3. The Agent executes the action in step 2 to update the current terminal environment.

[0037] 4. The agent loops through steps 1-3 until it deems the current reasoning process has successfully completed the given task, or the number of loop interactions exceeds the maximum number of interactions.

[0038] For example, for a given task of “help me find walking directions to nearby restaurants”, the agent will break it down into three subtasks:

[0039] 1. Go to the "Walking" page;

[0040] 2. Search for "nearby restaurants" on the "Walk" page. For example, you can enter "nearby restaurants" in the search box on the "Walk" page and click the "Search" button;

[0041] 3. Select a restaurant on the search results page to complete the given task.

[0042] In summary, while the above solutions reduce the difficulty by breaking down the original task and encourage the agent to select actions that are more conducive to task completion through reinforcement learning, they still have the potential to generate incorrect actions at each step. As the complexity of the task increases and the number of steps required to complete it increases, the probability of errors in the entire interaction process also increases exponentially. These solutions do not check whether the GUI page after executing an action is consistent with expectations, and once an incorrect action is executed, they do not actively recover from the erroneous GUI page. Therefore, their overall task completion capabilities still need to be improved.

[0043] Therefore, in response to at least one problem existing in the above-mentioned related technologies, the present application proposes an interactive method, device, electronic device and storage medium, which determines the execution action based on multimodal input data (including interactive tasks, interactive controls and executable action lists of GUI pages associated with interactive tasks, and pictures of GUI pages) by calling an intelligent agent integrated with a backtracking mechanism. Compared with the traditional method that focuses on improving VLM performance and pays less attention to the fit between actions and the final task goals, it can more accurately make each action closely revolve around the final task goal, reduce the situation where the execution stage does not meet expectations, thereby improving the quality of task completion, and avoid repeated operations or erroneous operations caused by the deviation of actions from the task goals, making the task execution process smoother, reducing unnecessary waste of resources, and thus improving the efficiency of task completion. In addition, when processing complex multi-step tasks, through the backtracking mechanism, the determined actions can be dynamically adjusted and optimized during the execution process, which helps to deal with various complex situations and uncertainties that may arise during the task execution process, so that the intelligent agent can better adapt to and complete complex multi-step interactive tasks and improve the user's interactive experience.

[0044] The following describes the interactive method, device, electronic device, and storage medium of the embodiments of the present application with reference to the accompanying drawings. Before describing the embodiments of the present application in detail, for ease of understanding, the following common technical terms are first introduced:

[0045] Time Step: In GUI interaction tasks, a time step (or interaction step) is a discrete unit of time or step that demarcates the interaction process between the agent and the GUI environment. Each time step represents a specific point in time or period during which the agent can perceive the environment state, make decisions, and perform actions.

[0046] Backtracking is an algorithmic strategy that explores all possible solutions through a cycle of trial and error. Starting from the initial state of the problem, it gradually builds a solution through recursion or iteration, continuously checking whether the current partial solution satisfies the problem's constraints. If so, the algorithm continues building. If not, it undoes the most recent choice, returning to the previous state and trying other possible solutions.

[0047] A page source file (such as an Extensible Markup Language (XML) file) is used to describe the structure, layout, and component properties of a user interface (UI) or GUI.

[0048] Figure 1 A flowchart of the first interaction method provided in an embodiment of the present application.

[0049] It should be noted that the interaction method of the embodiments of the present application can be applied to an interaction device. In some possible embodiments, the interaction device can be configured in an electronic device or chip so that the electronic device or chip can perform the interaction function. In addition, in some possible embodiments, the interaction device can also be software in the electronic device.

[0050] In any embodiment of the present application, the chip can be integrated into an electronic device. Among them, the chip includes a central processing unit (CPU), an image signal processing (ISP), an application-specific integrated circuit (ASIC), a microprocessor (DSP), a field programmable gate array (FPGA), a system on a chip (SOC), a reduced instruction set computer (RISC), etc., which are not listed one by one here.

[0051] Among them, electronic devices include but are not limited to: terminals, personal computers, etc., wherein a terminal is an entity on the user side for receiving or transmitting signals, such as a mobile phone. A terminal can also be referred to as a terminal device (terminal), user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal can be a car with communication functions, a smart car, a mobile phone, a wearable device, a tablet computer (Pad), a computer with wireless transceiver functions, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control (industrial control), a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety (transportation safety), a wireless terminal in a smart city (smart city), a wireless terminal in a smart home (smart home), etc. The embodiments of this application do not limit the specific technology and specific device form adopted by the terminal.

[0052] like Figure 1 As shown, the interaction method may include the following steps S101 to S102:

[0053] Step S101: Acquire an interactive task, a first interactive control of a first GUI associated with the interactive task, a first executable action list, and a first picture showing the first GUI.

[0054] Interactive tasks may be tasks input by the user for interacting with the GUI page. For example, interactive tasks include but are not limited to navigation tasks, route planning tasks, and meal ordering tasks. Input methods for interactive tasks include but are not limited to touch input (such as sliding, clicking, etc.), keyboard input, and voice input.

[0055] It should be noted that there is no restriction on the field to which the interactive tasks belong. For example, the fields to which the interactive tasks belong include but are not limited to: autonomous driving, smart furniture management, smart assistants and other fields.

[0056] The first GUI may be a GUI page (such as a homepage or a home page) of any application (Application, referred to as APP), and the first GUI is associated with an interactive task input by the user. For example, the interactive task may be a task input in the first GUI.

[0057] The first interactive control is a control or button that allows interaction in the first GUI, such as an input box, a search button, a drop-down box, etc.

[0058] The first action list, or action space, includes actions that can be executed in the first GUI, such as click actions, search actions, scroll actions, input actions, etc.

[0059] In an embodiment of the present application, an interactive task may be obtained, and a picture showing a first GUI associated with the interactive task may be obtained, which is referred to as a first picture in the present application. For example, a screenshot of the first GUI may be taken to obtain the first picture.

[0060] As a possible implementation manner, the interactive task is obtained in, for example, a manner of: in response to an input operation on a voice input control of the first GUI triggered by the user, obtaining the interactive task of the user's voice input.

[0061] Among them, the voice control is used to call the intelligent agent and receive voice input instructions.

[0062] As another possible implementation, the interactive task is obtained, for example, in response to an input operation on a text input control of the first GUI triggered by the user, to obtain the interactive task manually input by the user.

[0063] Among them, the text input control is used to call the intelligent agent and receive text input instructions.

[0064] As another possible implementation, the interactive task is obtained, for example, in response to a triggering operation on a target page element of the first GUI triggered by the user, obtaining the interactive task associated with the target page element.

[0065] The target page elements include page elements in the first GUI that allow interaction.

[0066] The triggering operation includes but is not limited to: clicking operation, dragging operation, etc.

[0067] For example, taking the first GUI including the homepage of a navigation APP or a map APP as an exemplary explanation, the user can drag a marker on the first GUI to the target location, and thus determine the interaction task as: navigating to the location indicated by the marker based on the dragging result of the marker.

[0068] As another possible implementation, the interactive task is obtained, for example, in response to an input operation on the dialog window of the first GUI triggered by the user, obtaining the interactive task input by the user in the dialog window.

[0069] Among them, the dialogue window is used to interact with the agent.

[0070] Therefore, users can use different methods to input interactive tasks, which can improve the flexibility and applicability of the method.

[0071] It should be understood that the above-mentioned method of obtaining interactive tasks is only an example, and other methods can also be used to obtain interactive tasks, which is not limited in the embodiments of the present application.

[0072] In the embodiment of the present application, a first interactive control that allows interaction and a first executable action list in the first GUI may also be obtained.

[0073] Exemplarily, a page source file (such as an XML file) of the first GUI may be obtained and parsed to obtain a first interactive control that allows interaction and a first executable action list.

[0074] Therefore, by parsing the page source file of the first GUI, the first interactive control and the executable first action list in the first GUI can be effectively and accurately obtained, thereby improving the effectiveness and accuracy of subsequent task execution.

[0075] Step S102: Call an intelligent agent integrated with a backtracking mechanism to determine and execute an execution action in at least one time step according to the interactive task, the first image, the first interactive control, and the first action list to complete the interactive task.

[0076] In an embodiment of the present application, an intelligent agent with an integrated backtracking mechanism can be called to determine the execution actions in at least one time step in sequence according to the interactive task, the first picture, the first interactive control and the first action list, and execute the execution actions in each time step to complete the interactive task.

[0077] The interactive method of the embodiment of the present application determines the execution action based on multimodal input data (including interactive tasks, interactive controls and executable action lists of GUI pages associated with interactive tasks, and pictures of GUI pages) by calling an intelligent agent integrated with a backtracking mechanism. Compared with the traditional method that focuses on improving VLM performance and pays less attention to the fit between actions and the final task goals, it can more accurately make each action closely revolve around the final task goal, reduce the situation where the execution stage does not meet expectations, thereby improving the quality of task completion, and avoid repeated operations or erroneous operations caused by actions deviating from the task goals, making the task execution process smoother, reducing unnecessary waste of resources, and thus improving the efficiency of task completion. In addition, when processing complex multi-step tasks, through the backtracking mechanism, the determined actions can be dynamically adjusted and optimized during the execution process, which helps to deal with various complex situations and uncertainties that may arise during the task execution process, so that the intelligent agent can better adapt to and complete complex multi-step interactive tasks and improve the user's interactive experience.

[0078] It should be noted that in any embodiment of the present application, the automatic execution of interactive tasks by the intelligent agent can provide strong support for accessibility services, thereby developing accessibility application programs (Application, referred to as APP) for disabled users or users who are temporarily unable to use electronic devices. This technology can significantly improve the user experience of these users, enabling them to access and operate various digital content and services more conveniently and efficiently.

[0079] In other words, intelligent agents can help develop apps with accessibility features that provide accessibility specifically for users with disabilities or those who are temporarily unable to interact with electronic devices. This not only improves the usability and accessibility of apps, but also provides these users with a more friendly and inclusive digital environment.

[0080] This embodiment of the application provides another interaction method. Figure 2 A flowchart of the second interaction method provided in an embodiment of the present application.

[0081] It should be noted that the interaction method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the relevant technology, and the embodiments of the present application do not limit this.

[0082] like Figure 2 As shown, the interaction method may include the following steps S201 to S209:

[0083] Step S201: Acquire an interactive task, a first interactive control of a first GUI associated with the interactive task, a first executable action list, and a first picture showing the first GUI.

[0084] It should be noted that the explanation of step S201 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0085] Step S202: Call an agent integrated with a backtracking mechanism to determine the execution action at the first time step based on the interactive task, the first image, the first interactive control, and the first action list.

[0086] The first time step is the first time step between the start and end of the interactive task.

[0087] In an embodiment of the present application, first, an intelligent agent integrated with a backtracking mechanism can be called to determine the execution action at the first time step based on the interactive task, the first picture, the first interactive control and the first action list.

[0088] Step S203: Call the agent to perform the execution action on the first GUI in the first time step to obtain and display the second GUI to which the first time step jumps.

[0089] Afterwards, the intelligent agent can be called to execute the execution steps under the first time step on the first GUI, obtain the second GUI to which the first time step jumps, and display the second GUI to which the first time step jumps.

[0090] Step S204 , determining whether the action category to which the action executed in the first time step belongs is completion, if so, executing step S205 , if not, executing steps S206 to S209 .

[0091] It should be noted that step S205 and steps S206 to S209 are two parallel implementation methods, and either one can be executed.

[0092] Step S205: determine that the interactive task is completed and end the calling of the intelligent agent.

[0093] In an embodiment of the present application, if the action category to which the execution action at the first time step belongs is Finish, it is determined that the interaction task is completed, and at this time, the call of the agent can be terminated.

[0094] Alternatively, if the action category to which the execution action at the i-th time step belongs is Finish, it is determined that the interaction task is completed, and at this time, the call of the agent can be ended.

[0095] Step S206, continue to call the intelligent agent, determine the execution action in the i-th time step based on the interactive task, the second picture, the execution action in the previous i-1 time steps, the second interactive control of the second GUI jumped to in the i-1th time step, and the executable second action list; wherein the second picture shows the second GUI jumped to in the i-1th time step.

[0096] Wherein, i is a positive integer greater than 1.

[0097] The second interactive control is a control or button that allows interaction in the second GUI, such as an input box, a search button, a drop-down box, etc.

[0098] The second action list, or action space, includes actions that can be executed in the second GUI, such as click actions, search actions, scroll actions, input actions, etc.

[0099] In an embodiment of the present application, if the action type of the execution action in the i-1th time step (i is a positive integer greater than 1) is incomplete, it is determined that the interaction task is not completed. At this time, the intelligent agent can continue to be called to determine the execution action in the i-th time step based on the interaction task, the second picture showing the second GUI to which the i-1th time step jumps, the execution action in the previous i-1 time step, the second interactive control of the second GUI to which the i-1th time step jumps, and the list of executable second actions.

[0100] The i-th time step refers to a non-first time step among the time steps between the start and end of the interactive task.

[0101] Step S207: Call the agent to execute the execution action under the i-th time step on the second GUI jumped to in the i-1th time step, so as to obtain and display the second GUI jumped to in the i-th time step.

[0102] In an embodiment of the present application, the intelligent agent can also be called to execute the execution action under the i-th time step on the second GUI jumped to in the i-1th time step, obtain the second GUI jumped to in the i-th time step, and display the second GUI jumped to in the i-th time step.

[0103] Step S208 , determining whether the action category to which the action executed at the i-th time step belongs is completion, if so, executing step S205 , if not, executing step S209 .

[0104] It should be noted that step S205 and step S209 are two parallel implementation methods, and either one can be executed.

[0105] Step S209: Continue to call the agent to determine and execute the execution action at the i+1th time step to complete the interaction task.

[0106] In an embodiment of the present application, if the action category to which the execution action at the i-th time step belongs is incomplete, it is determined that the interaction task is not completed. At this time, the intelligent agent can continue to be called to determine the execution action at the i+1-th time step, and the execution action at the i+1-th time step can be executed on the second GUI jumped to in the i-th time step to complete the interaction task.

[0107] The interactive method of the embodiment of the present application determines the execution action at each time step in turn through the intelligent agent, and determines whether the interactive task is completed based on the action category to which the execution action at each time step belongs, which can improve the rationality and reliability of the determination result. Among them, for non-first time steps, such as the i-th (i is a positive integer greater than 1) time step, the intelligent agent comprehensively integrates the interactive task, the execution action at the previous i-1 time steps, the second interactive control and the second action list of the second GUI jumped to at the i-1 time step, and the second picture of the second GUI jumped to at the i-1 time step, to determine the execution action at the i-th time step, which can improve the rationality and accuracy of the action prediction, and then perform accurate actions, which can improve the execution quality and efficiency of the interactive task.

[0108] This embodiment of the application provides another interaction method. Figure 3 A flowchart of the third interaction method provided in an embodiment of the present application.

[0109] It should be noted that the interaction method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the relevant technology, and the embodiments of the present application do not limit this.

[0110] like Figure 3 As shown, the interaction method may include the following steps S301 to S310:

[0111] Step S301: Acquire an interactive task, a first interactive control of a first GUI associated with the interactive task, a first executable action list, and a first picture showing the first GUI.

[0112] It should be noted that the explanation of step S301 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0113] Step S302: Call the agent to generate the initial action for the jth time step.

[0114] Wherein, j is a positive integer. In response to j=1, the initial action is generated by the intelligent agent based on the interactive task, the first picture, the first interactive control and the first action list. In response to j being a positive integer greater than 1, the initial action is generated based on the interactive task, the second picture, the execution action in the previous j-1 time steps, the second interactive control of the second GUI jumped to in the j-1th time step, and the second action list.

[0115] As an example, the agent may include a generator, which can generate the initial action of the j-th time step. For example, the first GUI is marked as P1, and the initial action of the j-th time step is a j For a given interactive task X, when j = 1, the generator can generate a1 based on the interactive task X, the first picture showing P1, the first interactive control in P1, and the first action list; when j is greater than 1, the generator can generate a1 based on the interactive task X, the first picture showing P1, the first interactive control in P1, and the first action list. j The second picture, the execution action before j-1 time steps (a <j ), P j The second interactive control and the second action list in generate a j .

[0116] Among them, P j is the second GUI jumped to in the j-1th time step, a <j Including a1, a2, ..., a j-1 .

[0117] Step S303: Call the agent to simulate the execution of the initial action of the j-th time step on the target GUI to obtain the initial GUI jumped to in the j-th time step.

[0118] In response to j=1, the target GUI is the first GUI, and in response to j being greater than 1, the target GUI is the second GUI jumped to in the j-1th time step.

[0119] In the embodiment of the present application, when j=1, the agent can be called to simulate and execute a1 on the first GUI to obtain the initial GUI jumped to in the first time step; when j>1, the agent can be called to execute a1 on the second GUI jumped to in the j-1th time step. j , and get the initial GUI jumped to at the j-th time step.

[0120] In step S304, the agent is called to determine whether the initial action of the jth time step is a correct action based on the target GUI and the initial GUI jumped to in the jth time step. If so, step S305 is executed; if not, step S306 is executed.

[0121] As a possible implementation method, the intelligent agent can also include a verifier and a judger. The verifier and the judger can judge whether the initial action of the j-th time step is the correct action based on the target GUI and the initial GUI jumped to in the j-th time step.

[0122] As an example, the verifier can judge whether the initial action of the jth time step is valid based on the target GUI, the initial GUI jumped to in the jth time step, and the action category to which the initial action of the jth time step belongs, and obtain a judgment result. The judger can classify the initial action of the jth time step based on the target GUI and the initial GUI jumped to in the jth time step, and obtain a classification result; wherein the classification result is used to indicate whether the initial action of the jth time step has the value of promoting the successful completion of the interactive task, that is, the classification result is used to indicate whether the initial action of the jth time step is helpful to successfully complete the interactive task. Therefore, in this application, it can be determined whether the initial action of the jth time step is a correct action based on the judgment result and / or classification result.

[0123] For example, the validator can be a rule-based module that evaluates whether an action is executable and valid. For example, the validator can check whether the action a at the jth time step is valid according to the following key principle: j Effectiveness:

[0124] a. Action a j Must be valid and executable, belonging to the specified action category, such as click, scroll, enter, and complete.

[0125] b. Action a j Should contain correctly formatted elements and parameters.

[0126] c. If a j Instead of a special action to complete the task and exit, the device environment should be changed after executing this action. That is, the validator will compare the page P j and P j+1 If the two GUI pages are the same, the action is considered invalid.

[0127] Among them, P j Can be target GUI, P j+1 The initial GUI to jump to at the jth time step.

[0128] For example, if the output of the validator is 1, it means action a j Valid, if the output of the validator is 0, it means action a j invalid.

[0129] That is to say, the judgment result can be determined based on at least one of the first judgment result, the second judgment result and the third judgment result, wherein the first judgment result is used to indicate whether the action category to which the initial action of the j-th time step belongs is the specified action category; the second judgment result is used to indicate whether the format of the initial action of the j-th time step matches the format corresponding to the specified action category, and whether the parameters contained in the initial action of the j-th time step match the parameters corresponding to the specified action category; the third judgment result is used to indicate whether the target GUI is consistent with the initial GUI jumped to in the j-th time step.

[0130] In the present application, when each of the following items is met, it can be determined that the output result of the validator indicates that the initial action of the j-th time step is valid, that is, the judgment result is used to indicate that the initial action of the j-th time step is valid: the first judgment result indicates that the action category to which the initial action of the j-th time step belongs is the specified action category; the format of the initial action of the j-th time step matches the format corresponding to the specified action category; the parameters contained in the initial action of the j-th time step match the parameters corresponding to the specified action category; the target GUI is inconsistent with the initial GUI jumped to in the j-th time step.

[0131] When at least one of the following is satisfied, it can be determined that the output result of the validator indicates that the initial action of the j-th time step is invalid, that is, the judgment result is used to indicate that the initial action of the j-th time step is invalid: the first judgment result indicates that the action category to which the initial action of the j-th time step belongs is not the specified action category; the format of the initial action of the j-th time step does not match the format corresponding to the specified action category; the parameters contained in the initial action of the j-th time step do not match the parameters corresponding to the specified action category; the target GUI is consistent with the initial GUI jumped to in the j-th time step.

[0132] Therefore, by integrating multiple judgment results to evaluate whether an action is effective, the rationality and reliability of the evaluation results can be improved.

[0133] For example, the judger can be a module based on a deep learning model for evaluating the action a j Will it lead to the wrong GUI page and whether it increases the possibility of achieving the task goal, wherein the judge can j and P j+1 , evaluate and execute this action a j Whether it helps to successfully complete the interactive task.

[0134] For example, the judge can be a binary classifier. If the output of the judge is 1, it means that the action a j has value, and if the output of the judge is 0, it means that the action a j No value.

[0135] In the present application, when the judgment result indicates that the initial action of the j-th time step is invalid, and / or the classification result indicates that the initial action of the j-th time step has no value, the initial action of the j-th time step can be determined to be an erroneous action; and when the judgment result indicates that the initial action of the j-th time step is valid, and the classification result indicates that the initial action of the j-th time step has value, the initial action of the j-th time step can be determined to be a correct action.

[0136] Therefore, whether the initial action at the jth time step is effective and / or whether the initial action at the jth time step has the value of promoting the successful completion of the interactive task can be comprehensively evaluated to evaluate whether the initial action at the jth time step is the correct action, which can improve the rationality and reliability of the evaluation results.

[0137] It should be noted that step S305 and step S306 are two parallel implementation methods, and either one can be executed.

[0138] Step S305: The initial action of the j-th time step is used as the execution action in the j-th time step.

[0139] In an embodiment of the present application, when the initial action at the j-th time step is a correct action, the initial action at the j-th time step can be used as the final execution action at the j-th time step.

[0140] Step S306, calling the intelligent agent to perform a page retraction operation to trace back to the target GUI, and updating the initial action of the j-th time step to obtain the execution action at the j-th time step.

[0141] In an embodiment of the present application, when the initial action at the jth time step is an erroneous action, the intelligent agent can be called to perform a page retraction operation on the initial GUI jumped to at the jth time step to trace back to the target GUI, and update the initial action at the jth time step to obtain the execution action at the jth time step.

[0142] Exemplarily, the intelligent agent may also include a reflector, which can update actions based on the target GUI and the initial GUI jumped to at the j-th time step, as well as all erroneous actions (or called reflective actions) at the j-th time step to obtain the final execution action at the j-th time step.

[0143] Step S307 , displaying the second GUI jumped to in the j-th time step.

[0144] The second GUI jumped to in the j-th time step is the GUI jumped to by executing the execution action in the j-th time step on the target GUI.

[0145] In an embodiment of the present application, the second GUI to which the jth time step jumps can be displayed; wherein, the second GUI to which the jth time step jumps is the GUI to which the agent jumps when performing the execution action on the target GUI in the jth time step.

[0146] Step S308 , determining whether the action category to which the action executed at the j-th time step belongs is completion, if so, executing step S309 , if not, executing step S310 .

[0147] It should be noted that step S309 and step S310 are two parallel implementation methods, and either one can be executed.

[0148] Step S309: determine that the interactive task is completed and end the calling of the intelligent agent.

[0149] In an embodiment of the present application, when the action category to which the execution action at the j-th time step belongs is Finish, it can be determined that the interaction task is completed, and at this time, the call of the intelligent agent can be ended.

[0150] Step S310, continue to call the intelligent agent, determine and execute the execution action at the j+1th time step to complete the interactive task.

[0151] In an embodiment of the present application, when the action category to which the execution action at the j-th time step belongs is incomplete, it can be determined that the interaction task is incomplete. At this time, the intelligent agent can continue to be called to determine the execution action at the j+1-th time step, and the execution action at the j+1-th time step can be executed on the second GUI jumped to in the j-th time step to complete the interaction task.

[0152] The interactive method of the embodiment of the present application, when the intelligent agent determines that the initial action of the j-th time step is an erroneous action based on the two GUI pages before and after the jump (i.e., the target GPU and the initial GUI jumped to in the j-th time step), performs a page retraction operation based on the backtracking mechanism to backtrack to the target GUI before the jump, and updates the initial action of the j-th time step (i.e., error recovery). This can improve the accuracy of the execution action prediction in the j-th time step, thereby improving the execution quality of the interactive task.

[0153] This embodiment of the application provides another interaction method. Figure 4 A flowchart of the fourth interaction method provided in an embodiment of the present application.

[0154] It should be noted that the interaction method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the relevant technology, and the embodiments of the present application do not limit this.

[0155] like Figure 4 As shown, the interaction method may include the following steps S401 to S410:

[0156] Step S401: Acquire an interactive task, a first interactive control of a first GUI associated with the interactive task, a first executable action list, and a first picture showing the first GUI.

[0157] Step S402: call the intelligent agent to generate the initial action of the jth time step, and call the intelligent agent to simulate the execution of the initial action of the jth time step on the target GUI to obtain the initial GUI jumped to in the jth time step.

[0158] Wherein, j is a positive integer. In response to j=1, the initial action is generated by the intelligent agent based on the interactive task, the first picture, the first interactive control and the first action list. In response to j being a positive integer greater than 1, the initial action is generated based on the interactive task, the second picture, the execution action in the previous j-1 time steps, the second interactive control of the second GUI jumped to in the j-1th time step, and the second action list.

[0159] In response to j=1, the target GUI is the first GUI, and in response to j being a positive integer greater than 1, the target GUI is the second GUI jumped to in the j-1th time step.

[0160] Step S403, calling the intelligent agent to determine whether the initial action of the jth time step is a correct action based on the target GUI and the initial GUI jumped to in the jth time step. If so, execute step S404; if not, execute steps S405 to S406.

[0161] It should be noted that step S404 and steps S405 to S406 are two parallel implementation methods, and either one can be executed.

[0162] Step S404: The initial action of the j-th time step is used as the execution action in the j-th time step.

[0163] It should be noted that the explanation of steps S401 to S404 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0164] Step S405, perform a page retraction operation to trace back to the target GUI, and call the agent to perform at least one round of iteration process on the initial action of the jth time step according to the target GUI and the initial GUI jumped to in the jth time step.

[0165] As an example, for the first round of iteration in at least one round of iteration, the intelligent agent can be called to update the initial action of the jth time step according to the target GUI and the initial GUI jumped to at the jth time step, and obtain the updated action of the jth time step output by the first round of iteration.

[0166] Exemplarily, the intelligent agent may include a reflector, which updates the initial action (i.e., the wrong action, or called the reflective action) of the j-th time step according to the target GUI and the initial GUI jumped to at the j-th time step, so as to obtain the updated action of the j-th time step output by the first round of iterative process.

[0167] For non-first round of iteration in at least one round of iteration, such as the k-th (k is a positive integer greater than 1) round of iteration, the intelligent agent can be called to determine whether k is less than or equal to the maximum number of iterations. If not (i.e., k is greater than the maximum number of iterations), the iteration process is terminated. If yes (i.e., k is less than or equal to the maximum number of iterations), the update action of the j-th time step output by the k-1-th round of iteration is simulated and executed on the target GUI to obtain the updated GUI jumped to at the j-th time step. Based on the target GUI and the updated GUI jumped to at the j-th time step, it is determined whether the update action of the j-th time step output by the k-1-th round of iteration is a correct action. The implementation principle is similar to that of step S304 and will not be repeated here.

[0168] If the update action of the jth time step output by the k-1th iterative process is a correct action, the iterative process ends; if the update action of the jth time step output by the k-1th iterative process is an incorrect action, the page retraction operation continues to be executed to trace back to the target GUI, and according to the target GUI, the updated GUI jumped to in the jth time step, and the update action of the jth time step output by the previous k-1 rounds of iterative process, the update action of the jth time step output by the k-1th round of iterative process is updated to obtain the update action of the jth time step output by the kth round of iterative process.

[0169] That is, in this application, the reflector in the intelligent agent can update the action according to the target GUI and the updated GUI jumped to at the j-th time step, as well as all the wrong actions (or called reflective actions) at the j-th time step to obtain the final execution action at the j-th time step.

[0170] In summary, during the error recovery process, the agent reflects on all erroneous actions in the jth time step and the GUI pages before and after the most recent erroneous action in the jth time step, and updates the most recent erroneous action. This can avoid the agent from generating repeated erroneous actions and take into account the execution efficiency and quality of the interactive task.

[0171] Step S406: The update action of the j-th time step outputted by the last round of iteration is used as the execution action at the j-th time step.

[0172] In an embodiment of the present application, the update action of the j-th time step output by the last round of iteration process can be used as the final execution action at the j-th time step.

[0173] Step S407 , displaying the second GUI jumped to in the j-th time step.

[0174] The second GUI jumped to in the j-th time step is the GUI jumped to by executing the execution action in the j-th time step on the target GUI.

[0175] Step S408 , determining whether the action category to which the action executed at the j-th time step belongs is completion, if so, executing step S409 , if not, executing step S410 .

[0176] It should be noted that step S409 and step S410 are two parallel implementation methods, and either one can be executed.

[0177] Step S409: determine that the interactive task is completed and end the calling of the intelligent agent.

[0178] Step S410: Continue to call the agent to determine and execute the execution action at the j+1th time step to complete the interaction task.

[0179] It should be noted that the explanation of steps S407 to S410 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0180] The interactive method of the embodiment of the present application adopts a multi-round iteration method to reflect on and update the generated action of the j-th time step, which can improve the reliability of the final generated execution action.

[0181] This embodiment of the application provides another interaction method. Figure 5 This is a flowchart of the fifth interaction method provided in an embodiment of the present application.

[0182] It should be noted that the interaction method can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the relevant technology, and the embodiments of the present application do not limit this.

[0183] like Figure 5 As shown, the interaction method may include the following steps S501 to S511:

[0184] Step S501: Acquire an interactive task, a first interactive control of a first GUI associated with the interactive task, a first executable action list, and a first picture showing the first GUI.

[0185] Step S502: call the intelligent agent to generate the initial action of the jth time step, and call the intelligent agent to simulate the execution of the initial action of the jth time step on the target GUI to obtain the initial GUI jumped to in the jth time step.

[0186] Wherein, j is a positive integer. In response to j=1, the initial action is generated by the intelligent agent based on the interactive task, the first picture, the first interactive control and the first action list. In response to j being a positive integer greater than 1, the initial action is generated based on the interactive task, the second picture, the execution action in the previous j-1 time steps, the second interactive control of the second GUI jumped to in the j-1th time step, and the second action list.

[0187] In response to j=1, the target GUI is the first GUI, and in response to j being a positive integer greater than 1, the target GUI is the second GUI jumped to in the j-1th time step.

[0188] Step S503, calling the intelligent agent to determine whether the initial action of the jth time step is a correct action based on the target GUI and the initial GUI jumped to in the jth time step. If so, execute step S504; if not, execute steps S505 to S507.

[0189] It should be noted that step S504 and steps S505 to S507 are two parallel implementation methods, and either one can be executed.

[0190] Step S504: The initial action of the j-th time step is used as the execution action in the j-th time step.

[0191] It should be noted that the explanation of steps S501 to S504 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0192] Step S505: perform a page retraction operation to trace back to the target GUI, and call the agent to annotate the third picture showing the target GUI according to the execution result of the initial action at the jth time step to obtain an annotated picture.

[0193] It should be noted that, when the target GUI (P j ) In the scenario where an incorrect action is performed, it is impossible to reproduce the GUI environment and perform non-standard actions to generate a new GUI page P j+1 To solve this problem, in this application, you can j Simulate and draw action a j The possible execution results of the action are presented, for example, drawing arrows for scrolling operations, marking interactive elements and input text for input operations, and then providing these annotated GUI pages to the reflector in the intelligent agent for action reflection to demonstrate the effects and potential impacts of the actions.

[0194] That is, in this application, the third picture showing the target GUI can be annotated according to the execution result of the initial action at the j-th time step to obtain a labeled picture.

[0195] Step S506: Call the agent to perform at least one round of iteration on the initial action of the jth time step based on the labeled image, the target GUI, and the initial GUI to which the jth subtask jumps.

[0196] In an embodiment of the present application, the intelligent agent can be called to perform at least one round of iterative process on the initial action of the jth time step based on the labeled image, the target GUI and the initial GUI jumped to by the jth subtask.

[0197] As an example, for the first round of iteration in at least one round of iteration, the calling agent can update the initial action of the jth time step based on the labeled image, the target GUI and the initial GUI jumped to at the jth time step, and obtain the updated action of the jth time step output by the first round of iteration.

[0198] Exemplarily, the intelligent agent may include a reflector, which updates the initial action (or reflected action) of the jth time step according to the labeled image, the target GUI and the initial GUI jumped to at the jth time step, so as to obtain the updated action of the jth time step output by the first round of iteration process.

[0199] For at least one non-first round of iteration in the iterative process, such as the k-th (k is a positive integer greater than 1) round of iteration, the calling agent can determine whether k is less than or equal to the maximum number of iterations. If not, the iterative process is terminated. If so, the update action of the j-th time step output by the k-1-th round of iteration is simulated and executed on the target GUI to obtain the updated GUI jumped to at the j-th time step. Based on the target GUI and the updated GUI jumped to at the j-th time step, it is determined whether the update action of the j-th time step output by the k-1-th round of iteration is a correct action. The implementation principle is similar to step S304 and will not be repeated here.

[0200] If the update action of the jth time step output by the k-1th iterative process is a correct action, the iterative process is terminated. If the update action of the jth time step output by the k-1th iterative process is an incorrect action, the page retraction operation is continued to trace back to the target GUI, and according to the target GUI, the updated GUI jumped to in the jth time step, and the update action of the jth time step output by the previous k-1 rounds of iterative process, the update action of the jth time step output by the k-1th round of iterative process is updated to obtain the update action of the jth time step output by the kth round of iterative process.

[0201] In summary, during the error recovery process, the agent reflects on all erroneous actions in the jth time step and the GUI pages before and after the most recent erroneous action in the jth time step, and updates the most recent erroneous action. This can avoid the agent from generating repeated erroneous actions and take into account the execution efficiency and quality of the interactive task.

[0202] Step S507: The update action of the j-th time step outputted by the last round of iteration is used as the execution action at the j-th time step.

[0203] In an embodiment of the present application, the update action of the j-th time step output by the last round of iteration process can be used as the final execution action at the j-th time step.

[0204] Step S508 : Display the second GUI jumped to in the j-th time step.

[0205] The second GUI jumped to in the j-th time step is the GUI jumped to by executing the execution action in the j-th time step on the target GUI.

[0206] Step S509 , determining whether the action category to which the action executed at the j-th time step belongs is completion, if so, executing step S510 , if not, executing step S511 .

[0207] It should be noted that step S510 and step S511 are two parallel implementation methods, and either one can be executed.

[0208] Step S510: determine that the interactive task is completed and end the calling of the intelligent agent.

[0209] Step S511, continue to call the intelligent agent, determine and execute the execution action at the j+1th time step to complete the interaction task.

[0210] It should be noted that the explanation of steps S508 to S511 can be found in the relevant description in any embodiment of the present application and will not be repeated here.

[0211] The interactive method of the embodiment of the present application adopts a multi-round iteration method to reflect on and update the generated action of the j-th time step, which can improve the reliability of the final generated execution action.

[0212] In any embodiment of the present application, an agent using a backtrack mechanism may be referred to as a BacktrackAgent, which uses a backtrack mechanism to observe the execution results of actions, enabling it to effectively detect and recover error states. During each interaction, the BacktrackAgent can access the GUI environment in the electronic device or simulator, take a screenshot to obtain a picture showing the GUI page, and analyze the page source file of the GUI page to obtain interactive page elements and a list of executable actions, link the given interactive task with the currently completed execution action, generate the correct action that should be executed at the current time step, send it to the GUI environment and execute it, and finally update the GUI page to perform the action observation, generation and execution cycle for the next time step.

[0213] For example, suppose the interactive task is to order coffee, such as "I want to order a large cup of XX coffee with extra YY syrup, delivered to my home." If the BacktrackAgent goes to the order page without selecting a specific coffee, it will recognize this as an error and return to the homepage. Later, on the product search page, if the BacktrackAgent does not enter "XX coffee" in the search box but instead selects the similar "ZZ coffee" in the recommendation column, the BacktrackAgent will detect the incorrect selection by comparing the interactive tasks and can then perform a page retraction. The BacktrackAgent can then navigate to the correct product page by selecting "XX coffee" from the recommended items in the drop-down menu or by selecting "XX coffee" on the search results page. Then, on the product page, it can select specific parameters (such as large). If the BacktrackAgent made the incorrect selection, after retraction, it can successfully add the large cup of "XX coffee" to the shopping bag. At this point, BacktrackAgent can enter the checkout page by clicking the shopping bag or clicking the "Go to Order" button, and finally place an order. If the order is successfully placed, the task execution is completed.

[0214] It should be noted that related technologies mainly focus on achieving single-step accuracy when completing tasks, and are difficult to recover from error states. Figure 6 As shown in Figure 2, a task may require more than a dozen actions to complete, and any incorrect action can cause the entire task to fail. Furthermore, related techniques rely on supervised fine-tuning using annotated page navigation datasets. They primarily mimic data from successful interactions while neglecting to understand and analyze data from incorrect interactions. Furthermore, these techniques fail to consider the results of actions, making it difficult to determine whether the current GUI page has deviated from the task and to recover from any error states.

[0215] Figure 6 In

[15] , SFT refers to the agent trained using the Supervised Fine-Tuning method, and DPO refers to the agent trained using the Direct Preference Optimization method.

[0216] In this application, BacktrackAgent uses the backtracking mechanism to enhance the task completion capability in the GUI environment, and can effectively detect and recover the error state. Among them, BacktrackAgent consists of four key components: Generator, Verifier, Judger and Reflector, which together complete Task X. Among them, the generator creates and executes actions based on the current interactive task and GUI environment, the verifier and judger act as error detection modules to determine whether the current state (i.e., GUI page) needs to be backtracked, and the reflector acts as an error recovery module to improve the action based on the judgment result and guide BacktrackAgent back to the state most likely to successfully complete the task. In addition, the rewards of the verifier and judger can be used to further improve the capabilities of BacktrackAgent.

[0217] As an example, the reasoning process of BacktrackAgent at time step t can be as follows Figure 7 As shown, Figure 7 The upper part of the represents the action generated by the generator Is executed, the GUI page is changed from P t Updated to Figure 7 The lower half of represents the action generated in the i-th reflection process (referred to as the iterative process in this application) at the t-th time step It is judged as an error action by the error detection module and rewritten by the error recovery module

[0218] Among them, the reasoning process of BacktrackAgent at the tth time step mainly includes the following steps:

[0219] 1. Generator based on task X, GUI page P t and historical execution action a <t =a1,...,a t-1 , generate the current action a at the t-th time step t .

[0220] For example, for a given task X, the Generator can t , from P t The list of executable actions, interactive controls, and historical execution actions extracted from <t , generating action a t Among them, a t Theoretically, it must belong to P tThe list of executable actions in the action list. Actions not in the action list are usually invalid operations such as clicking a blank space or sliding a non-sliding element.

[0221] Furthermore, in Generator, action a is generated t After that, BacktrackAgent simulates the execution of the action a t , change the page from P t Update to the subsequent page P t+1 Then, a is evaluated by both the validator and the judge. t Is it the correct action?

[0222] Among them, the validator checks action a according to the following two key principles t Effectiveness:

[0223] a. Actions must be valid and executable, falling into one of the following four categories: click, scroll, enter, and complete. Furthermore, actions should contain correctly formatted elements and parameters.

[0224] b. As long as a t It is not a special action to complete the task and exit. t After that, the GUI environment should change accordingly. The validator will compare the page P t and P t+1 If the two pages are the same, then the action a t Invalid. Here, the output of the validator is defined as Indicates that the action a t efficient, Indicates that the action a t invalid.

[0225] Among them, the judge can pass page P t 、Action a t and page P t+1 , evaluate and execute the action a t Whether it helps to successfully complete task X. For example, the judge can be a binary classifier, and the output of the judge is defined as Indicates that the action a t valuable, Indicates that the action a t No value.

[0226] 2. During error detection, BacktrackAgent executes the generated actions in the GUI environment Thus generating a new page P t+1 , where i represents the i-th reflection or rewrite at the t-th time step. The verifier and judge evaluate the action The evaluation of the verifier and judge can consider whether the action is effective and helps to complete task X. implement GUI pages before and after (P t and P t+1 ) and related background information (task X and previous execution action a <t ).

[0227] The validator is a rule-based module that evaluates actions Is it executable and effective? The judge is a model-based module that evaluates the execution action. Whether it leads to an error page and whether it increases the possibility of achieving the task goal.

[0228] 3. If both the validator and the judge evaluate If it is a correct action, BacktrackAgent regards it as the final action of the tth time step and continues to execute the t+1th time step. Otherwise, BacktrackAgent will turn to the reflector for error recovery.

[0229] 4. During the error recovery process, the reflector will reflect all the reflected actions (i.e., error actions) at the t-th time step, as well as the GUI pages before and after the action is executed. Updated to

[0230] That is, BacktrackAgent decides whether to modify the action according to the output of the verifier and the judge. During the i-th reflection or rewriting process, if the verifier or the judge believes that the action is invalid or does not contribute to the completion of the task, the reflector will modify the current action. Updated to BacktrackAgent repeats the "verifier-judge-reflector" phase in each interaction until both the verifier and the judge evaluate the action as valid, or until the maximum number of reflections (referred to as the maximum number of iterations in this application) is reached or exceeded.

[0231] 5. BacktrackAgent repeats steps 2 to 4 above at each time step until is judged to be correct or i exceeds the maximum number of reflections (referred to as the maximum number of iterations in this application).

[0232] It should be noted that for the action execution mechanism: executing action a t And change the GUI page from P t Update to P t+1 The process of performing the task is called “actual execution”. However, for some manually annotated datasets, the agent cannot reproduce the GUI environment and perform non-standard actions.t To generate page data P t+1 To solve this problem, in this application, you can t A t The possible execution results of the action are shown in the following example, such as drawing arrows for scrolling actions and marking interactive elements and input text for input actions. This process is called "simulated execution". These annotated pages are then fed into the error detection and recovery module to demonstrate the action a t effects and potential impacts.

[0233] To set a coffee task, such as "I want to order a large cup of XX coffee, with more YY syrup, and have it delivered to my home", P t For example, when BacktrackAgent performs a click("ExtraLargeCup") action on a product page, selecting a specific parameter (e.g., large cup) on the product page, the "actual execution" updates the cup type on the GUI page to extra large. In contrast, the "simulated execution" colors the bounding box of the "ExtraLargeCup" element on the product page with a prominent color, such as red.

[0234] As an example, taking Task X as "help me find the route to a nearby restaurant", the interaction process between the BacktrackAgent provided in this application and the Agent and GUI page in the related art can be as follows: Figure 8 As shown, the agent in the related art correctly predicted several steps, but when it mistakenly entered the favorites, it was unable to realize its mistake and continued to browse in the favorites, trying to find a nearby restaurant, which resulted in the agent being unable to complete the task; while the BacktrackAgent provided by the present application can still detect the error and restore to the correct page when it mistakenly slides to the driving page, and finally complete the task.

[0235] In summary, the BacktrackAgent provided in this application has at least the following advantages: better task completion capability is achieved by learning to detect and recover from erroneous GUI pages; the backtracking mechanism can better capture the errors of BacktrackAgent and further improve the performance of BacktrackAgent compared with pre-sampled paired positive and negative data; the simulated execution mechanism can improve the task completion capability of BacktrackAgent on different data sets.

[0236] The above are various embodiments corresponding to the application method (ie, interaction method) of the intelligent agent. This application also proposes a training method for the intelligent agent.

[0237] Figure 9A flowchart of a method for training an intelligent agent provided in an embodiment of the present application.

[0238] It should be noted that the training method of the intelligent agent can be executed alone, or it can be executed in combination with any embodiment of the present application or a possible implementation method in the embodiment, or it can be executed in combination with any technical solution in the relevant technology, and the embodiments of the present application do not limit this.

[0239] like Figure 9 As shown, the training method of the agent includes the following steps S901 to S903:

[0240] Step S901, obtain a first training sample and a second training sample; wherein the first training sample includes a first sample task, a first sample GUI associated with the first sample task, and a sample action for implementing the first sample task; the second training sample includes a second sample task and a second sample GUI associated with the second sample task.

[0241] The number of the first training samples may be at least one, and the number of the second training samples may also be at least one.

[0242] The method for obtaining the first training sample and the second training sample is not limited. For example, the first training sample and the second training sample can be obtained from a training set or a test set. Alternatively, the first training sample and the second training sample can be manually generated. Alternatively, information can be collected from a real GUI environment to obtain the first training sample and the second training sample, and so on. This embodiment of the present application does not limit this.

[0243] Step S902: Train the generator in the agent according to the first training sample; wherein the generator is used to generate execution actions of the task.

[0244] As an example, the generator in the intelligent agent can be a base multimodal large model. In this application, a first training sample (such as a training sample in a multi-round page navigation task dataset) can be used to supervise the training of the generator.

[0245] For example, the generator can be trained using cross entropy loss, where the loss function of the generator can be:

[0246]

[0247] Among them, Acts refers to the abbreviation of action space (ActionSpace), Acts (P t ) refers to P t A list of actions or action space that can be executed in.

[0248] Step S903: Use the generator to generate the execution action corresponding to the second sample task in the second training sample, and train the judge and reflector in the intelligent agent based on the execution action of the second sample task and the second training sample; wherein the judge is used to determine whether the execution action is valid, and the reflector is used to update the execution action.

[0249] In an embodiment of the present application, a trained generator can be used to generate execution actions corresponding to the second sample task in the second training sample, and the judge and reflector in the intelligent agent can be trained based on the execution actions of the second sample task and the second training sample.

[0250] For example, the judger and the reflector can also be trained using cross entropy loss, where the loss function of the judger can be:

[0251]

[0252] The loss function of the reflector can be:

[0253]

[0254] in, It refers to the output result of the judge's i-th reflection in the t-th time step, refers to the update action generated by the i-th reflection in the t-th time step, It refers to the update action generated by the first i-1 reflections in the t-th time step.

[0255] Alternatively, the present application can also utilize reinforcement learning principles to strengthen the agent based on the outputs of the verifier and judger in the agent. For example, the actions generated by the generator and reflector at each time step can be scored, and the various modules in the agent can be jointly trained based on these scores and cross-entropy loss.

[0256] For example, the final total loss L can be calculated using the following formula:

[0257] L=L g +β1L verifier +β2L judger ;(4)

[0258] in, β1 and β2 are preset weights.

[0259] The training method for the intelligent agent provided in this application can improve the task completion ability of the intelligent agent by using training samples to supervise the intelligent agent.

[0260] In order to implement the above embodiment, the embodiment of the present application also proposes an interaction device.

[0261] Figure 10 A schematic diagram of the structure of an interactive device provided in an embodiment of the present application.

[0262] like Figure 10 As shown, the interactive device 1000 may include: an acquisition module 1010 and a calling module 1020.

[0263] The acquisition module 1010 is configured to acquire an interactive task, a first interactive control of a first GUI associated with the interactive task, a first executable action list, and a first image showing the first GUI;

[0264] The calling module 1020 is used to call an intelligent agent integrated with a backtracking mechanism, determine and execute an execution action in at least one time step according to the interactive task, the first picture, the first interactive control and the first action list to complete the interactive task.

[0265] Furthermore, in one implementation of the embodiment of the present application, the acquisition module 1010 is used to: obtain the input interaction task in response to the input operation of the voice input control of the first GUI; wherein the voice control is used to call the intelligent agent and receive voice input instructions; obtain the input interaction task in response to the input operation of the text input control of the first GUI; wherein the text input control is used to call the intelligent agent and receive text input instructions; obtain the interaction task associated with the target page element in response to the trigger operation of the target page element of the first GUI; obtain the input interaction task in response to the input operation of the dialogue window of the first GUI; wherein the dialogue window is used to interact with the intelligent agent.

[0266] In one implementation of the embodiment of the present application, the acquisition module 1010 is used to: acquire a page source file of a first GUI; and parse the page source file to obtain a first interactive control that allows interaction and a first executable action list.

[0267] In one implementation of the embodiment of the present application, the calling module 1020 is used to: call the intelligent agent to determine the execution action at the first time step based on the interactive task, the first picture, the first interactive control and the first action list; call the intelligent agent to execute the execution action at the first time step on the first GUI to obtain and display the second GUI to which the first time step jumps; in response to the action category to which the execution action at the first time step belongs being completed, determine that the interactive task is completed, and end the calling of the intelligent agent.

[0268] In one implementation of the embodiment of the present application, the calling module 1020 is also used to: in response to the action type of the execution action in the i-1th time step being incomplete, continue to call the intelligent agent, and determine the execution action in the i-th time step based on the interactive task, the second picture, the execution action in the previous i-1 time step, the second interactive control of the second GUI jumped to in the i-1th time step, and the list of executable second actions; wherein i is a positive integer greater than 1, and the second picture shows the second GUI jumped to in the i-1th time step; call the intelligent agent to execute the execution action in the i-th time step on the second GUI jumped to in the i-1th time step, so as to obtain and display the second GUI jumped to in the i-1th time step; in response to the action category of the execution action in the i-th time step being completed, determine that the interactive task is completed, and end the calling of the intelligent agent.

[0269] In one implementation of the embodiment of the present application, module 1020 is called to: generate an initial action for the j-th time step; wherein, in response to j=1, the initial action is generated based on the interactive task, the first image, the first interactive control and the first action list; in response to j being a positive integer greater than 1, the initial action is generated based on the interactive task, the second image, the execution action in the previous j-1 time steps, the second interactive control of the second GUI jumped to in the j-1 time step and the second action list; simulate the execution of the initial action of the j-th time step on the target GUI to obtain the initial GUI jumped to in the j-1 time step; wherein, in response to j=1, the target GUI is the first GUI; in response to j being greater than 1, the target GUI is the second GUI jumped to in the j-1 time step; based on the target GUI and the initial GUI jumped to in the j-1 time step, determine whether the initial action of the j-th time step is a correct action; based on whether the initial action of the j-th time step is a correct action, determine the execution action in the j-th time step.

[0270] In one implementation of the embodiment of the present application, the calling module 1020 is configured to: if the initial action at the j-th time step is a correct action, use the initial action at the j-th time step as the execution action at the j-th time step.

[0271] In one implementation of an embodiment of the present application, module 1020 is called to: if the initial action of the j-th time step is an erroneous action, perform a page retraction operation to trace back to the target GUI, and update the initial action of the j-th time step to obtain the execution action at the j-th time step.

[0272] In one implementation of the embodiment of the present application, the calling module 1020 is used to: judge whether the initial action of the jth time step is valid based on the target GUI, the initial GUI jumped to in the jth time step, and the action category to which the initial action of the jth time step belongs, and obtain a judgment result; classify the initial action of the jth time step based on the target GUI and the initial GUI jumped to in the jth time step, and obtain a classification result; wherein the classification result is used to indicate whether the initial action of the jth time step has the value of promoting the successful completion of the interactive task; and determine whether the initial action of the jth time step is a correct action based on the judgment result and / or the classification result.

[0273] In one implementation of an embodiment of the present application, the judgment result is determined based on at least one of a first judgment result, a second judgment result, and a third judgment result. The first judgment result is used to indicate whether the action category to which the initial action of the j-th time step belongs is a specified action category; the second judgment result is used to indicate whether the format of the initial action of the j-th time step matches the format corresponding to the specified action category, and whether the parameters contained in the initial action of the j-th time step match the parameters corresponding to the specified action category; the third judgment result is used to indicate whether the target GUI is consistent with the initial GUI jumped to in the j-th time step.

[0274] In one implementation of the embodiment of the present application, in response to satisfying each of the following conditions, the judgment result is used to indicate that the initial action at the j-th time step is valid: a first judgment result indicates that the action category to which the initial action at the j-th time step belongs is a specified action category; a format of the initial action at the j-th time step matches a format corresponding to the specified action category; parameters included in the initial action at the j-th time step match parameters corresponding to the specified action category; the target GUI is inconsistent with the initial GUI jumped to at the j-th time step;

[0275] or,

[0276] In response to satisfying at least one of the following, the judgment result is used to indicate that the initial action of the j-th time step is invalid: the first judgment result indicates that the action category to which the initial action of the j-th time step belongs is not the specified action category; the format of the initial action of the j-th time step does not match the format corresponding to the specified action category; the parameters contained in the initial action of the j-th time step do not match the parameters corresponding to the specified action category; the target GUI is consistent with the initial GUI jumped to in the j-th time step.

[0277] In one implementation of the embodiment of the present application, the calling module 1020 is used to: in response to the judgment result indicating that the initial action of the j-th time step is valid, and the classification result indicating that the initial action of the j-th time step has value, determine that the initial action of the j-th time step is a correct action; in response to the judgment result indicating that the initial action of the j-th time step is invalid, and / or the classification result indicating that the initial action of the j-th time step has no value, determine that the initial action of the j-th time step is an incorrect action.

[0278] In one implementation of an embodiment of the present application, calling module 1020 is used to: perform at least one round of iterative process on the initial action of the j-th time step based on the target GUI and the initial GUI jumped to at the j-th time step; and use the updated action of the j-th time step output by the last round of iterative process as the execution action at the j-th time step.

[0279] In one implementation of the embodiment of the present application, module 1020 is called to execute a first round of iteration in at least one round of iteration, specifically: updating the initial action of the j-th time step according to the target GUI and the initial GUI jumped to at the j-th time step, and obtaining an updated action of the j-th time step output by the first round of iteration;

[0280] Call module 1020 to execute the kth iteration process in at least one iteration process, specifically: determine whether k is less than or equal to the maximum number of iterations; where k is a positive integer greater than 1; if k is less than or equal to the maximum number of iterations, simulate the update action of the jth time step output by the k-1th iteration process on the target GUI to obtain the updated GUI jumped to at the jth time step; determine whether the update action of the jth time step output by the k-1th iteration process is a correct action based on the target GUI and the updated GUI jumped to at the jth time step; if the update action of the jth time step output by the k-1th iteration process is an incorrect action, perform a page retraction operation to trace back to the target GUI, and update the update action of the jth time step output by the k-1th iteration process based on the target GUI, the updated GUI jumped to at the jth time step, and the update action of the jth time step output by the previous k-1th iteration process to obtain the update action of the jth time step output by the k-1th iteration process.

[0281] In one implementation of the embodiment of the present application, calling module 1020 is used to: end the iterative process when any of the following conditions is met: k is greater than the maximum number of iterations; the update action of the jth time step output by the k-1th round of iterative process is the correct action.

[0282] In one implementation of an embodiment of the present application, module 1020 is called to: annotate a third image displaying the target GUI according to the execution result of the initial action of the j-th time step to obtain an annotated image; and perform at least one round of iterative process on the initial action of the j-th time step according to the annotated image, the target GUI and the initial GUI to which the j-th subtask jumps.

[0283] In one implementation of the embodiment of the present application, the agent is trained using the following modules:

[0284] The acquisition module 1010 is further configured to: acquire a first training sample and a second training sample; wherein the first training sample includes a first sample task, a first sample GUI associated with the first sample task, and a sample action for implementing the first sample task; and the second training sample includes a second sample task and a second sample GUI associated with the second sample task;

[0285] A training module, configured to train a generator in the agent based on the first training sample; wherein the generator is configured to generate execution actions for the task;

[0286] A generator, configured to generate an execution action corresponding to a second sample task in a second training sample using the generator;

[0287] The training module is also used to train the judge and reflector in the intelligent agent based on the execution action of the second sample task and the second training sample; wherein the judge is used to determine whether the execution action is valid, and the reflector is used to update the execution action.

[0288] It should be noted that the aforementioned explanations of any method embodiment are also applicable to the interactive device of this embodiment and will not be repeated here.

[0289] In the interactive device of the embodiment of the present application, by calling an intelligent agent integrated with a backtracking mechanism, the execution action is determined based on multimodal input data (including interactive tasks, interactive controls of GUI pages associated with interactive tasks and executable action lists, and pictures of GUI pages). Compared with the traditional method that focuses on improving VLM performance and less considers the fit between actions and the final task goals, each action can be more accurately closely centered around the final task goal, reducing the situation where the execution stage does not meet expectations, thereby improving the quality of task completion, and avoiding repeated operations or erroneous operations caused by actions deviating from the task goals, making the task execution process smoother, reducing unnecessary waste of resources, and thus improving the efficiency of task completion. In addition, when processing complex multi-step tasks, through the backtracking mechanism, the determined actions can be dynamically adjusted and optimized during the execution process, which helps to deal with various complex situations and uncertainties that may arise during the task execution process, so that the intelligent agent can better adapt to and complete complex multi-step interactive tasks and improve the user's interactive experience.

[0290] In order to implement the above embodiments, the present application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the interaction method as described in any of the above embodiments is implemented.

[0291] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. For example, the electronic device 1100 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0292] Reference Figure 11 The electronic device 1100 may include one or more of the following components: a processing component 1102 , a memory 1104 , a power component 1106 , a multimedia component 1108 , an audio component 1110 , an input / output (I / O) interface 1112 , a sensor component 1114 , and a communication component 1116 .

[0293] The processing component 1102 generally controls the overall operation of the electronic device 1100, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 1102 may include one or more processors 1120 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 1102 may include one or more modules to facilitate interaction between the processing component 1102 and other components. For example, the processing component 1102 may include a multimedia module to facilitate interaction between the multimedia component 1108 and the processing component 1102.

[0294] The memory 1104 is configured to store various types of data to support operations on the electronic device 1100. Examples of such data include instructions for any application or method operating on the electronic device 1100, contact data, phone book data, messages, pictures, videos, etc. The memory 1104 can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0295] The power component 1106 provides power to the various components of the electronic device 1100. The power component 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 1100.

[0296] The multimedia component 1108 includes a screen that provides an output interface between the electronic device 1100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, slides, and gestures on the touch panel. The touch sensor can not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 1108 includes a front camera and / or a rear camera. When the electronic device 1100 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0297] The audio component 1110 is configured to output and / or input audio signals. For example, the audio component 1110 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 1100 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 1104 or transmitted via the communication component 1116. In some embodiments, the audio component 1110 also includes a speaker for outputting audio signals.

[0298] I / O interface 1112 provides an interface between processing component 1102 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0299] Sensor assembly 1114 includes one or more sensors for providing various aspects of the status assessment of electronic device 1100. For example, sensor assembly 1114 can detect the open / closed state of electronic device 1100, the relative positioning of components, such as the display and keypad of electronic device 1100. Sensor assembly 1114 can also detect changes in the position of electronic device 1100 or a component of electronic device 1100, the presence or absence of user contact with electronic device 1100, the orientation or acceleration / deceleration of electronic device 1100, and changes in the temperature of electronic device 1100. Sensor assembly 1114 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1114 can also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 1114 can also include an accelerometer, a gyroscope, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0300] The communication component 1116 is configured to facilitate wired or wireless communication between the electronic device 1100 and other devices. The electronic device 1100 can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 1116 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1116 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0301] In an exemplary embodiment, the electronic device 1100 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0302] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is further provided, such as a memory 1104 including instructions. The instructions can be executed by the processor 1120 of the electronic device 1100 to perform the above method. For example, the non-transitory computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0303] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the interactive method described in any of the above method embodiments is implemented.

[0304] In order to implement the above embodiments, the present application also proposes a computer program product on which a computer program is stored. When the computer program is executed by a processor, the interaction method as described in any of the above method embodiments is implemented.

[0305] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0306] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0307] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0308] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0309] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the present invention: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0310] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0311] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0312] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. An interactive method, characterized in that: include: Acquire an interactive task, a first interactive control of a first graphical user interface (GUI) associated with the interactive task, a first executable action list, and a first image showing the first GUI; Call an intelligent agent integrated with a backtracking mechanism to determine and execute an execution action in at least one time step based on the interaction task, the first picture, the first interaction control and the first action list to complete the interaction task.

2. The method according to claim 1, characterized in that The interactive task is obtained by using any one of the following: Responding to an input operation on a voice input control of the first GUI, obtaining an input interactive task; wherein the voice control is used to call the agent and receive a voice input instruction; Responding to an input operation on a text input control of the first GUI, obtaining an input interactive task; wherein the text input control is used to call the agent and receive a text input instruction; In response to a triggering operation on a target page element of the first GUI, acquiring an interactive task associated with the target page element; In response to an input operation on a dialog window of the first GUI, an input interactive task is acquired; wherein the dialog window is used to interact with the agent.

3. The method according to claim 1, characterized in that The first interactive control and the first action list are obtained by following the steps below: Obtaining a page source file of the first GUI; The page source file is parsed to obtain the first interactive control that allows interaction and the first executable action list.

4. The method according to claim 1, wherein The calling of an agent integrated with a backtracking mechanism, determining and executing an execution action in at least one time step based on the interaction task, the first image, the first interaction control, and the first action list to complete the interaction task, includes: Invoking the agent to determine an execution action at a first time step based on the interaction task, the first image, the first interaction control, and the first action list; Calling the agent to execute the execution action at the first time step on the first GUI, so as to obtain and display the second GUI jumped to by the first time step; In response to the action category to which the execution action at the first time step belongs being completion, it is determined that the interaction task is completed, and the calling of the agent ends.

5. The method according to claim 4, characterized in that The calling of an agent integrated with a backtracking mechanism, determining and executing an execution action in at least one time step based on the interaction task, the first image, the first interaction control, and the first action list to complete the interaction task, further includes: In response to the action type of the action executed at the (i-1)th time step being incomplete, the agent is continuously called to determine the action to be executed at the (i-1)th time step based on the interactive task, the second image, the action executed at the previous (i-1) time step, the second interactive control of the second GUI redirected to at the (i-1)th time step, and the list of executable second actions; wherein i is a positive integer greater than 1, and the second image displays the second GUI redirected to at the (i-1)th time step; Calling the agent to execute the execution action under the i-th time step on the second GUI jumped to in the i-1th time step, so as to obtain and display the second GUI jumped to in the i-th time step; In response to the action category to which the execution action at the i-th time step belongs being completion, it is determined that the interaction task is completed, and the calling of the agent is terminated.

6. The method according to claim 4 or 5, characterized in that The action to be executed at the jth time step is determined by the agent using the following steps: Generate an initial action for the j-th time step; wherein, in response to j=1, the initial action is generated based on the interactive task, the first image, the first interactive control, and the first action list; in response to j being a positive integer greater than 1, the initial action is generated based on the interactive task, the second image, the executed action in the previous j-1 time steps, the second interactive control of the second GUI jumped to in the j-1-th time step, and the second action list; Simulating the initial action of the j-th time step on the target GUI to obtain the initial GUI jumped to in the j-th time step; wherein, in response to j=1, the target GUI is the first GUI, and in response to j being greater than 1, the target GUI is the second GUI jumped to in the j-1-th time step; Determining whether the initial action at the j-th time step is a correct action based on the target GUI and the initial GUI jumped to at the j-th time step; The execution action at the j-th time step is determined according to whether the initial action at the j-th time step is a correct action.

7. The method according to claim 6, characterized in that The determining, based on whether the initial action at the j-th time step is a correct action, the execution action at the j-th time step includes: If the initial action of the j-th time step is a correct action, the initial action of the j-th time step is used as the execution action at the j-th time step.

8. The method according to claim 6, characterized in that The determining, based on whether the initial action at the j-th time step is a correct action, the execution action at the j-th time step includes: If the initial action at the j-th time step is an erroneous action, a page retraction operation is performed to trace back to the target GUI, and the initial action at the j-th time step is updated to obtain the execution action at the j-th time step.

9. The method according to claim 6, characterized in that The determining, based on the target GUI and the initial GUI jumped to in the j-th time step, whether the initial action in the j-th time step is a correct action includes: Determining whether the initial action at the j-th time step is valid according to the target GUI, the initial GUI to which the j-th time step jumps, and the action category to which the initial action at the j-th time step belongs, and obtaining a determination result; Classifying the initial action at the j-th time step according to the target GUI and the initial GUI to which the user jumps at the j-th time step, and obtaining a classification result; wherein the classification result is used to indicate whether the initial action at the j-th time step has the value of promoting the successful completion of the interactive task; According to the judgment result and / or the classification result, it is determined whether the initial action of the j-th time step is a correct action.

10. The method according to claim 9, characterized in that The judgment result is determined based on at least one of the first judgment result, the second judgment result and the third judgment result. The first judgment result is used to indicate whether the action category to which the initial action of the j-th time step belongs is a specified action category; The second judgment result is used to indicate whether the format of the initial action of the j-th time step matches the format corresponding to the specified action category, and whether the parameters included in the initial action of the j-th time step match the parameters corresponding to the specified action category; The third judgment result is used to indicate whether the target GUI is consistent with the initial GUI jumped to in the j-th time step.

11. The method according to claim 10, characterized in that In response to satisfying each of the following conditions, the judgment result is used to indicate that the initial action of the j-th time step is valid: the first judgment result indicates that the action category to which the initial action of the j-th time step belongs is the specified action category; the format of the initial action of the j-th time step matches the format corresponding to the specified action category; the parameters included in the initial action of the j-th time step match the parameters corresponding to the specified action category; the target GUI is inconsistent with the initial GUI jumped to in the j-th time step; or, In response to satisfying at least one of the following, the judgment result is used to indicate that the initial action of the j-th time step is invalid: the first judgment result indicates that the action category to which the initial action of the j-th time step belongs is not the specified action category; the format of the initial action of the j-th time step does not match the format corresponding to the specified action category; the parameters contained in the initial action of the j-th time step do not match the parameters corresponding to the specified action category; the target GUI is consistent with the initial GUI jumped to in the j-th time step.

12. The method according to claim 9, characterized in that The determining, based on the judgment result and / or the classification result, whether the initial action at the j-th time step is a correct action includes: In response to the judgment result indicating that the initial action at the j-th time step is valid, and the classification result indicating that the initial action at the j-th time step has the value, determining that the initial action at the j-th time step is a correct action; In response to the judgment result indicating that the initial action of the j-th time step is invalid, and / or the classification result indicating that the initial action of the j-th time step does not have the value, the initial action of the j-th time step is determined to be an erroneous action.

13. The method according to claim 8, characterized in that The updating of the initial action at the j-th time step to obtain the execution action at the j-th time step includes: performing at least one iteration process on the initial action of the j-th time step according to the target GUI and the initial GUI jumped to in the j-th time step; The update action of the j-th time step output by the last round of iteration process is used as the execution action at the j-th time step.

14. The method according to claim 13, characterized in that A first iterative process in the at least one iterative process includes: updating the initial action of the j-th time step according to the target GUI and the initial GUI jumped to in the j-th time step, to obtain an updated action of the j-th time step output by the first iterative process; The kth round of iteration in the at least one round of iteration includes: Determine whether k is less than or equal to the maximum number of iterations; where k is a positive integer greater than 1; If k is less than or equal to the maximum number of iterations, then simulate the update action of the j-th time step output by the k-1-th round of iteration process on the target GUI to obtain the updated GUI jumped to in the j-th time step; According to the target GUI and the updated GUI jumped to in the j-th time step, determining whether the updated action of the j-th time step output by the k-1-th round of iterative process is a correct action; If the update action of the jth time step output by the k-1th round of iterative process is an erroneous action, a page retraction operation is performed to trace back to the target GUI, and the update action of the jth time step output by the k-1th round of iterative process is updated according to the target GUI, the updated GUI jumped to in the jth time step, and the update action of the jth time step output by the previous k-1 round of iterative process to obtain the update action of the jth time step output by the kth round of iterative process.

15. The method according to claim 14, characterized in that The kth round of iteration in the at least one round of iteration further includes: The iterative process ends when any of the following conditions is met: k is greater than the maximum number of iterations; The updated action of the jth time step output by the k-1th round of iterative process is the correct action.

16. The method according to claim 13, characterized in that The performing at least one round of iterative process on the initial action of the j-th time step according to the target GUI and the initial GUI jumped to in the j-th time step includes: Annotating a third image showing the target GUI according to an execution result of the initial action at the j-th time step to obtain an annotated image; At least one round of iteration is performed on the initial action of the j-th time step according to the labeled image, the target GUI, and the initial GUI to which the j-th subtask jumps.

17. The method according to claim 1, wherein The agent is trained using the following steps: Obtain a first training sample and a second training sample; wherein the first training sample includes a first sample task, a first sample GUI associated with the first sample task, and a sample action for implementing the first sample task; and the second training sample includes a second sample task and a second sample GUI associated with the second sample task; Training a generator in the agent according to the first training sample; wherein the generator is used to generate execution actions of the task; The generator is used to generate the execution action corresponding to the second sample task in the second training sample, and the judge and reflector in the intelligent agent are trained according to the execution action of the second sample task and the second training sample; wherein the judge is used to judge whether the execution action is valid, and the reflector is used to update the execution action.

18. An interactive device, characterized in that: For implementing the interaction method according to any one of claims 1 to 17, the device comprises: an acquisition module, configured to acquire an interactive task, a first interactive control of a first GUI associated with the interactive task, a first executable action list, and a first image showing the first GUI; A calling module is used to call an intelligent agent integrated with a backtracking mechanism, determine and execute an execution action in at least one time step based on the interaction task, the first picture, the first interaction control and the first action list to complete the interaction task.

19. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method according to any one of claims 1 to 17 are implemented.

20. A non-transitory computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

21. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 17.

Images