Interaction task processing, model training method and device, electronic equipment and medium

Abstract

This disclosure provides interactive task processing, model training methods, devices, electronic devices, and media, relating to the field of artificial intelligence, particularly large-scale models, computer vision, and graphical user interface automation. The specific implementation scheme is as follows: Obtain the current interface state information of the graphical user interface; based on the task description information, the current interface state information, and the expected interface state description corresponding to the previous operation, determine the validity verification result of the previous operation, the operation instruction of the current operation, and the expected interface state description of the current operation. The validity verification result is generated by comparing the expected interface state description of the previous operation with the current interface state information, and the expected interface state description of the current operation is used to generate the operation instruction for the next operation; execute the operation instruction of the current operation on the graphical user interface. This scheme can improve the success rate of graphical user interface interactive tasks.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more particularly to the fields of computer vision and graphical user interface automation technology. Specifically, this disclosure relates to an interactive task processing, model training method, apparatus, electronic device, and medium. Background Technology

[0002] In the field of graphical user interface (GUI) automation, a graphical user interface automation agent typically understands the current interface state, makes decisions, and executes specific actions to complete a particular interactive task.

[0003] However, existing graphical user interface automated agents generally assume that the action is effective after making a decision and executing an action, but they cannot effectively handle the situation where the action is not actually effective, which affects the success rate of interactive tasks. Summary of the Invention

[0004] To address at least one of the aforementioned deficiencies, this disclosure provides an interactive task processing method, model training method, apparatus, electronic device, and medium.

[0005] According to a first aspect of this disclosure, an interactive task processing method is provided, the method comprising: Obtain the current interface state information of the graphical user interface; Based on the task description information, the current interface state information, and the expected interface state description corresponding to the previous operation, the validity verification result of the previous operation, the operation instruction of the current operation, and the expected interface state description of the current operation are determined. The validity verification result is generated by comparing the expected interface state description of the previous operation with the current interface state information, and the expected interface state description of the current operation is used to generate the operation instruction of the next operation. Operation commands that execute the current action on the graphical user interface.

[0006] According to a second aspect of this disclosure, a model training method is provided, the method comprising: Acquire training data, which includes the current interface state information of the sample graphical user interface, the sample task description information, the expected interface state description of the sample corresponding to the previous sample operation action, the truth value of the sample validity verification result, the truth value of the sample operation instruction of the current sample operation action, and the truth value of the expected interface state description of the current sample operation action. Training data is provided to a large model so that the large model can determine the predicted value of the sample validity verification result of the previous sample operation, the predicted value of the sample operation instruction of the current sample operation, and the predicted value of the expected interface state description of the current sample operation based on the sample's current interface state information, sample task description information, and the sample's expected interface state description corresponding to the previous sample operation action. The predicted value of the sample validity verification result is generated by comparing the sample's expected interface state description of the previous sample operation action with the sample's current interface state information. The first loss is determined by comparing the predicted value of the sample validity verification result with the true value of the predicted value of the sample validity verification result; the second loss is determined by comparing the predicted value of the sample operation instruction based on the current sample operation action with the true value of the sample operation instruction based on the current sample operation action; and the third loss is determined by comparing the predicted value of the sample expected interface state description based on the current sample operation action with the true value of the sample expected interface state description based on the current sample operation action. The comprehensive loss is determined based on the first loss, the second loss, and the third loss, and the large model is then fine-tuned based on the comprehensive loss.

[0007] According to a third aspect of this disclosure, an interactive task processing apparatus is provided, the apparatus comprising: The current interface state information acquisition module is used to acquire the current interface state information of the graphical user interface; The operation action decision module is used to determine the validity verification result of the previous operation action, the operation instruction of the current operation action, and the expected interface state description of the current operation action based on the task description information, the current interface state information, and the expected interface state description corresponding to the previous operation action. The validity verification result is generated by comparing the expected interface state description of the previous operation action with the current interface state information, and the expected interface state description of the current operation action is used to generate the operation instruction of the next operation action. Operation action execution refers to the operation instructions used to execute the current operation action on the graphical user interface.

[0008] According to a fourth aspect of this disclosure, a model training apparatus is provided, the apparatus comprising: The training data acquisition module is used to acquire training data, which includes the current interface state information of the sample graphical user interface, the sample task description information, the expected interface state description of the sample corresponding to the previous sample operation action, the truth value of the sample validity verification result, the truth value of the sample operation instruction of the current sample operation action, and the truth value of the expected interface state description of the current sample operation action. The inference module provides training data to the large model, enabling the large model to determine the predicted value of the sample validity verification result of the previous sample operation, the predicted value of the sample operation instruction of the current sample operation, and the predicted value of the expected interface state description of the current sample operation based on the current interface state information of the sample, the sample task description information, and the expected interface state description of the sample corresponding to the previous sample operation. The predicted value of the sample validity verification result is generated by comparing the expected interface state description of the previous sample operation with the current interface state information of the sample. The loss determination module is used to determine a first loss based on the predicted value of the sample validity verification result and the true value of the predicted value of the sample validity verification result, to determine a second loss based on the predicted value of the sample operation instruction of the current sample operation action and the true value of the sample operation instruction of the current sample operation action, and to determine a third loss based on the predicted value of the sample expected interface state description of the current sample operation action and the true value of the sample expected interface state description of the current sample operation action. The model training module is used to determine the comprehensive loss based on the first loss, the second loss, and the third loss, and to perform the first fine-tuning training on the large model based on the comprehensive loss.

[0009] According to a fifth aspect of this disclosure, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to at least one of the aforementioned processors; wherein, The memory stores instructions that can be executed by at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the interactive task processing or model training method.

[0010] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the above-described interactive task processing or model training method.

[0011] According to a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the above-described interactive task processing or model training method.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure.

[0014] Figure 1 , Figure 2 This is a schematic diagram of a graphical user interface in related technologies.

[0015] Figure 3 This is a flowchart illustrating an interactive task processing method provided in an embodiment of the present disclosure.

[0016] Figure 4 This is a schematic diagram of a graphical user interface provided in an embodiment of this disclosure.

[0017] Figure 5 This is a schematic flowchart of a model training method provided in an embodiment of this disclosure.

[0018] Figure 6 This is a flowchart illustrating a specific implementation of the model training method provided in this disclosure.

[0019] Figure 7 This is a schematic diagram of the structure of an interactive task processing device provided in an embodiment of this disclosure.

[0020] Figure 8 This is a schematic diagram of the structure of a model training device provided in an embodiment of this disclosure.

[0021] Figure 9 This is a block diagram of an electronic device used to implement the interactive task processing or model training method provided in the embodiments of this disclosure. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] First, let's introduce and explain several terms used in this application: Graphical User Interface (GUI) automation refers to the technology of automatically executing interactive tasks on a graphical user interface using technical means. This may include, but is not limited to, automated processes that perform operations based on predefined scripts or automated intelligent agents based on graphical user interfaces for decision-making and control.

[0024] A graphical user interface (GUI) automated agent is an intelligent agent capable of acting as an agent for users or programs, performing specified tasks through perception, decision-making, and execution within a graphical user interface environment. GUI automated agents can be driven by visual language models, meaning they perceive visual information from the graphical user interface through these models to make operational decisions.

[0025] A visual language model is a machine learning model that can understand visual information (such as interface images) and natural language information, and can perform reasoning tasks based on this information.

[0026] Currently, in the field of graphical user interface (GUI) automation, GUI automation agents generally understand the current interface state, make decisions, and execute specific operations to complete specific interactive tasks.

[0027] In related technologies, a graphical user interface automated agent can make a decision and output an action instruction (such as clicking, inputting, etc.) for an operation, then execute the operation and assume that the operation has taken effect. Then it directly proceeds to the decision-making and execution of the next operation, but does not verify whether the operation has actually taken effect.

[0028] like Figure 1 As shown, a graphical user interface automated agent performs the interactive task of searching for items in a shopping application. Figure 1 (1a) in the image represents the application's homepage. The graphical user interface (GUI) automated agent can make decisions and execute actions such as clicking the search box. If the action is successful, it will display... Figure 1 The search page shown in (1b). Based on the successful execution of this operation, the graphical user interface automated agent can perform the operation of entering search terms in the search box on this search page.

[0029] However, in actual operating environments, due to many uncertainties such as network latency, page rendering delays, page element offsets, system pop-up interruptions, or device malfunctions, the operation may not actually take effect, such as not being executed successfully or only partially executed. However, the graphical user interface automated agent cannot perceive the ineffective state and still makes subsequent action decisions based on the assumption that the operation has taken effect. This may lead to the operation entering an invalid loop, deviation from the task path, or even the final task failure.

[0030] Continued Figure 1 Examples, such as Figure 2 As shown in the diagram, after the graphical user interface automated agent makes a decision and executes the action of clicking the search box, as... Figure 2As shown in (2a), the click operation failed to trigger the search box, and therefore the search page was not displayed; that is, the click operation on the search box was ineffective. At this time, as... Figure 2 As shown in (2b), the application page displays the application homepage. If the operation of clicking the search box is still assumed to be effective and the operation of entering search terms in the search box on the search page continues, it will cause the inability to enter search terms, thus causing the interactive task of searching for items to fail.

[0031] If a graphical user interface (GUI) automated agent can effectively detect situations where actions are not actually effective and adjust its action strategy accordingly, it helps ensure the normal execution of interactive tasks. Therefore, how to effectively detect situations where actions are not effective and provide action strategies in such situations has become an important technical problem.

[0032] The interactive task processing, model training methods, apparatus, electronic devices, and media provided in the embodiments of this disclosure are intended to solve at least one of the above-mentioned technical problems of the prior art.

[0033] Figure 3 This is a flowchart illustrating the interactive task processing method provided in the embodiments of this disclosure, as shown below. Figure 3 As shown, the method may include the following steps: Step S310: Obtain the current interface state information of the graphical user interface; Step S320: Based on the task description information, the current interface state information, and the expected interface state description corresponding to the previous operation, determine the validity verification result of the previous operation, the operation instruction of the current operation, and the expected interface state description of the current operation. The validity verification result is generated by comparing the expected interface state description of the previous operation with the current interface state information, and the expected interface state description of the current operation is used to generate the operation instruction of the next operation.

[0034] Step S330: Execute the operation instruction for the current operation action on the graphical user interface.

[0035] As can be seen from the above process, this disclosure obtains the current interface state information of the graphical user interface, and determines the validity verification result of the previous operation, the operation instruction of the current operation, and the expected interface state description of the current operation based on the task description information, the current interface state information, and the expected interface state description corresponding to the previous operation. The validity verification result is generated by comparing the expected interface state description of the previous operation with the current interface state information, and the expected interface state description of the current operation is used to generate the operation instruction for the next operation. Then, the operation instruction for the current operation is executed on the graphical user interface. This disclosure effectively handles situations where the operation is not actually effective by verifying whether the previous operation is effective and making intelligent decisions about the current operation based on the verification result, ensuring the normal execution of interactive tasks and improving the success rate of interactive task execution.

[0036] The following describes in detail each step of the above process and the effects that can be further produced, with reference to the embodiments.

[0037] First, the above step S310, namely "obtaining the current interface state information of the graphical user interface", will be described in detail with reference to the embodiments.

[0038] The graphical user interface (GUI) is an interface that is presented graphically and allows users to interact with computer programs through operations such as clicking, dragging, and inputting. It may include, but is not limited to, mobile application interfaces, web pages, desktop software windows, or any other form of visual operation interface.

[0039] Current interface state information refers to the information of interface elements (such as icons, buttons, text boxes, lists, images, menus, etc.) presented in the graphical user interface at the moment of execution of the current step in the interactive task, which is used to represent the current content and state of the graphical user interface.

[0040] For example, the current interface state information may include, but is not limited to, screenshots of the graphical user interface, visual features extracted from the screenshots of the graphical user interface, and document object model tree information obtained by identifying and structurally describing interface elements.

[0041] In this embodiment of the disclosure, the current interface state information can objectively reflect the content and state of the graphical user interface at the current moment, and can serve as the decision basis for the current operation action.

[0042] The following describes in detail step S320, namely, "Based on task description information, current interface state information, and expected interface state description corresponding to the previous operation, determine the validity verification result of the previous operation, the operation instruction of the current operation, and the expected interface state description of the current operation, wherein the validity verification result is generated by comparing the expected interface state description of the previous operation with the current interface state information, and the expected interface state description of the current operation is used to generate the operation instruction of the next operation."

[0043] The task description information is a specific description of the interactive task, usually expressed in natural language.

[0044] For example, task description information can be provided by the user, such as by the user inputting natural language commands, and then the task description information can be extracted based on the user's input. Task description information can also be read from configuration files or task templates used to specify particular interactive tasks. For example, for some standardized tasks with fixed processes, their task description information can be pre-written and stored in configuration files or task templates. The system can read the task description information for the corresponding steps based on triggering conditions (such as time or events).

[0045] The expected interface state description corresponding to the previous operation refers to the description of the content or state of the graphical user interface after the previous operation was successfully executed, and can be expressed in natural language.

[0046] For example, the expected interface state description of the previous action can be generated based on the desired effect of the previous action when the previous action is decided. For instance, for the previous action: clicking the "Add to Cart" button on the item details page, the expected interface state description might be: a notification bar briefly pops up at the bottom of the page, displaying "Item successfully added to cart".

[0047] The validity verification result of the previous operation is used to indicate whether the previous operation was effective. The validity verification result is a binary value, including verification success and verification failure.

[0048] The validity verification result can be generated based on the comparison between the expected interface state description of the previous operation and the current interface state information. As an example, the current interface state information is a visual feature extracted from a screenshot of the graphical user interface. This visual feature and the previous expected interface state description (i.e., text feature) can be projected into the same semantic space for similarity measurement. When the similarity between the two meets the preset similarity condition, it means that the semantics of the expected interface state description of the previous operation and the current interface state information are consistent, and the validity verification result is successful. When the similarity between the two does not meet the preset similarity condition, it means that the semantics of the expected interface state description of the previous operation and the current interface state information are inconsistent, and the validity verification result is unsuccessful.

[0049] The operation instruction for the current operation refers to the actual interactive command for the operation to be executed at the current moment. For example, this operation instruction can be a low-level instruction that can be executed directly so that it can be executed quickly to realize the current operation.

[0050] The expected interface state description of the current operation refers to the prediction of the state that the graphical user interface will display after the current operation is successfully executed. This expected interface state description can be expressed in natural language and will be used as a validation benchmark when making decisions for the next operation.

[0051] In this embodiment of the disclosure, when the validity verification result is successful, it indicates that the previous operation has been confirmed to be effective and the state of the graphical user interface is proceeding along the correct task path. At this time, the core task of the current step is to advance the execution of the task. The next operation that can best advance the task toward the final goal in the current interface state can be deduced and its operation instruction can be output. At the same time, based on the expected effect of executing the current operation, the expected interface state description of the current operation is generated.

[0052] As an example, when searching for an item in a shopping app, the previous step was entering the item name in the search box and clicking the search button. The expected interface state is described as follows: the page redirects to the search results page, displaying a list of multiple related items. If the validity verification result of the previous action is successful, meaning the current page does indeed display a list of related items, then the generated current action could be something like "click the first item card," thus advancing the interaction task normally.

[0053] When the validity verification result is a failure, it indicates that the previous operation did not produce the expected effect and the graphical user interface is in an abnormal state. At this time, the core task of the current step is no longer to advance the execution of the task, but to handle and recover from the abnormal state. By reasoning out a restorative, corrective or exploratory operation, the graphical user interface can be automatically recovered from the abnormality, or the task can be guided back to the correct task path. At the same time, the expected interface state description of the current operation is generated.

[0054] In the event that the validity verification result is a failure, the following actions may be taken: Waiting: If the abnormal state is caused by system delays or other factors that prevent the graphical user interface from updating in a timely manner, you can pause the operation for a preset time and wait for the graphical user interface to automatically update to the normal state.

[0055] Refresh is used when the graphical user interface (GUI) freezes due to factors such as network failures. It can reload the current GUI content to restore the GUI to a normal state.

[0056] Retry: If the abnormal state of the graphical user interface is caused by temporary factors, such as network latency, an action command that is the same as or slightly modified (such as fine-tuning coordinates) as the previous operation can be generated to retry the previous operation.

[0057] Repositioning elements can overcome the failure of operations caused by positioning errors when the positioning of a certain interface element is off. This can be achieved by identifying the accurate position of the interface element and generating new operation instructions based on the new positioning information.

[0058] Alternatively, you can perform the previous action again in another way that achieves the same goal. For example, if clicking the target button fails, try using a keyboard shortcut to trigger the target button.

[0059] State rollback involves performing operations such as going back one step or closing the pop-up window, bringing the interface back to a known stable state.

[0060] Task reorganization allows for the replanning of remaining task steps based on the current deviation of the graphical user interface when it deviates significantly from its normal state.

[0061] As an example, such as Figure 4 As shown, when searching for an item in a shopping application, the first action is to click the search box on the application's homepage, with the expected interface state described as: displaying the search page. For example... Figure 4As shown in (4a), when performing the action of clicking the search box, due to the positioning deviation of the search box, the click operation failed to trigger the search box, and the graphical user interface did not change as expected. Figure 4 As shown in (4b), the application is still on its home screen. At this point, the determined current action is to reposition the search box and re-trigger it, with the expected interface state described as: displaying the search page. For example... Figure 4 As shown in (4c), the search box can be repositioned and the search box action can be retried. This action can accurately trigger the search box, and then the graphical user interface can change as expected, i.e., the search page is displayed, as shown in (4c). Figure 4 As shown in (4b).

[0062] In this embodiment of the disclosure, based on the validity verification result of the previous operation, the current operation can be dynamically switched between two operation paths: "task advancement" and "abnormal recovery". This ensures rapid advancement when the operation is executed normally to complete the task efficiently, while timely recovery when unexpected factors cause the operation to be executed abnormally, thus ensuring the success rate of the task. This enables intelligent decision-making based on the execution status of the operation, which helps to ensure the normal progress of interactive tasks, improves robustness in real complex environments, and increases the success rate of interactive task execution.

[0063] The above step S330, namely "the operation instruction for executing the current operation action on the graphical user interface", will be described in detail below with reference to the embodiments.

[0064] In this step, step S120 can be executed to determine the operation instruction of the current operation action. After the current operation action is completed, the graphical user interface will refresh its state according to its internal logic, thereby generating new "current interface state information", which will then trigger the decision step of the next operation action.

[0065] In summary, based on steps S310 to S330, this disclosure verifies whether the previous operation has taken effect and makes intelligent decisions on the current operation based on the verification results, thereby effectively handling situations where the operation has not actually taken effect, ensuring the normal execution of interactive tasks, and improving the success rate of interactive task execution.

[0066] In one optional embodiment of this disclosure, the operation instruction for the current operation and the expected interface state description for the current operation are generated based on task description information, current interface state information, and the expected interface state description corresponding to the previous operation, including: Based on the task description information, the current interface state information, the expected interface state description corresponding to the previous operation, and the context information, a prompt instruction is generated. The prompt instruction is used to prompt the large model to generate the operation instruction for the current operation and the expected interface state description for the current operation. The context information includes the historical operation before the current operation, the validity verification result of the historical operation, and the expected interface state description of the historical operation. The prompts are provided to the large model to obtain the validity verification result of the previous operation, the operation instructions for the current operation, and the expected interface state description of the current operation.

[0067] Contextual information refers to the historical interaction records during the execution of this task that can be referenced when generating the current operation action instruction.

[0068] The context information may include at least the historical operation actions, the validity verification results of the historical operation actions, and a description of the expected interface state of the historical operation actions.

[0069] Historical actions are those actions that have been performed before the current action. Multiple historical actions can be arranged in chronological order to form a historical action sequence.

[0070] The validity verification results of historical operation actions refer to the validity verification results generated in previous steps to determine whether each historical operation action is effective.

[0071] The expected interface state description of historical operation actions refers to the description of the expected effect of historical operation actions generated in previous steps.

[0072] In this embodiment of the disclosure, by constructing prompts for a large model based on contextual information, the large model can perceive the decision-making and execution status of historical operation actions, thereby making decisions that are logically consistent with historical operation actions, thus improving the decision-making quality for complex interactive tasks that require multi-step planning and increasing the success rate of task execution.

[0073] In one alternative embodiment of this disclosure, the current interface state information includes visual information of the graphical user interface, and the large model includes a visual language large model.

[0074] The visual information of the graphical user interface refers to the visual presentation of the graphical user interface at the current execution moment, recorded in the form of images. For example, it can be a screenshot of the graphical user interface or a visual feature map extracted from the screenshot that preserves spatial and visual semantics.

[0075] In this embodiment of the disclosure, by using the original visual information as a representation of the interface state, the authenticity of the interface state can be guaranteed. By using a large visual language model to fuse and reason across modal data, it is possible to accurately verify whether the operation action is effective, thereby helping to improve the accuracy of decision-making.

[0076] In one alternative embodiment of this disclosure, the prompting instruction is further used to prompt the visual language big model to generate a validity verification result by comparing the semantic consistency between the expected interface state description of the previous operation and the visual information of the graphical user interface.

[0077] In this embodiment of the disclosure, when constructing the prompt instruction, an instruction for generating validity verification results can also be embedded therein, guiding the visual language big model to focus on the semantic consistency analysis of the expected description in text form and the visual information, thereby achieving an accurate judgment on whether the previous operation action has taken effect based on the cross-modal understanding capability of the visual language big model.

[0078] For example, after receiving the aforementioned prompt instruction, the internal mechanism of the visual language large model can be as follows: The visual encoder in the visual language large model can convert visual information (such as a screenshot of a graphical user interface) into a visual semantic feature vector. This vector encodes information about the positional relationships between elements in the graphical user interface. The text encoder in the visual language large model can convert the expected interface state description into a text semantic feature vector. The visual semantic feature vector and the text semantic feature vector are mapped to the same pre-trained and optimized cross-modal semantic space, and the semantic similarity score (such as cosine similarity) between them is calculated. If the score exceeds a preset threshold, it is determined that the two are semantically consistent, and a successful validity verification result is generated; if the score does not exceed the preset threshold, it is determined that the two are semantically inconsistent, and a failed validity verification result is generated.

[0079] In one alternative embodiment of this disclosure, the prompting instruction is also used to prompt the video language big model to ignore non-semantic differences in the image user interface.

[0080] In this embodiment of the disclosure, the prompting instruction, in addition to prompting the visual language big model to pay attention to semantic consistency analysis, can also explicitly prompt the big model to ignore non-semantic changes that occur in the image user interface, such as style differences of the same functional elements, minor adjustments in position, or replacement of synonymous text.

[0081] For example, the prompt instruction may specifically include: Please determine whether the current screen content realizes or satisfies the expected interface state description in terms of core semantics, while ignoring non-substantial style differences, minor position adjustments, or replacement of synonymous text, and focus on determining whether the key state changes or element changes expressed in the expected interface state description have occurred in the current screen.

[0082] For example, the expected interface state description is "The quantity of items in the shopping cart has increased by 1". If the number on the shopping cart icon in the graphical user interface changes from "3" to "4", it can be determined that this conforms to the semantics of "the quantity has increased by 1". In this case, even if the color or font of the number changes, or its position is slightly adjusted, as long as the core semantics of "the quantity has increased by 1" are satisfied, the validity verification result can be determined to have failed.

[0083] The solution provided in this disclosure, through the cross-modal understanding capability of the visual language big model, can accurately determine whether the previous operation has taken effect from the semantic consistency level. At the same time, it can also perceive non-semantic changes in the image user interface, avoiding the impact of non-semantic changes (such as style differences or minor adjustments in the position of functional elements) on the accuracy of determining whether the previous operation has taken effect, thereby providing a robust operation effectiveness determination method based on intelligent understanding.

[0084] In one optional embodiment of this disclosure, the prompting instruction is further used to prompt the large model to output a thought chain. The thought chain includes the validity verification result of the previous operation, the operation instruction of the current operation, and the expected interface state description of the current operation. In response to the validity verification result being a verification failure, the thought chain also includes the difference information between the current interface state information and the expected interface state description corresponding to the previous operation, as well as the recovery strategy information for controlling the current interface state to be restored to the state of the interface before the previous operation.

[0085] Among them, the thought chain refers to the explicit and structured natural language expression of the internal reasoning steps or logical basis of a large model in the process of generating reasoning results, which is used to explain how the large model thinks and draws conclusions.

[0086] In addition to the validity verification result of the previous operation, the operation instruction of the current operation, and the expected interface state description of the current operation, the thought chain in this embodiment also includes, when the previous operation fails to be verified, the difference information between the current interface state information and the expected interface state description corresponding to the previous operation, as well as the recovery strategy information for controlling the current interface state to be restored to the state of the interface before the previous operation.

[0087] The difference information is a detailed description of the difference between the current interface state information and the expected interface state description corresponding to the previous operation. For example, refer to... Figure 4 In the example shown in (4b), the difference information is: the expected interface state description states that clicking the search box should trigger the search page, and the application homepage should be displayed on the current interface.

[0088] Recovery strategy information refers to the strategy description derived from the aforementioned difference information, used to automatically recover the graphical user interface from an anomaly or to guide the task back to the correct task path. For example, refer to... Figure 4 In the example shown, the recovery strategy information is: since the click operation did not trigger the search box, reposition the search box and then retry the same search click action.

[0089] In this embodiment of the disclosure, by prompting the large model to perform explicit reasoning and outputting its thinking process in a structured way as a thought chain, the large model can be constrained to think according to the logical steps of the thought chain, thereby improving the accuracy and robustness of the final decision. At the same time, the output thought chain can also provide a basis for fault diagnosis and recovery for complex faults.

[0090] In one optional embodiment of this disclosure, before generating the first operation action instruction for the interactive task, the method further includes: Based on the task description information and the initial interface state information, determine the operation command for the first operation and the expected interface state description for the first operation.

[0091] At the start of an interactive task, when making a decision on the first operation, there is no expected interface state description of the previous operation, nor is there any context information. The operation instruction and expected interface state description of the first operation can be determined directly based on the task description information and the initial interface state information.

[0092] For example, a prompt instruction can be constructed for the initial state. This prompt instruction is used to guide the large model to make a decision on the operation instruction for the first operation and generate the expected interface state description for the first operation based solely on the task description information and the initial interface state information.

[0093] The solution provided in this disclosure offers a method for processing the starting point of an interactive task. It ensures that when deciding on the first action, a description of its expected interface state is generated simultaneously, providing a foundation for verifying the effectiveness of the action in subsequent steps. Furthermore, the accuracy of the first action and its expected interface state description directly affects the entire interactive task's trajectory. Ensuring the accuracy of the first action and its expected interface state description provides a solid foundation for the smooth execution of subsequent tasks.

[0094] In one alternative embodiment of this disclosure, after executing the operation instruction for the current operation action on the graphical user interface, the method further includes: In response to the operation command that executes the current operation action, the interface state information of the graphical user interface matches the task objective of the interactive task, and the execution of the interactive task ends.

[0095] In particular, during the continuous operation of interactive tasks, it is necessary to judge the execution status of the tasks in a timely manner so as to terminate the tasks in a timely manner when the task objectives are achieved, and avoid meaningless operations.

[0096] In this embodiment of the disclosure, after each step of the operation is performed, it can be determined whether the interface state information of the graphical user interface has met the task objective of the interaction task. Specifically, the semantic consistency matching between the interface state information of the graphical user interface and the task description information can be performed to determine whether the task objective has been achieved.

[0097] For example, the interface state information of the graphical user interface (such as the current screenshot) can be input into the visual language big model along with the task description. Based on the multimodal understanding capability of the video language big model, it can be determined whether the interactive task has been completed.

[0098] The solution provided in this disclosure achieves complete automated control of interactive tasks by autonomously sensing whether the interactive task has been completed and automatically stopping after the task is completed.

[0099] Figure 5 This is a flowchart illustrating the model training method provided in the embodiments of this disclosure, as shown below. Figure 5 As shown, the method may include the following steps: Step S510: Obtain training data. The training data includes the current interface state information of the sample graphical user interface, the sample task description information, the expected interface state description of the sample corresponding to the previous sample operation action, the truth value of the sample validity verification result, the truth value of the sample operation instruction of the current sample operation action, and the truth value of the expected interface state description of the current sample operation action.

[0100] Step S520: Provide the training data to the large model so that the large model can determine the predicted value of the sample validity verification result of the previous sample operation, the predicted value of the sample operation instruction of the current sample operation, and the predicted value of the expected interface state description of the current sample operation based on the sample's current interface state information, sample task description information, and the sample's expected interface state description corresponding to the previous sample operation action. The predicted value of the sample validity verification result is generated by comparing the sample's expected interface state description of the previous sample operation action with the sample's current interface state information.

[0101] Step S530: Determine the first loss based on the predicted value of the sample validity verification result and the true value of the predicted value of the sample validity verification result; determine the second loss based on the predicted value of the sample operation instruction based on the current sample operation action and the true value of the sample operation instruction based on the current sample operation action; and determine the third loss based on the predicted value of the sample expected interface state description based on the current sample operation action and the true value of the sample expected interface state description based on the current sample operation action. Step S540: Determine the comprehensive loss based on the first loss, the second loss, and the third loss, and perform the first fine-tuning training on the large model based on the comprehensive loss.

[0102] Among them, the current interface state information of the sample, the sample task description information and the expected interface state description of the sample corresponding to the previous sample operation action obtained in step S520 can correspond one-to-one with the current interface state information, task description information and the expected interface state description of the previous sample operation action in step S310.

[0103] The predicted values ​​of the validity verification results of the large model output in step S520 above, the predicted values ​​of the current sample operation instructions and the predicted values ​​of the expected interface state description of the current sample can correspond one-to-one with the three inference results in step S310 above: the validity verification results, the operation instructions of the current operation, and the expected interface state description.

[0104] The first loss is determined based on the difference between the predicted value and the true value of the sample validity verification result. Optimizing the first loss can improve the accuracy of the large model in verifying whether the previous operation has taken effect.

[0105] The second loss is determined based on the difference between the predicted value of the sample operation command and the true value of the operation command. Optimizing the second loss can improve the accuracy of the large model in generating operation commands for the current operation action, thereby improving the quality of decision-making.

[0106] The third loss is determined by the difference between the predicted value of the expected interface state description of the current sample operation and the true value of the expected interface state description of the current sample operation. Optimizing the third loss can enable large models to generate high-quality expected interface state descriptions.

[0107] For example, the overall loss can be expressed by the following Formula 1: ;(Formula 1) in, Indicates the overall loss. Indicates the third loss. Indicates the first loss. This indicates the second loss. and This is a hyperparameter.

[0108] In this example, when the predicted value of the sample validity verification result matches the true value, a positive value (such as +1) can be assigned to the first loss to achieve a positive reward; when the predicted value of the sample validity verification result does not match the true value (such as failure to detect verification failure or false detection of verification failure), a negative value (such as -2 or -0.5) can be assigned to the first loss to achieve a penalty. By targeting the misjudgment of the validity verification result, the accumulation of errors can be effectively suppressed.

[0109] When the predicted value of a sample operation instruction matches the true value of the sample operation instruction, a positive value (such as +1) can be assigned to the second loss to achieve a positive reward. When the predicted value of the sample expected interface state description has a high semantic similarity to the true value of the sample expected interface state description, a positive value can be assigned to the third loss to achieve a positive reward.

[0110] The solution provided in this disclosure improves the training effect of large models by performing collaborative optimization in the three dimensions of verification, decision-making, and description during the fine-tuning training process.

[0111] In one optional embodiment of this disclosure, the third loss is determined based on the predicted value of the sample's expected interface state description based on the current sample operation action and the true value of the sample's expected interface state description based on the current sample operation action, including: Determine the text similarity between the predicted value of the expected interface state description of the current sample operation action and the true value of the expected interface state description of the current sample operation action. The third loss is determined based on text similarity.

[0112] In calculating the third loss, the text similarity between the predicted value of the expected interface state description of the sample and the true value of the expected interface state description of the sample can be used as a metric to assess whether the core meaning, intent and information content expressed by the two are consistent, rather than simply overlapping in terms of characters, word order or surface form, thereby ensuring the accuracy of the determined third loss.

[0113] For example, "pop up a dialog box" and "appear a prompt window" are not similar at the character or word level, but they have a high degree of textual semantic similarity. Based on the solution, it can be determined that the two are essentially the same.

[0114] The solution provided in this disclosure determines a third loss based on the text similarity between the predicted value and the true value of the expected interface state description of the sample. This ensures that even if the wording differs (e.g., using "dialog box" instead of "pop-up"), the predicted value and the true value remain semantically consistent, resulting in low loss. This guides the large model to understand the essence of the interface state changes represented by the expected interface, enabling it to learn and generate more essential and flexible expected interface state descriptions, rather than learning fixed text templates. This allows the large model to generate accurate and usable expected interface state descriptions even when faced with interface states that have not appeared during training, enhancing the reliability of the solution.

[0115] In one optional embodiment of this disclosure, the training data includes a first training sample and a second training sample, wherein the true value of the sample validity verification result corresponding to the first training sample is verification success, and the true value of the sample validity verification result corresponding to the second training sample is verification failure. The method further includes: The large model is then fine-tuned based on the training data.

[0116] The training data may include a first training sample constructed from data of operational actions that have been verified as actually effective, and a second training sample constructed from data of operational actions that have been verified as not actually effective.

[0117] For example, the first training sample and the second training sample can be constructed based on data from performing real interactive tasks.

[0118] In this embodiment, a large model is fine-tuned by constructing mixed training data containing both successful and failed samples, enabling the model to learn prior knowledge about the success and failure of operational actions. Compared to fine-tuning using only training data containing successful samples, this approach allows the large model to learn prior knowledge about recognizing failures and performing recovery, thus improving the robustness of the large model.

[0119] In this embodiment of the disclosure, the first fine-tuning training and the second fine-tuning training can be performed separately. For example, the second fine-tuning training can be performed first, followed by the first fine-tuning training.

[0120] For example, Figure 6 This is a flowchart illustrating a specific implementation of the model training method provided in this disclosure.

[0121] like Figure 6As shown, the first stage of fine-tuning training (i.e., the second stage of fine-tuning training) can specifically employ robust supervised fine-tuning (Robust SFT). The training data used in this stage can include both first and second training samples. By using mixed training data containing both successful and synthesized failure samples to fine-tune the large model, it is possible for the large model to learn prior knowledge about identifying failures and performing recovery, thus improving the robustness of the large model.

[0122] In this example, after completing the first stage of fine-tuning training, the second stage of fine-tuning training can be performed.

[0123] The second stage of fine-tuning training (i.e., the first stage of fine-tuning training) can specifically employ a Verification-Action-Effect based Group Relative Policy Optimization (VAE-GRPO) approach. This stage can effectively improve the training performance of large models by constructing a first loss, a second loss, and a third loss to collaboratively optimize across the verification, decision-making, and description dimensions.

[0124] The methods provided in this disclosure can be applied to the following scenarios: automated testing and maintenance of graphical interfaces, interactive tasks automatically executed by intelligent assistants (such as automatic ticket booking, shopping, sending emails, etc.), remote device control, and graphical user interface intelligent agent frameworks for multi-step task execution.

[0125] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0126] According to another embodiment, an interactive task processing apparatus is provided. Figure 7 A schematic diagram of the structure of the interactive task processing device according to one embodiment is shown. Figure 7 As shown, the interactive task processing device 700 includes: The current interface state information acquisition module 710 is used to acquire the current interface state information of the graphical user interface; The operation action decision module 720 is used to determine the validity verification result of the previous operation action, the operation instruction of the current operation action, and the expected interface state description of the current operation action based on the task description information, the current interface state information, and the expected interface state description corresponding to the previous operation action. The validity verification result is generated based on the comparison between the expected interface state description of the previous operation action and the current interface state information. The expected interface state description of the current operation action is used to generate the operation instruction of the next operation action. The operation action execution module 730 is used to execute operation instructions for the current operation action on the graphical user interface.

[0127] As an optional approach, the operation action decision module 720 is specifically used for: Based on the task description information, the current interface state information, the expected interface state description corresponding to the previous operation, and the context information, a prompt instruction is generated. The prompt instruction is used to prompt the large model to generate the operation instruction for the current operation and the expected interface state description for the current operation. The context information includes the historical operation before the current operation, the validity verification result of the historical operation, and the expected interface state description of the historical operation. The prompts are provided to the large model to obtain the validity verification result of the previous operation, the operation instructions for the current operation, and the expected interface state description of the current operation.

[0128] As an optional approach, the current interface state information includes visual information of the graphical user interface, and the large model includes a visual language large model.

[0129] As an optional approach, prompts are also used to prompt the visual language big model to generate validity verification results by comparing the semantic consistency of the expected interface state description of the previous operation with the visual information of the graphical user interface.

[0130] As an optional approach, the prompting instruction is also used to prompt the large model to output a thought chain. The thought chain includes the validity verification result of the previous operation, the operation instruction of the current operation, and the expected interface state description of the current operation. In response to the validity verification result being a verification failure, the thought chain also includes the difference information between the current interface state information and the expected interface state description corresponding to the previous operation, as well as the recovery strategy information used to control the current interface state to be restored to the state of the interface before the previous operation.

[0131] As an alternative, the above-mentioned device further includes a first-step execution module (not shown in the figure), used for: Before generating the first operation action instruction of the interactive task, the operation instruction of the first operation action and the expected interface state description of the first operation action are determined based on the task description information and the initial interface state information.

[0132] As an alternative, the above-mentioned device also includes a task termination module (not shown in the figure), used for: After the operation instruction for executing the current operation action on the graphical user interface is executed, in response to the interface state information of the graphical user interface matching the task objective of the interaction task after the operation instruction for executing the current operation action is executed, the execution of the interaction task ends.

[0133] According to another embodiment, a model training apparatus is provided. Figure 8 A schematic diagram of the model training apparatus according to one embodiment is shown. Figure 8 As shown, the model training device 800 includes: The training data acquisition module 810 is used to acquire training data, which includes the current interface state information of the sample graphical user interface, the sample task description information, the expected interface state description of the sample corresponding to the previous sample operation action, the truth value of the sample validity verification result, the truth value of the sample operation instruction of the current sample operation action, and the truth value of the expected interface state description of the current sample operation action. The inference module 820 is used to provide training data to the large model so that the large model can determine the predicted value of the sample validity verification result of the previous sample operation, the predicted value of the sample operation instruction of the current sample operation, and the predicted value of the expected interface state description of the current sample operation based on the sample's current interface state information, sample task description information, and the sample's expected interface state description corresponding to the previous sample operation action. The predicted value of the sample validity verification result is generated by comparing the sample's expected interface state description of the previous sample operation action with the sample's current interface state information. The loss determination module 830 is used to determine a first loss based on the predicted value of the sample validity verification result and the true value of the predicted value of the sample validity verification result, to determine a second loss based on the predicted value of the sample operation instruction of the current sample operation action and the true value of the sample operation instruction of the current sample operation action, and to determine a third loss based on the predicted value of the sample expected interface state description of the current sample operation action and the true value of the sample expected interface state description of the current sample operation action. The model training module 840 is used to determine the comprehensive loss based on the first loss, the second loss, and the third loss, and to perform the first fine-tuning training on the large model based on the comprehensive loss.

[0134] As an optional approach, the training data includes a first training sample and a second training sample, wherein a true value for the sample validity verification result corresponding to the first training sample is considered successful, and a true value for the sample validity verification result corresponding to the second training sample is considered unsuccessful. The model training module 840 is also used for: The large model is then fine-tuned based on the training data.

[0135] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0136] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0137] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0138] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0139] like Figure 9As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0140] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0141] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the interactive task processing or model training methods described above. For example, in some embodiments, the interactive task processing or model training methods described above can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by computing unit 901, one or more steps of the interactive task processing or model training methods described above can be performed. Alternatively, in other embodiments, computing unit 901 may be configured by any other suitable means (e.g., by means of firmware) to perform the interactive task processing or model training methods described above.

[0142] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0146] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0147] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0148] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0149] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An interactive task processing method, comprising: Obtain the current interface state information of the graphical user interface; Based on the task description information, the current interface state information, and the expected interface state description corresponding to the previous operation, the validity verification result of the previous operation, the operation instruction of the current operation, and the expected interface state description of the current operation are determined. The validity verification result is generated based on the comparison between the expected interface state description of the previous operation and the current interface state information. The expected interface state description of the current operation is used to generate the operation instruction of the next operation. An operation instruction is given to the graphical user interface to execute the current operation action.

2. The method according to claim 1, wherein, The process of generating the operation instruction for the current operation and the expected interface state description for the current operation based on the task description information, the current interface state information, and the expected interface state description corresponding to the previous operation includes: Based on the task description information, the current interface state information, the expected interface state description corresponding to the previous operation, and the context information, a prompt instruction is generated. The prompt instruction is used to prompt the large model to generate the operation instruction for the current operation and the expected interface state description of the current operation based on the task description information, the current interface state information, the expected interface state description corresponding to the previous operation, and the context information. The context information includes historical operation actions before the current operation, the validity verification results of the historical operation actions, and the expected interface state description of the historical operation actions. The prompt instructions are provided to the large model to obtain the validity verification result of the previous operation, the operation instructions of the current operation, and the expected interface state description of the current operation output by the large model.

3. The method according to claim 2, wherein, The current interface state information includes the visual information of the graphical user interface, and the large model includes the visual language large model.

4. The method according to claim 3, wherein, The prompting instruction is also used to prompt the visual language big model to generate the validity verification result by comparing the semantic consistency between the expected interface state description of the previous operation and the visual information of the graphical user interface.

5. The method according to any one of claims 2-4, wherein, The prompting instruction is also used to prompt the large model to output a thought chain. The thought chain includes the validity verification result of the previous operation, the operation instruction of the current operation, and the expected interface state description of the current operation. In response to the validity verification result being a verification failure, the thought chain also includes the difference information between the current interface state information and the expected interface state description corresponding to the previous operation, as well as the recovery strategy information for controlling the current interface state to be restored to the state of the interface before the previous operation.

6. The method according to any one of claims 1-5, wherein before generating the first operation action instruction of the interactive task, the method further comprises: Based on the task description information and the initial interface state information, determine the operation instruction for the first operation and the expected interface state description for the first operation.

7. The method according to any one of claims 1-6, wherein after the operation instruction for executing the current operation action on the graphical user interface is provided, the method further comprises: In response to the operation instruction to execute the current operation, the interface state information of the graphical user interface matches the task objective of the interaction task, and the execution of the interaction task ends.

8. A model training method, comprising: Acquire training data, which includes the current interface state information of the sample graphical user interface, the sample task description information, the expected interface state description of the sample corresponding to the previous sample operation action, the truth value of the sample validity verification result, the truth value of the sample operation instruction of the current sample operation action, and the truth value of the expected interface state description of the current sample operation action. The training data is provided to a large model so that the large model can determine the predicted value of the sample validity verification result of the previous sample operation, the predicted value of the sample operation instruction of the current sample operation, and the predicted value of the expected interface state description of the current sample operation based on the current interface state information of the sample, the sample task description information, and the expected interface state description of the sample corresponding to the previous sample operation. The predicted value of the sample validity verification result is generated by comparing the expected interface state description of the previous sample operation with the current interface state information of the sample. A first loss is determined based on the predicted value of the sample validity verification result and the true value of the predicted value of the sample validity verification result; a second loss is determined based on the predicted value of the sample operation instruction of the current sample operation action and the true value of the sample operation instruction of the current sample operation action; and a third loss is determined based on the predicted value of the sample expected interface state description of the current sample operation action and the true value of the sample expected interface state description of the current sample operation action. A comprehensive loss is determined based on the first loss, the second loss, and the third loss, and the large model is then subjected to a first fine-tuning training based on the comprehensive loss.

9. The method according to claim 8, wherein, The determination of the third loss based on the predicted value of the sample expected interface state description based on the current sample operation action and the true value of the sample expected interface state description based on the current sample operation action includes: Determine the text similarity between the predicted value of the expected interface state description of the current sample operation action and the true value of the expected interface state description of the current sample operation action. The third loss is determined based on the text similarity.

10. The method according to claim 8 or 9, wherein, The training data includes a first training sample and a second training sample, wherein a true value for the sample validity verification result corresponding to the first training sample is verification success, and a true value for the sample validity verification result corresponding to the second training sample is verification failure. The method further includes: The large model is then subjected to a second fine-tuning training based on the training data.

11. An interactive task processing device, comprising: The current interface state information acquisition module is used to acquire the current interface state information of the graphical user interface; The operation action decision module is used to determine the validity verification result of the previous operation action, the operation instruction of the current operation action, and the expected interface state description of the current operation action based on the task description information, the current interface state information, and the expected interface state description corresponding to the previous operation action. The validity verification result is generated based on the comparison between the expected interface state description of the previous operation action and the current interface state information. The expected interface state description of the current operation action is used to generate the operation instruction of the next operation action. The operation action execution module is used to execute the operation instructions of the current operation action on the graphical user interface.

12. A model training device, comprising: The training data acquisition module is used to acquire training data, which includes the current interface state information of the sample graphical user interface, the sample task description information, the expected interface state description of the sample corresponding to the previous sample operation action, the truth value of the sample validity verification result, the truth value of the sample operation instruction of the current sample operation action, and the truth value of the expected interface state description of the current sample operation action. The inference module is used to provide the training data to the large model, so that the large model can determine the predicted value of the sample validity verification result of the previous sample operation, the predicted value of the sample operation instruction of the current sample operation, and the predicted value of the expected interface state description of the current sample operation based on the current interface state information of the sample, the sample task description information, and the expected interface state description of the previous sample operation. The predicted value of the sample validity verification result is generated by comparing the expected interface state description of the previous sample operation with the current interface state information of the sample. The loss determination module is used to determine a first loss based on the predicted value of the sample validity verification result and the true value of the predicted value of the sample validity verification result, to determine a second loss based on the predicted value of the sample operation instruction of the current sample operation action and the true value of the sample operation instruction of the current sample operation action, and to determine a third loss based on the predicted value of the sample expected interface state description of the current sample operation action and the true value of the sample expected interface state description of the current sample operation action. The model training module is used to determine a comprehensive loss based on the first loss, the second loss, and the third loss, and to perform a first fine-tuning training on the large model based on the comprehensive loss.

13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images