A method, device and equipment for training an interface navigation model

Abstract

The embodiment of the specification discloses a training method, device and equipment of an interface navigation model, which comprises the following steps: obtaining a plurality of task description information and an initial state and a target action associated with each task description information, the task description information being used for describing a specified task expected by a user to be completed in a graphical user interface; inputting each task description information and the associated initial state into an interface navigation model, and taking the target action associated with the initial state as a prompt condition to guide the interface navigation model to generate a first exploration track corresponding to the task description information, the first exploration track being used for describing a complete process of the interface navigation model executing the specified task under the guidance of the target action; and then, based on the first exploration track of the plurality of task description information, adjusting the strategy of the interface navigation model to determine a strengthened interface navigation model, the strengthened interface navigation model being used for predicting and executing a task navigation action in the graphical user interface.

Description

Technical Field

[0001] This document relates to the field of artificial intelligence technology, and in particular to a training method, apparatus and equipment for an interface navigation model. Background Technology

[0002] Graphical User Interface (GUI) Automation aims to build intelligent agents (such as interface navigation models) capable of understanding natural language instructions, observing interface screenshots, and completing user tasks through multi-step decision-making. These agents autonomously understand natural language task descriptions, analyze interface screenshots, and interact with software or systems through interactive elements in the graphical interface, such as moving or switching between different functions, pages, or content to complete user-specified tasks. The training process of the intelligent agent plays a crucial role in its ability to complete specified tasks, and the training effect significantly impacts the accuracy of task completion.

[0003] Currently, reinforcement learning based on verifiable rewards can be used to train agents. This involves using binary signals indicating whether a given task was successfully completed to optimize the agent's policy, learning from trial and error without requiring extensive manual labeling. However, this training method involves a vast exploration space, where random exploration is almost impossible to discover the correct action. Furthermore, when the task difficulty exceeds the agent's current capabilities, all exploration paths end in failure, resulting in identical rewards for all paths and falling into a zero-odds trap. This ultimately eliminates the learning signal, leading to poor training performance and an inability to accurately complete the user's specified tasks. Therefore, a technical solution with better training performance and more accurate interface navigation results is needed. Summary of the Invention

[0004] The purpose of the embodiments in this specification is to provide a technical solution with better training results and more accurate interface navigation results.

[0005] To achieve the above technical solution, the embodiments in this specification are implemented as follows: This specification provides an embodiment of a training method for an interface navigation model. The method includes: acquiring multiple task description information, and an initial state and target action associated with each task description information, wherein the task description information describes a specified task that a user expects to complete within a graphical user interface; for any task description information, inputting the task description information and the associated initial state into the interface navigation model, and using the target action associated with the initial state as a prompt condition to guide the interface navigation model to generate a first exploration trajectory corresponding to the task description information, wherein the first exploration trajectory describes the complete process of the interface navigation model executing the specified task under the guidance of the target action; and adjusting the strategy of the interface navigation model based on the first exploration trajectory of the multiple task description information to determine a reinforcement-learned interface navigation model, wherein the reinforcement-learned interface navigation model is used to predict and execute task navigation actions within the graphical user interface.

[0006] This specification provides an embodiment of a training device for an interface navigation model. The device includes: a trajectory generation module, configured to acquire multiple task description information, and an initial state and target action associated with each task description information, wherein the task description information describes a specified task that a user expects to complete within a graphical user interface; for any task description information, inputting the task description information and the associated initial state into the interface navigation model, and using the target action associated with the initial state as a prompt condition to guide the interface navigation model to generate a first exploration trajectory corresponding to the task description information, wherein the first exploration trajectory describes the complete process of the interface navigation model executing the specified task under the guidance of the target action; and a strategy adjustment module, configured to adjust the strategy of the interface navigation model based on the first exploration trajectory of the multiple task description information to determine a reinforcement-learned interface navigation model, wherein the reinforcement-learned interface navigation model is used to predict and execute task navigation actions within the graphical user interface.

[0007] This specification provides an embodiment of a training device for an interface navigation model. The device includes a processor and a memory arranged to store computer-executable instructions. When executed, the executable instructions cause the processor to: acquire multiple task description information, and an initial state and target action associated with each task description information, wherein the task description information describes a specified task that a user expects to complete within a graphical user interface; for any task description information, input the task description information and the associated initial state into the interface navigation model, and use the target action associated with the initial state as a prompt condition to guide the interface navigation model to generate a first exploration trajectory corresponding to the task description information, wherein the first exploration trajectory describes the complete process of the interface navigation model executing the specified task under the guidance of the target action; and adjust the strategy of the interface navigation model based on the first exploration trajectory of the multiple task description information to determine a reinforcement-learned interface navigation model, wherein the reinforcement-learned interface navigation model is used to predict and execute task navigation actions within the graphical user interface.

[0008] This specification also provides a storage medium for storing computer-executable instructions. When executed by a processor, these instructions implement the following process: acquiring multiple task description information, and an initial state and target action associated with each task description information, wherein the task description information describes a specified task that the user expects to complete within a graphical user interface; for any task description information, inputting the task description information and its associated initial state into an interface navigation model, and using the target action associated with the initial state as a prompt condition to guide the interface navigation model to generate a first exploration trajectory corresponding to the task description information, wherein the first exploration trajectory describes the complete process of the interface navigation model executing the specified task under the guidance of the target action; and adjusting the strategy of the interface navigation model based on the first exploration trajectory of the multiple task description information to determine a reinforcement-learned interface navigation model, wherein the reinforcement-learned interface navigation model is used to predict and execute task navigation actions within the graphical user interface.

[0009] This specification also provides a computer program product, including a computer program that, when executed by a processor, implements the following process: acquiring multiple task description information, and an initial state and target action associated with each task description information, wherein the task description information is used to describe a specified task that the user expects to complete within a graphical user interface; for any task description information, inputting the task description information and the associated initial state into an interface navigation model, and using the target action associated with the initial state as a prompt condition to guide the interface navigation model to generate a first exploration trajectory corresponding to the task description information, wherein the first exploration trajectory is used to describe the complete process of the interface navigation model executing the specified task under the guidance of the target action; adjusting the strategy of the interface navigation model based on the first exploration trajectory of the multiple task description information to determine a reinforcement learning-enhanced interface navigation model, wherein the reinforcement learning-enhanced interface navigation model is used to predict and execute task navigation actions within the graphical user interface. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Figure 1 This is a flowchart illustrating a training method for an interface navigation model as described in this specification. Figure 2 This is a flowchart illustrating another training method for the interface navigation model described in this manual. Figure 3 This is a flowchart illustrating a method for determining the target advantage value of an exploration trajectory, as described in this specification. Figure 4 This is a flowchart illustrating a method for determining the modulation factor of an exploration trajectory, as described in this specification. Figure 5 This is a flowchart illustrating a process for constructing a strategy optimization function and adjusting the strategy of the interface navigation model as described in this specification. Figure 6 This document presents a flowchart illustrating a process for implementing an interface navigation task based on an interface navigation model derived from reinforcement learning. Figure 7a This is a flowchart illustrating another training method for an interface navigation model as described in this specification. Figure 7b This is a logical diagram illustrating another training method for an interface navigation model in this specification. Figure 8 This is a schematic diagram of a training device for an interface navigation model as described in this specification; Figure 9 This is a schematic diagram of a training device for an interface navigation model as described in this specification. Detailed Implementation

[0011] This specification provides a method, apparatus, and device for training an interface navigation model.

[0012] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0013] Graphical User Interface (GUI) Automation aims to build intelligent agents (such as interface navigation models) capable of understanding natural language instructions, observing interface screenshots, and completing user tasks through multi-step decision-making. These agents autonomously understand natural language task descriptions, analyze interface screenshots, and interact with software or systems through interactive elements in the graphical interface, such as moving or switching between different functions, pages, or content to complete user-specified tasks. The training process of the intelligent agent plays a crucial role in its ability to complete specified tasks, and the training effect significantly impacts the accuracy of task completion.

[0014] Currently, reinforcement learning based on verifiable rewards can be used to train agents. This involves using binary signals indicating whether a given task was successfully completed to optimize the agent's policy, allowing learning from trial and error without extensive manual labeling. However, this training method involves a vast exploration space, where random exploration is almost impossible to discover the correct action. Furthermore, when the task difficulty exceeds the agent's current capabilities, all exploration paths end in failure, resulting in identical rewards for all paths, leading to a zero-odds trap. This completely eliminates the learning signal, fundamentally hindering the agent's learning process and resulting in poor training performance. Consequently, the trained agent is unable to accurately complete the user's designated task.

[0015] To address the aforementioned issues, an attempt was made to introduce a more powerful expert demonstration model to provide high-quality guiding signals. However, the exploration trajectories generated by the expert demonstration model far exceeded the distribution support of the agent, leading to distribution mismatch, severe entropy increase, and chaotic training of the agent, resulting in poor learning outcomes.

[0016] To address this, the present invention proposes a superior data processing scheme. This scheme involves acquiring multiple task descriptions and their associated initial states and target actions. Each task description and its associated initial state are input into the interface navigation model, with the target action used as a prompt. This guides the interface navigation model to generate a first exploration trajectory corresponding to the task descriptions. Then, based on the first exploration trajectories of the multiple task descriptions, the interface navigation model's strategy is adjusted to determine the reinforcement-learned interface navigation model. This reinforcement-learned interface navigation model can be used to perform task navigation operations within the interface. By using the target action as a prompt to generate a self-explanatory exploration trajectory within a distribution, the zero-odds problem is effectively solved, improving the training performance of the interface navigation model. Specific processing details can be found in the following embodiments.

[0017] like Figure 1 As shown in the embodiments of this specification, a training method for an interface navigation model is provided. The execution subject of this method can be a terminal device or a server, etc. The terminal device can be a mobile terminal device such as a mobile phone or tablet computer, a computer device such as a laptop or desktop computer, or an IoT device (specifically, a smartwatch, in-vehicle device, etc.). The server can be a single server or a server cluster composed of multiple servers. The server can be a backend server in fields such as finance or online shopping, or a backend server of an application. This embodiment uses a server as the execution subject for detailed explanation. For the case where the execution subject is a terminal device, please refer to the following server case processing, which will not be repeated here. The method may specifically include the following steps: In step S102, multiple task description information and the initial state and target action associated with each task description information are obtained.

[0018] The task description information describes the specific task that the user expects to complete within the graphical user interface (GUI). A GUI is an interface design that allows users to interact with a computer system through graphical elements (such as windows, charts, buttons, menus, etc.) rather than plain text commands. The specified task refers to reproducing the functionality achievable within the GUI. For example, the task description information could be "Search for 'smartphone' on a shopping website." The initial state associated with the task description information refers to the starting point for executing the specified task. The initial state can be a screenshot of the GUI, a specified element within the screenshot, etc., such as the homepage or search bar of a shopping website. The target action associated with the task description information refers to the actual action path required to complete the specified task. This action path can include one or more actions, such as "Click the search box - type 'smartphone' - click the search button."

[0019] During implementation, multiple task description information can be acquired. This information can be historical task descriptions processed by the interface navigation model or task descriptions configured by technical personnel. Then, an initial state can be randomly assigned to each task description information. There can be one or more initial states, and a corresponding real path can be manually configured for each initial state to determine the target action associated with the task description information.

[0020] It should be understood that each task description can be associated with multiple initial states and multiple target actions. There is a relationship between the initial states and the target actions. Each set of relationships contains one initial state and one target action, and different sets of relationships can contain the same initial state. That is, an initial state can be randomly assigned to each task description. For each initial state, there may be multiple actual paths to achieve the specified task. However, based on the same task description and the same initial state, different input combinations can be formed with different actual paths. Each input combination has only one relationship between "task description - initial state - target action", and each type of task description can correspond to multiple input combinations.

[0021] In step S104, for any task description information, the task description information and the associated initial state are input into the interface navigation model, and the target action associated with the initial state is used as a prompt condition to guide the interface navigation model to generate the first exploration trajectory corresponding to the task description information.

[0022] The interface navigation model is an untrained agent used to automate the graphical user interface (GUI), but it can still predict and execute navigation actions within the GUI. The exploration trajectory is a complete sequence of interactions between the interface navigation model and the interface environment. The first exploration trajectory describes the complete process of the interface navigation model executing a specified task under the guidance of a target action. The first exploration trajectory generated under the guidance of the target action is a self-explanatory trajectory; it not only contains the complete sequence of interactions between the interface navigation model and the environment but also reflects the basis for action selection through natural language, structured graphs, or symbol sequences, actively explaining "why this action was chosen." That is, the first exploration trajectory may include the initial state, the action taken by the agent (i.e., the first predicted action), the reward from environmental feedback, the basis for action selection, and the trajectory length.

[0023] In implementation, task description information, the associated initial state, and the associated target action can be provided to the interface navigation model. The task description information and the initial state are used as input to the interface navigation model, and the target action is used as a prompt condition to guide the interface navigation model to output a first predicted action and a first exploration trajectory. The first predicted action is the task navigation action predicted by the interface navigation model for the specified task under the guidance of the target action. The first predicted action may be consistent with or inconsistent with the target action (i.e., the actual action). Constraints can be enforced through a loss function or hard cues, allowing the interface navigation model to directly use the target action as the first predicted action. Alternatively, soft cues can be used to allow the generation of a first predicted action that is different from the target action but logically reasonable. This embodiment does not limit this.

[0024] For example, if the task description information t is associated with multiple initial states s1, s2, and s3, where initial state s1 is associated with target action a1, initial state s2 with target action a2, and initial state s3 with target action a3, then a first exploration trajectory c1 can be obtained based on the task description information t, initial state s1, and target action a1; a first exploration trajectory c2 can be obtained based on the task description information t, initial state s2, and target action a2; and a first exploration trajectory c3 can be obtained based on the task description information t, initial state s3, and target action a3. Furthermore, for any input combination consisting of interrelated task description information, initial states, and target actions, multiple explorations can be performed using the interface navigation model to obtain multiple first exploration trajectories. For example, multiple first exploration trajectories can be obtained based on the task description information t, initial state s1, and target action a1.

[0025] In step S106, the interface navigation model is adjusted according to the first exploration trajectory based on multiple task description information to determine the interface navigation model after reinforcement learning. The interface navigation model after reinforcement learning is used to predict and execute task navigation actions within the graphical user interface.

[0026] As mentioned earlier, for each input combination consisting of relevant task description information, initial state and target action, one or more first exploration trajectories can be obtained, and one or more first exploration trajectories corresponding to each input combination can be combined into a training sample group.

[0027] In implementation, based on the relationship between task description information, initial state and target action, the initial state and target action associated with multiple task description information can be divided into different input combinations. A training sample group can be constructed based on one or more first exploration trajectories corresponding to each input combination. The interface navigation model can be trained based on multiple training sample groups, and the policy parameters of the model can be adjusted to obtain the interface navigation model after reinforcement learning.

[0028] In practical applications, training the interface navigation model based on the training sample set is an iterative process. A termination condition can be set for adjusting the model's policy, such as reaching a preset number of training epochs or the training effect converging. When this termination condition is met, the adjustment of the model's policy parameters stops, resulting in the final reinforcement-learned interface navigation model. The reinforced interface navigation model is used to predict and execute task navigation actions within the graphical user interface. Specific application details of the reinforced interface navigation model will be explained below and will not be elaborated upon here.

[0029] This specification provides a training method for an interface navigation model. It acquires multiple task descriptions and their associated initial states and target actions. Each task description and its associated initial state are input into the interface navigation model, and the target action associated with the initial state is used as a prompt. This guides the interface navigation model to generate a first exploration trajectory corresponding to the task descriptions. Then, based on the first exploration trajectories of the multiple task descriptions, the interface navigation model's strategy is adjusted to determine the interface navigation model after reinforcement learning. This specification designs a reinforcement learning strategy that combines action prompts with interpretability generation. Using real actions as external prompts, it guides the interface navigation model to generate self-explaining inference trajectories within a distribution while generating predicted actions. This not only maintains the in-distribution characteristics and avoids entropy increase and chaos, but also provides the model with effective non-zero reward learning signals, improving the model's strategy adjustment accuracy. This effectively optimizes the model's training effect and improves the implementation accuracy of the interface navigation model after reinforcement learning.

[0030] In the above or below embodiments, the strategy adjustment of the interface navigation model based on the first exploration trajectory of multiple task description information in step S106 to determine the implementation of the interface navigation model after reinforcement learning can be of various types. In one exemplary implementation, it may specifically include the following steps.

[0031] Step S1062: For any task description information, input the task description information and associated initial state into the interface navigation model, and generate the second exploration trajectory corresponding to the task description information through the interface navigation model.

[0032] The exploration trajectory is a complete interaction sequence between the interface navigation model and the environment. The second exploration trajectory describes the complete process of the interface navigation model performing a specified task without the guidance of a target action. It records how the interface navigation model attempts to complete the specified task under the guidance of a target action, what feedback it encounters, and how it adjusts its behavior to optimize the target. The second exploration trajectory may include the initial state, the action taken by the agent (i.e., the second predicted action), the reward from environmental feedback, the trajectory length, etc.

[0033] In implementation, task description information and its associated initial state can be input into the interface navigation model. The model then uses the initial state as its starting point to explore interactive actions that match the task description information within the exploration space. Based on the exploration results, it outputs a second predicted action and a second exploration trajectory. The second predicted action is the task navigation action predicted by the interface navigation model for a specified task without the guidance of a target action. Furthermore, for any input combination consisting of task description information and its associated initial state, the interface navigation model can perform one or more target-free action predictions to obtain one or more second exploration trajectories.

[0034] Step S1064: Adjust the strategy of the interface navigation model based on the first exploration trajectory and the second exploration trajectory based on multiple task description information, so as to determine the interface navigation model after reinforcement learning.

[0035] In implementation, for the input combinations corresponding to the first exploration trajectory and the input combinations corresponding to the second exploration trajectory, a training sample group can be formed by combining the first exploration trajectory and the second exploration trajectory, based on the association between the same task description information and the initial state. Furthermore, the target actions corresponding to different first exploration trajectories within the same training sample group are the same. The number of first and second exploration trajectories in each training sample group can be flexibly set; for example, each training sample group can contain one first exploration trajectory and N-1 second exploration trajectories. Then, based on multiple training sample groups constructed from first and second exploration trajectories with multiple task description information, the interface navigation model can be trained, and the model's policy parameters can be adjusted to obtain the reinforcement learning-adjusted interface navigation model.

[0036] In practical applications, training the interface navigation model based on the training sample set is an iterative process. A termination condition can be set for adjusting the model's policy. When this termination condition is met, the adjustment of the model's policy parameters stops, and the final reinforcement learning interface navigation model is obtained.

[0037] In this embodiment, the first and second exploration trajectories obtained based on the same task information and initial state are used as a training sample group. The strategy is adjusted based on the first and second exploration trajectories contained in each training sample group. This hybrid sampling strategy can effectively improve the zero advantage problem. When the second exploration trajectory cannot find the correct action, the first exploration trajectory obtained based on the self-explanation mechanism can provide a constructive positive signal, ensuring that there is a non-zero reward in the training sample group to maintain effective advantage value calculation, thereby enabling effective strategy adjustment of the model.

[0038] In the above or following embodiments, the implementation of adjusting the interface navigation model based on the first and second exploration trajectories of multiple task description information in step S1064 can be varied. An exemplary implementation is provided below, such as... Figure 2 As shown, this implementation method may specifically include the following steps.

[0039] Step S202: Based on a preset reward function, determine the reward value for each exploration trajectory, where the exploration trajectory is either any first exploration trajectory or any second exploration trajectory.

[0040] The reward function is a core component of reinforcement learning. It defines the immediate feedback or cumulative reward that a user interface navigation model receives for taking a certain action in a specific state. In simpler terms, the reward function can be understood as a scoring rule for each predicted action output by the user interface navigation model. The immediate reward refers to the score received immediately after performing the action, while the cumulative reward is the total score after completing the task.

[0041] In implementation, a reward function can be predefined to guide the optimization strategy of the interface navigation model. Based on the reward function, the predicted actions contained in each exploration trajectory are scored to determine the reward value of the exploration trajectory. The reward value can be used to characterize the suitability of the exploration trajectory for achieving the specified task. That is, the reward value can reflect the suitability of the predicted actions contained in the exploration trajectory for achieving the specified task.

[0042] Step S204: Determine the target advantage value of the exploration trajectory based on the reward value of the exploration trajectory.

[0043] The target dominance value describes the superiority of the action matched with the confidence level of the exploration trajectory. Action superiority refers to the degree to which the exploration trajectory is better or worse than the average expectation, which refers to the average performance of all exploration trajectories corresponding to the training sample group of the exploration trajectory. Compared with the traditional dominance value, the target dominance value takes into account the confidence level of the predicted action itself. Based on the target dominance value, the learning weights can be dynamically adjusted, giving more learning weights to the training sample group with higher confidence, thereby improving sample utilization efficiency.

[0044] In practice, the confidence level of each exploration trajectory in its corresponding training sample group can be calculated. The confidence level can be used to characterize the evaluation value of the interface navigation model on the reliability or correctness of the current exploration trajectory. Then, based on the reward value and confidence level of the exploration trajectory, the target advantage value of the exploration trajectory can be determined.

[0045] Step S206: Construct a strategy optimization function based on the target advantage value of the exploration trajectory, and use the strategy optimization function to adjust the strategy of the interface navigation model.

[0046] The purpose of the strategy optimization function is to improve the performance of the interface navigation model by adjusting the strategy parameters of the interface navigation model.

[0047] In implementation, the target advantage value can be used to quantify, considering confidence levels, how well the current predicted action performs compared to the average expectation of its corresponding training sample group, and to guide the adjustment direction of policy parameters based on the quantification results. A policy optimization function can be constructed based on the policy adjustment direction guided by the target advantage value, and this function can be used to adjust the policy of the interface navigation model.

[0048] It should be noted that the process of adjusting the strategy of the interface navigation model according to steps S202 to S206 can also be applied to the process of adjusting the strategy of the interface navigation model based on the first exploration trajectory of multiple task description information in step S106. For specific implementation details, please refer to the relevant description above. This implementation will not be repeated here.

[0049] Based on this, the processing method for determining the reward value of each exploration trajectory based on the preset reward function in step S202 above can be varied. The following provides an optional processing method, which may specifically include the following steps.

[0050] Step S2022: Determine the format reward for the exploration trajectory. The format reward is used to describe the compliance of the format of the output data of the interface navigation model.

[0051] It should be understood that the format reward is to verify whether the output of the interface navigation model contains the correct structured format.

[0052] In implementation,<think>Tags and<tool_call> Tags are used to structurally describe the decision-making process of intelligent agents (such as reinforcement learning models, large language models, or autonomous systems), and typically appear in agent interaction logs, debugging tools, or interpretability frameworks. They can be based on... <think>Tags and<tool_call> Labels verify the compliance of the model's output format, for example, when <think>The tags contain reasoning content, and<tool_call> If the label contains a predicted action, the format reward for determining the exploration trajectory is 1; otherwise, the format reward for determining the exploration trajectory is 0.

[0053] Step S2024: Determine the action type reward for the exploration trajectory. The action type reward is used to describe the degree of matching between the type of the predicted action and the type of the target action.

[0054] The types of predicted actions and target actions refer to the types of operations performed on the graphical user interface. Operation types can include click operations, long press operations, swipe operations, text input operations, etc.

[0055] In practice, the types of the predicted action and the target action can be obtained. It is then determined whether the types of the predicted action and the target action are the same. If they are the same, the action type reward for the exploration trajectory is determined to be 1; otherwise, the action type reward for the exploration trajectory is determined to be 0.

[0056] Step S2026: Determine the action parameter reward for the exploration trajectory. The action parameter reward is used to describe the similarity between the action parameters of the predicted action and the action parameters of the target action.

[0057] The trajectory's motion parameters are specific parameters used to control the execution of the action. These parameters are closely related to the type of action (i.e., the type of action) of the exploration trajectory. For example, when the action type of the exploration trajectory is a click operation, the trajectory's motion parameters can be action coordinates; when the action type of the exploration trajectory is a swipe operation, the trajectory's motion parameters can be the swipe direction. These are merely examples, and this embodiment will not exhaustively list them.

[0058] In implementation, the similarity between the motion parameters of the predicted action and the motion parameters of the target action can be calculated. If the similarity is higher than a pre-set threshold, the motion parameter reward for the exploration trajectory is set to 1; otherwise, the reward is set to 0. In practical applications, different similarity calculation methods can be used depending on the type of action in the exploration trajectory. For example, when the action type of the exploration trajectory is a click operation, the Euclidean distance between the predicted coordinates and the actual coordinates can be calculated as the similarity; when the action type of the exploration trajectory is a swipe operation, the matching degree between the predicted direction and the actual direction is calculated as the similarity; when the action type of the exploration trajectory is a text input operation, the word-level F1 score between the predicted text and the actual text is calculated as the similarity.

[0059] Step S2028: The reward value of the exploration trajectory is obtained by weighting the format reward, action type reward and action parameter reward based on the exploration trajectory.

[0060] In implementation, weights can be assigned to the format reward, action type reward, and action parameter reward respectively. The reward value of the exploration trajectory is obtained by weighted summing of these three rewards. Alternatively, a first weight can be assigned to the format reward to control the importance of correct formatting, and a second weight can be assigned to the sum of the action type reward and action parameter reward to control the importance of accurate actions. The reward value of the exploration trajectory is then obtained by weighted summing of the format reward, action type reward, and action parameter reward based on the first and second weights.

[0061] For example, the formula for calculating the reward value of an exploration trajectory could be: ,in, As the first weight, As the second weight, Indicates a formatted reward. Indicates the reward for the action type. Indicates the reward for the action parameters. The reward value represents the exploration trajectory. This reward value is obtained by weighting the format reward, action type reward, and action parameter reward. This reward value comprehensively reflects the structural correctness and action accuracy of the model's prediction results, improving the accuracy of policy adjustments based on the reward value and thus effectively enhancing the model's training performance.

[0062] In practical applications, the processing method for determining the target advantage value of the exploration trajectory based on the reward value of the exploration trajectory in step S204 above can be varied. The following provides one optional processing method, such as... Figure 3 As shown, this processing method may specifically include the following steps.

[0063] Step S302: Calculate the original advantage value of the exploration trajectory based on the reward value of the exploration trajectory.

[0064] The original advantage value is used to characterize the degree of superiority or inferiority of the exploration trajectory compared to the average expectation, which refers to the average performance of all exploration trajectories corresponding to the training sample group of the exploration trajectory.

[0065] In implementation, the reward values ​​of all exploration trajectories in a training sample group can be used to normalize the calculation for each exploration trajectory, and the calculated normalized value can be determined as the original advantage value of that exploration trajectory. By determining the original advantage value of the exploration trajectory through normalization calculation, better prediction results within the training sample group can be strengthened, while poor prediction results can be suppressed. At the same time, it can prevent samples with high absolute rewards but not outstanding relative rewards from being over-strengthened.

[0066] For example, the formula for calculating the initial advantage value of an exploration trajectory could be:

[0067] Where N represents the number of exploration trajectories in the current training sample group, and the N exploration trajectories can simultaneously include the first exploration trajectory and the second exploration trajectory, and i and j represent the numbers of the exploration trajectories, respectively. This represents the reward value for the i-th exploration trajectory. This represents the reward value for the j-th exploration trajectory. It is the mean function. This represents the average reward value of N exploration paths. It is the standard deviation function. This represents the standard deviation of the reward values ​​for N exploration trajectories. This represents the initial advantage value of the i-th exploration trajectory.

[0068] Step S304: Determine the confidence level of multiple exploration trajectories corresponding to the task description information, and determine the modulation factor of each exploration trajectory based on the multiple confidence levels.

[0069] Among them, the confidence level of the exploration trajectory can be used to characterize the evaluation value of the reliability or correctness of the interface navigation model for the exploration trajectory, and the modulation factor corresponding to the exploration trajectory is used to adjust the original advantage value of the exploration trajectory to a target advantage value that matches the confidence level of the exploration trajectory.

[0070] As mentioned earlier, input combinations can be formed based on task description information and initial state. Each training sample group can contain multiple exploration trajectories obtained based on the same input combination. These exploration trajectories can include a first exploration trajectory and / or a second exploration trajectory. Furthermore, when a training sample group contains multiple first exploration trajectories, the multiple first exploration trajectories correspond to the same target action.

[0071] In practice, the confidence level of each exploration trajectory can be calculated. Based on the confidence levels of multiple exploration trajectories contained in the same training sample group, the modulation factor corresponding to each training sample group is determined. The modulation factor corresponding to the training sample group is then determined as the modulation factor corresponding to the exploration trajectory contained in that training sample group. In other words, the modulation factors corresponding to multiple exploration trajectories in the same training sample group are the same.

[0072] Step S306: Based on the original advantage value and modulation factor of the exploration trajectory, determine the target advantage value of the exploration trajectory.

[0073] Among them, the target advantage value of the exploration trajectory is used to describe the action superiority that matches the confidence level of the exploration trajectory. The action superiority is used to characterize the degree of superiority or inferiority of the exploration trajectory compared to the average performance of all exploration trajectories corresponding to its training sample group.

[0074] In implementation, the original dominance value of the exploration trajectory can be adjusted based on the modulation factor. The core of the adjustment is to map the effect of the modulation factor onto the original dominance value through mathematical operations. This mapping can be achieved through multiplicative adjustment, additive adjustment, exponential adjustment, logarithmic adjustment, etc. By adjusting the original dominance value of the exploration trajectory, the adjusted result is determined as the target dominance value that matches the confidence level of the exploration trajectory.

[0075] For example, a formula for calculating the target advantage value for determining an exploration trajectory could be: .in, This represents the initial advantage value of the exploration trajectory. The modulation factor, representing task description information, can be used to multiply the original advantage value of the exploration trajectory with the modulation factor, and the resulting product is determined as the target advantage value of the exploration trajectory. The target advantage value is used to drive differentiated policy gradient updates.

[0076] Based on this, the process of determining the confidence levels of multiple exploration trajectories corresponding to the task description information in step S304, and determining the modulation factor of each exploration trajectory based on multiple confidence levels, can take many forms. The following provides one optional implementation method, such as... Figure 4 As shown, this implementation method may specifically include the following steps.

[0077] Step S402: Calculate the predicted entropy of multiple action words in the exploration trajectory, and determine the average entropy value of the predicted entropy of multiple action words as the confidence level of the exploration trajectory.

[0078] In implementation, the action sequence contained in the exploration trajectory can be decomposed into multiple action terms. Each action term represents the predicted action output by the interface navigation model for a certain interface state during task execution; that is, each action term corresponds to an interface state. Based on the action probability distribution output by the interface navigation model for the interface state corresponding to the action term, the prediction entropy of the corresponding action term can be calculated. The larger the prediction entropy, the more uncertain the model's choice of that action, and the more uniform the probability distribution; the smaller the prediction entropy, the more confident the model is in predicting that action, and the sharper the probability distribution. Then, the average entropy value of the prediction entropies of multiple action terms in the action sequence can be calculated, and the calculated average entropy value is determined as the confidence level of the exploration trajectory.

[0079] Step S404: Determine other exploration trajectories that have the same task description information and initial state as the exploration trajectory, and perform normalization processing based on the confidence of the exploration trajectory and the confidence of other exploration trajectories to obtain the normalized entropy value of the exploration trajectory and other exploration trajectories.

[0080] As mentioned earlier, multiple exploration trajectories in the same training sample group correspond to the same task description information and initial state. Therefore, determining other exploration trajectories that correspond to the same task description information and initial state as the exploration trajectory can be understood as determining other exploration trajectories that correspond to the same training sample group as the exploration trajectory.

[0081] In implementation, based on the task description information and initial state corresponding to the exploration trajectory, exploration trajectories with the same task description information and initial state can be identified as other exploration trajectories associated with the exploration trajectory. A training sample group can be constructed based on the exploration trajectory and other exploration trajectories. The confidence scores of multiple exploration trajectories in this training sample group are normalized to obtain the normalized entropy value of the training sample group. This normalized entropy value is then used as the normalized entropy value for the exploration trajectory and other exploration trajectories. The normalized entropy value of the training sample group can intuitively reflect the spatial or state-based dispersion of multiple exploration trajectory samples within the training sample group. A larger normalized entropy value indicates a more dispersed sample distribution and greater differences between the various exploration trajectories; a smaller normalized entropy value indicates a more concentrated sample distribution and smaller differences between the various exploration trajectories.

[0082] Step S406: Based on the normalized entropy value, calculate the modulation factor of the exploration trajectory using an exponential decay function.

[0083] The decay calculation function is a mathematical function whose output value decreases rapidly according to an exponential law as the input variable (usually time or spatial distance) increases, eventually approaching zero (but not equal to zero).

[0084] In practice, the formula for calculating the modulation factor of the exploration trajectory using the exponential decay function can be: .in, This represents the expected / mean value of all exploration trajectories in the training sample group. This represents an exponential function with the natural constant e as the base. This is the normalized entropy value.

[0085] The mechanism of this formula is as follows: for exploration trajectories with lower normalized entropy (higher confidence), the modulation factor is larger, and the learning weight is higher; conversely, for exploration trajectories with higher normalized entropy (greater uncertainty), the modulation factor is smaller, and the learning weight is lower. The denominator in the formula serves as a normalization function, ensuring that the expected value of the modulation weight within the training sample group is 1, without changing the magnitude of the overall update amplitude. Furthermore, the exponential decay function provides a stronger amplification effect for extremely low entropy samples and a stronger decay effect for extremely high entropy samples, which better meets the needs of sensitive responses to confidence differences and can more accurately achieve the target advantage value of dynamically adjusting the exploration trajectory based on confidence differences.

[0086] Based on this, the normalization process for the confidence of the exploration trajectory and the confidence of other exploration trajectories in step S404 to obtain the normalized entropy value of the exploration trajectory and other exploration trajectories can be varied. The following provides an optional implementation method, which may include the following steps.

[0087] Step S4042: Calculate the average confidence level of multiple confidence levels based on the confidence level of the exploration trajectory and the confidence levels of other exploration trajectories; Step S4044: Based on the confidence of the exploration trajectory and the confidence of other exploration trajectories, determine the maximum confidence and minimum confidence among multiple confidence levels; Step S4046: Based on the average confidence level, maximum confidence level, and minimum confidence level, determine the normalized entropy value of the exploration trajectory and other exploration trajectories.

[0088] In implementation, the confidence levels of multiple exploration trajectories can be ranked based on their confidence levels and those of other exploration trajectories. The maximum and minimum confidence levels among these trajectories are then determined based on the ranking results. Furthermore, the average confidence level of the multiple exploration trajectories can be calculated by averaging their confidence levels. Finally, based on the maximum, minimum, and average confidence levels, the normalized entropy value between the exploration trajectory and other exploration trajectories can be determined. The normalized entropy value typically ranges from [0, 1].

[0089] For example, the normalized entropy value can be calculated as follows: .in, Represents the normalized entropy value. This represents the average confidence level. Indicates the maximum confidence level. This represents the minimum confidence level. By performing min-max normalization on the average entropy values ​​of multiple exploration trajectories, the scale differences in absolute entropy values ​​caused by different training phases can be eliminated.

[0090] In this embodiment, a dynamic weight adjustment mechanism based on prediction entropy and reward signal is proposed. Prediction entropy is used as a natural proxy for confidence. A modulation factor is calculated based on the prediction entropy, and the original advantage is modulated using the modulation factor to achieve refined confidence allocation: for high-confidence and correct predictions (low entropy and high reward), amplified learning weights are given to encourage the model to strengthen this confident correct behavior; for overconfident but incorrect predictions (low entropy and low reward), strong penalties are applied to force the model to correct the erroneous cognition; for uncertain exploratory behavior (high entropy), regardless of the reward level, attenuation processing is performed to prevent noise gradients from interfering with the learning process.

[0091] In the above or following embodiments, the process of constructing a strategy optimization function based on the target advantage value of the exploration trajectory in step S206, and using the strategy optimization function to adjust the strategy of the interface navigation model, can be varied. One optional implementation is provided below, such as... Figure 5 As shown, this implementation method may specifically include the following steps.

[0092] Step S502: Determine the importance sampling ratio of the interface navigation model.

[0093] The importance sampling ratio measures the difference in probability between using a new strategy and using an old strategy for the same state-action pair in the interface navigation model. The formula for calculating the importance sampling ratio is:

[0094] in, This represents the importance sampling ratio of the t-th training sample group. The strategy parameter is Interface navigation model, The interface navigation model representing the old strategy. This represents the initial state in the t-th training sample group. This represents the predicted action inferred based on the initial state in the t-th training sample group.

[0095] Step S504: Construct a strategy optimization function based on the importance sampling ratio and the target advantage value of the exploration trajectory.

[0096] The core idea behind constructing the policy optimization function is to correct the policy update direction by using the importance sampling ratio and to measure the relative merit of actions by using the target advantage value, thereby maximizing long-term cumulative rewards while ensuring policy stability.

[0097] An exemplary method for constructing a policy optimization function is as follows: Based on an existing GRPO (Group Relative Policy Optimization) policy optimization function, the original advantage value is replaced with the target advantage value to construct the policy optimization function in this embodiment. The formula of the constructed policy optimization function is as follows:

[0098] in, Indicates the strategy The expected value of the rewards generated from interacting with the environment. It is a minimum value function. This represents the target advantage value at step t. This is a truncation function to prevent overly aggressive single-step updates. Indicates the truncation parameter. This represents the KL divergence regularization hyperparameter. This represents the KL divergence. The KL divergence term helps prevent the policy from deviating too far from the reference policy. This is a reference strategy, typically the initial parameter version of the strategy.

[0099] Of course, other methods can be used to construct the policy optimization function, as long as it can achieve policy optimization based on the target advantage value. This embodiment does not impose any further limitations on this.

[0100] Step S506: With the goal of increasing the strategy optimization function, the strategy of the interface navigation model is adjusted through backpropagation.

[0101] The goal of policy adjustment is to find a new policy that tends to select actions with higher advantage values ​​while avoiding excessive differences between the old and new policies. A larger policy optimization function means the model can perform the task better.

[0102] In implementation, the goal is to increase the policy optimization function. The gradient of the policy parameters is calculated through backpropagation, and the policy is optimized via gradient ascent (or descent). Multiple iterations of updating the model's policy parameters are performed until the parameters converge, thus completing the reinforcement learning of the interface navigation model.

[0103] In the above or following embodiments, the interface navigation model after reinforcement learning can be used to predict and execute task navigation actions within the graphical user interface. For example... Figure 6 As shown, a process for automatically performing tasks within a graphical user interface based on a reinforcement learning-based interface navigation model may include the following steps.

[0104] Step S602: Obtain the description information of the target task to be processed.

[0105] The target task description information describes the target task that the user expects to complete within the graphical user interface. The target task can be a related operation performed on a certain interface element on the target graphical user interface. The interface navigation model of the target graphical user interface after reinforcement learning has access to any graphical user interface.

[0106] Step S604: Determine the target interface screenshot associated with the target task description information, and determine the target interface screenshot as the target initial state for executing the target task.

[0107] The initial state of the target refers to the starting point for executing the target task. The target interface screenshot can be a screenshot of the graphical user interface where the interface navigation model after reinforcement learning is currently located, or a screenshot of the graphical user interface specified by the user as the starting point for the operation.

[0108] Step S606: Input the target task description information and the target initial state into the interface navigation model after reinforcement learning, so as to obtain the target prediction action for the target initial state through the interface navigation model after reinforcement learning.

[0109] The reinforcement learning-based interface navigation model refers to the reinforcement learning-based interface navigation model obtained through training as described above. The training process of the reinforcement learning-based interface navigation model can be found in the relevant description above, and will not be repeated here.

[0110] In practice, the target task description information and the target initial state are input into the reinforcement learning interface navigation model. The reinforcement learning interface navigation model autonomously generates an exploration trajectory containing reasoning process and predicted actions without any real action prompts. The exploration trajectory generated by the model is analyzed, and the predicted actions are extracted and executed.

[0111] Afterwards, the execution results can be observed. If the target task is not completed, the updated screenshot of the interface is determined as the new initial state of the target, and step S606 is repeated based on the target task description information until the target task is completed or the number of predictions by the model reaches the preset prediction limit.

[0112] In this embodiment, an interface navigation model obtained after reinforcement learning based on self-explanatory trajectory and entropy modulation mechanism is used to automatically execute tasks within the graphical user interface. This can effectively solve the zero-odds trap and sample quality heterogeneity problems in GUI automated reinforcement learning, significantly improve the performance of the interface navigation model, and enhance the accuracy of task execution within the interface.

[0113] like Figure 7a and Figure 7b As shown, this specification provides a detailed explanation of a training method for an interface navigation model, tailored to a specific application scenario. This method may specifically include the following steps: Step S702: Determine a GUI task and obtain the task description information, starting interface screenshot, and actual actions to implement the GUI task. Step S704: Input the task description information and the screenshot of the starting interface into the interface navigation model, and use real actions as prompts for the model to generate a first exploration trajectory containing self-explanatory content. Step S706: Input the task description information and the screenshot of the starting interface into the interface navigation model, and obtain N-1 second exploration trajectories by sampling with N-1 normal strategies. Step S708: Combine one first exploration trajectory and N-1 second exploration trajectories into a training sample group, and calculate the reward value for each exploration trajectory in the training sample group; wherein, the reward value is obtained by weighted summation of format reward, action type reward and action parameter reward; Step S710: Based on the reward value of the exploration trajectory, calculate the original advantage value of each exploration trajectory in the training sample group; Step S712: Calculate the predicted entropy of multiple action words in each exploration trajectory, and determine the average entropy value of the predicted entropy of multiple action words as the confidence level of the exploration trajectory. Step S714: Based on the confidence level of each exploration trajectory in the training sample group, calculate the normalized entropy value of the training sample group, and based on the normalized entropy value, calculate the modulation factor of the training sample group through the exponential decay function. Step S716: Multiply the original advantage value of each exploration trajectory by the modulation factor to obtain the target advantage value of each exploration trajectory; Step S718: Based on the importance sampling ratio of the interface navigation model and the target advantage value of the exploration trajectory, construct a strategy optimization function, and adjust the strategy of the interface navigation model through backpropagation with the goal of making the strategy optimization function tend to increase. Step S720: Determine whether the preset number of training rounds has been reached or whether the model's policy parameters have converged. If any of the conditions in step S720 is met, then step S722 is executed to end the training and obtain the interface navigation model after reinforcement learning; otherwise, steps S702 to S718 are repeated until the preset number of training rounds or the policy parameters of the model converge.

[0114] This specification provides a training method for an interface navigation model. By using real actions as prompts, the method guides the policy model to generate self-explanatory inference trajectories within a distribution, thereby providing the model with effective non-zero reward learning signals. Based on the word-by-word prediction entropy of each exploration trajectory, the method calculates the normalized entropy value and modulation factor, and performs entropy modulation credit allocation on the advantage value of each exploration trajectory. This dynamically recalibrates the learning weights based on the predicted confidence, amplifies and updates correct actions with high confidence, and decays and updates uncertain explorations. Based on the modulated advantage value, the model's policy parameters are optimized. This method can effectively solve the zero-dominance trap and sample quality heterogeneity problems in reinforcement learning, optimize the reinforcement learning effect of the model, and thus significantly improve the performance of the interface navigation model.

[0115] The above describes the training method for the interface navigation model provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a training device for the interface navigation model, such as... Figure 8 As shown.

[0116] The training device for the interface navigation model includes: a trajectory generation module 801 and a policy adjustment module 802, wherein: The trajectory generation module 801 is used to acquire multiple task description information, as well as the initial state and target action associated with each task description information. The task description information is used to describe a specified task that the user expects to complete within the graphical user interface. For any task description information, the task description information and the associated initial state are input into the interface navigation model, and the target action associated with the initial state is used as a prompt condition to guide the interface navigation model to generate a first exploration trajectory corresponding to the task description information. The first exploration trajectory is used to describe the complete process of the interface navigation model executing the specified task under the guidance of the target action. The strategy adjustment module 802 is used to adjust the interface navigation model based on the first exploration trajectory of the multiple task description information to determine the interface navigation model after reinforcement learning. The interface navigation model after reinforcement learning is used to predict and execute task navigation actions within the graphical user interface.

[0117] In this embodiment of the specification, the trajectory generation module 801 includes: The second trajectory generation unit, for any task description information, inputs the task description information and associated initial state into the interface navigation model, and generates a second exploration trajectory corresponding to the task description information through the interface navigation model. The second exploration trajectory is used to describe the complete process of the interface navigation model executing the specified task without the guidance of the target action.

[0118] In this embodiment of the specification, the trajectory generation module 802 includes: Based on the first and second exploration trajectories of the multiple task description information, the interface navigation model is adjusted in terms of strategy to determine the interface navigation model after reinforcement learning.

[0119] In the embodiments of this specification, the training device for the interface navigation model further includes a reward calculation module, which includes: Based on a preset reward function, a reward value is determined for each exploration trajectory. The reward value is used to characterize the adaptability of the exploration trajectory to achieving the specified task. The exploration trajectory can be any first exploration trajectory or any second exploration trajectory.

[0120] In the embodiments of this specification, the training device for the interface navigation model further includes an advantage value modulation module, which includes: Based on the reward value of the exploration trajectory, a target advantage value is determined for the exploration trajectory. The target advantage value is used to describe the superiority of the action that matches the confidence level of the exploration trajectory. The address superiority is used to characterize the degree of superiority or inferiority of the exploration trajectory compared to the average expectation.

[0121] In this embodiment of the specification, the strategy adjustment module 802 includes: A strategy optimization function is constructed based on the target advantage value of the exploration trajectory, and the strategy optimization function is used to adjust the interface navigation model.

[0122] In the embodiments of this specification, the reward calculation module includes: The format reward for the exploration trajectory is determined, wherein the format reward is used to describe the compliance of the format of the output data of the interface navigation model; The action type reward for determining the exploration trajectory is used to describe the degree of matching between the type of the predicted action and the type of the target action; The action parameter reward for determining the exploration trajectory is used to describe the similarity between the action parameters of the predicted action and the action parameters of the target action; The reward value for each exploration trajectory is obtained by weighting the format reward, action type reward, and action parameter reward based on the exploration trajectory.

[0123] In the embodiments described in this specification, the dominance value modulation module includes: Based on the reward value of the exploration trajectory, the original advantage value of the exploration trajectory is calculated, which is used to characterize the degree of superiority or inferiority of the exploration trajectory compared to the average expectation; Determine the confidence level of multiple exploration trajectories corresponding to the task description information, and determine the modulation factor of each exploration trajectory based on the multiple confidence levels; Based on the original advantage value of the exploration trajectory and the modulation factor, the target advantage value of the exploration trajectory is determined.

[0124] In the embodiments of this specification, the dominance value modulation module includes: Calculate the prediction entropy of multiple action words in the exploration trajectory, and determine the average entropy value of the prediction entropy of the multiple action words as the confidence level of the exploration trajectory; Identify other exploration trajectories that correspond to the same task description information and initial state as the exploration trajectory, and perform normalization processing based on the confidence of the exploration trajectory and the confidence of the other exploration trajectories to obtain the normalized entropy value of the exploration trajectory and the other exploration trajectories; Based on the normalized entropy value, the modulation factor of the exploration trajectory is calculated using an exponential decay function.

[0125] In the embodiments of this specification, the dominance value modulation module includes: The confidence scores of the exploration trajectory and the other exploration trajectories are normalized to obtain the normalized entropy values ​​of the exploration trajectory and the other exploration trajectories, and the average confidence score of multiple confidence scores is calculated. Based on the confidence scores of the exploration trajectory and the other exploration trajectories, normalization processing is performed to obtain the normalized entropy values ​​of the exploration trajectory and the other exploration trajectories, and the maximum and minimum confidence scores among multiple confidence scores are determined. Based on the average confidence level, the maximum confidence level, and the minimum confidence level, the normalized entropy values ​​of the exploration trajectory and the other exploration trajectories are determined.

[0126] In this embodiment of the specification, the strategy adjustment module 802 includes: Determine the importance sampling ratio of the interface navigation model, which is used to measure the change in the probability of the interface navigation model outputting predicted actions when adopting a new strategy compared to when adopting an old strategy; Based on the importance sampling ratio and the target advantage value of the exploration trajectory, a strategy optimization function is constructed; With the goal of increasing the optimization function of the strategy, the strategy of the interface navigation model is adjusted through backpropagation.

[0127] In the embodiments of this specification, the training device for the interface navigation model further includes a testing module, which includes: Obtain the target task description information to be processed, which describes the target task that the user expects to complete within the interface; Identify the target interface associated with the target task description information, and determine the target interface as the target initial state for executing the target task; The target task description information and the target initial state are input into the reinforcement learning-based interface navigation model to obtain the target predicted action for the target initial state through the reinforcement learning-based interface navigation model.

[0128] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more embodiments of this specification, the functions of each module or unit can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative; the division of each module and unit is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or modules can be combined or integrated into another system, or some features can be ignored or not executed, etc.

[0129] This specification provides a training device for an interface navigation model. It acquires multiple task descriptions and their associated initial states and target actions. Each task description and its associated initial state are input into the interface navigation model, and the target action associated with the initial state is used as a prompt. This guides the interface navigation model to generate a first exploration trajectory corresponding to the task descriptions. Then, based on the first exploration trajectories of the multiple task descriptions, the interface navigation model's strategy is adjusted to determine the interface navigation model after reinforcement learning. It is evident that a reinforcement learning strategy combining action prompts and interpretability generation is designed. Utilizing real actions as external prompts, it guides the interface navigation model to generate self-explanatory inference trajectories within a distribution while generating predicted actions. This not only maintains the in-distribution characteristics and avoids entropy-increasing chaos but also provides the model with effective non-zero reward learning signals, improving the model's strategy adjustment accuracy. This effectively optimizes the model's training effect and enhances the implementation accuracy of the interface navigation model after reinforcement learning.

[0130] The above describes the training apparatus for the interface navigation model provided in the embodiments of this specification. Based on the same idea, the embodiments of this specification also provide a training device for the interface navigation model, such as... Figure 9 As shown.

[0131] The training device for the interface navigation model can be a terminal device or server, as described in the above embodiments.

[0132] The training device for the interface navigation model can vary significantly due to differences in configuration or performance. It may comprise a communication interface 902, a user interface 904, a processor 906, and data storage 908. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 910. The communication interface 902 enables the training device 900 for the interface navigation model to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 902 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 902 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 902 may also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 902 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0133] User interface 904 includes receiving user input and providing output to the user. Therefore, user interface 904 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 904 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 904 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, training device 900 for the interface navigation model may support remote access from other devices via communication interface 902 or another physical interface (not shown). User interface 904 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 904 can also be configured as a display device for rendering or displaying text fragments.

[0134] Processor 906 may contain one or more general-purpose processors and / or special-purpose processors.

[0135] Data storage 908 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 906. Data storage 908 may include removable and non-removable components.

[0136] Processor 906 is capable of executing program instructions 918 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 908 to perform the various functions described herein. Data storage 908 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by training device 900 of the interface navigation model, enable training device 900 of the interface navigation model to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 918 by processor 906 may result in processor 906 using data 912.

[0137] For example, program instructions 918 may include an operating system 922 (e.g., an operating system kernel, device drivers, and / or other modules) and one or more applications 920 (e.g., a browser, social application, or game application) installed on the training device 900 of the interface navigation model. Similarly, data 912 may include operating system data 916 and application data 914. Operating system data 916 is primarily accessible to the operating system 922, while application data 914 is primarily accessible to one or more applications 920. Application data 914 may reside in a file system visible or hidden from the user on the training device 900 of the interface navigation model.

[0138] Application 920 can communicate with operating system 912 through one or more application programming interfaces (APIs). These APIs help application 920 read and / or write application data 914, transmit or receive information via communication interface 902, receive or display information on user interface 904, etc.

[0139] In some terminology, application 920 may be simply referred to as "app". Furthermore, application 920 can be downloaded to the training device 900 of the interface navigation model through one or more online app stores or app markets. However, the application can also be installed on the training device 900 of the interface navigation model in other ways, such as through a web browser or a physical interface (e.g., a USB port) on the training device 900 of the interface navigation model.

[0140] Specifically, in this embodiment, the training device 900 for the interface navigation model includes a data storage 908 and one or more program instructions 918, wherein one or more program instructions 918 are stored in the data storage 908, and one or more program instructions 918 are configured to be executed by one or more processors. The one or more program instructions include computer-executable instructions for performing the following: Acquire multiple task description information, as well as the initial state and target action associated with each task description information. The task description information is used to describe the specified task that the user expects to complete within the graphical user interface. For any task description information, the task description information and the associated initial state are input into the interface navigation model, and the target action associated with the initial state is used as a prompt condition to guide the interface navigation model to generate a first exploration trajectory corresponding to the task description information. The first exploration trajectory is used to describe the complete process of the interface navigation model executing the specified task under the guidance of the target action. The interface navigation model is adjusted based on the first exploration trajectory of the multiple task description information to determine the interface navigation model after reinforcement learning. The interface navigation model after reinforcement learning is used to predict and execute task navigation actions within the graphical user interface.

[0141] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the embodiment for the training device of the interface navigation model is relatively simple in description because it is fundamentally similar to the method embodiment; relevant parts can be referred to the descriptions in the method embodiment.

[0142] This specification provides a training device for an interface navigation model. It acquires multiple task descriptions and their associated initial states and target actions. Each task description and its associated initial state are input into the interface navigation model, and the target action associated with the initial state is used as a prompt. This guides the interface navigation model to generate a first exploration trajectory corresponding to the task descriptions. Then, based on the first exploration trajectories of the multiple task descriptions, the interface navigation model's strategy is adjusted to determine the interface navigation model after reinforcement learning. It is evident that a reinforcement learning strategy combining action prompts and interpretability generation is designed. Utilizing real actions as external prompts, it guides the interface navigation model to generate self-explaining inference trajectories within a distribution while generating predicted actions. This not only maintains the in-distribution characteristics and avoids entropy-increasing chaos but also provides the model with effective non-zero reward learning signals, improving the model's strategy adjustment accuracy. This effectively optimizes the model's training effect and enhances the implementation accuracy of the interface navigation model after reinforcement learning.

[0143] Furthermore, based on the above Figures 1 to 7b This specification also provides a storage medium for storing computer-executable instruction information in one or more embodiments. In one specific embodiment, the storage medium may be a USB flash drive, optical disc, hard disk, etc. When the computer-executable instruction information stored in the storage medium is executed by a processor, it can realize the following process: Acquire multiple task description information, as well as the initial state and target action associated with each task description information. The task description information is used to describe the specified task that the user expects to complete within the graphical user interface. For any task description information, the task description information and the associated initial state are input into the interface navigation model, and the target action associated with the initial state is used as a prompt condition to guide the interface navigation model to generate a first exploration trajectory corresponding to the task description information. The first exploration trajectory is used to describe the complete process of the interface navigation model executing the specified task under the guidance of the target action. The interface navigation model is adjusted based on the first exploration trajectory of the multiple task description information to determine the interface navigation model after reinforcement learning. The interface navigation model after reinforcement learning is used to predict and execute task navigation actions within the graphical user interface.

[0144] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described storage medium embodiment is basically similar to the method embodiment, so the description is relatively simple; relevant parts can be referred to the description of the method embodiment.

[0145] This specification provides an embodiment of a storage medium. This study acquires multiple task descriptions and their associated initial states and target actions. Each task description and its associated initial state are input into the interface navigation model, with the target action associated with the initial state serving as a cue. This guides the interface navigation model to generate a first exploration trajectory corresponding to the task description. Then, based on the first exploration trajectories of multiple task descriptions, the strategy of the interface navigation model is adjusted to determine the interface navigation model after reinforcement learning. It demonstrates that a reinforcement learning strategy combining action cues and interpretability generation is designed. Utilizing real actions as external cues, it guides the interface navigation model to generate self-explaining inference trajectories within a distribution while simultaneously generating predicted actions. This not only maintains the in-distribution characteristics and avoids entropy-increasing chaos but also provides the model with effective non-zero reward learning signals, improving the model's strategy adjustment accuracy. This effectively optimizes the model's training performance and enhances the implementation accuracy of the interface navigation model after reinforcement learning.

[0146] Furthermore, based on the above Figures 1 to 7b This specification also provides one or more embodiments of a computer program product, including a computer program, which, when executed by a processor, can perform the following processes: Acquire multiple task description information, as well as the initial state and target action associated with each task description information. The task description information is used to describe the specified task that the user expects to complete within the graphical user interface. For any task description information, the task description information and the associated initial state are input into the interface navigation model, and the target action associated with the initial state is used as a prompt condition to guide the interface navigation model to generate a first exploration trajectory corresponding to the task description information. The first exploration trajectory is used to describe the complete process of the interface navigation model executing the specified task under the guidance of the target action. The interface navigation model is adjusted based on the first exploration trajectory of the multiple task description information to determine the interface navigation model after reinforcement learning. The interface navigation model after reinforcement learning is used to predict and execute task navigation actions within the graphical user interface.

[0147] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the above-described embodiment of a computer program product is relatively simple in description because it is fundamentally similar to the method embodiment; relevant parts can be referred to the description of the method embodiment.

[0148] This specification provides a computer program product that acquires multiple task descriptions and their associated initial states and target actions. Each task description and its associated initial state are input into an interface navigation model, and the target action associated with the initial state is used as a prompt condition to guide the interface navigation model in generating a first exploration trajectory corresponding to the task descriptions. Then, based on the first exploration trajectories of the multiple task descriptions, the interface navigation model's strategy is adjusted to determine the interface navigation model after reinforcement learning. It is evident that a reinforcement learning strategy combining action prompts and interpretability generation is designed. Utilizing real actions as external prompts, it guides the interface navigation model to generate self-explaining inference trajectories within a distribution while generating predicted actions. This not only maintains the characteristics of the distribution and avoids entropy-increasing chaos, but also provides the model with effective non-zero reward learning signals, improving the model's strategy adjustment accuracy. This effectively optimizes the model's training effect and improves the implementation accuracy of the interface navigation model after reinforcement learning.

[0149] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous. Moreover, although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is merely one possible execution order among many steps and does not represent the only execution order. Therefore, when method steps are involved in the claims, adjustments to the order of those steps, or parallelism between steps, are also within the scope of protection of the claims.

[0150] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement to the methodology cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0151] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0152] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0153] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0154] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] Embodiments in this specification are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable parallel device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable parallel device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable fraud device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions can also be loaded onto a computer or other programmable device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0159] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0160] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0161] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one…" does not exclude the presence of other identical or equivalent elements in the process, method, article, or apparatus that includes said element. Furthermore, "a," "an," and "the" do not specifically refer to the singular and may also include the plural. Ordinal numbers such as "first," "second," etc., do not necessarily indicate order; often they are used to distinguish objects. For example, "first server" and "second server" usually refer to two servers, described as "first server" and "second server" to differentiate them; however, sometimes these two servers may be the same server. Moreover, in this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can be indirect receiving and sending (i.e., receiving and sending indirectly through one or more entities). Similarly, in this specification, unless otherwise stated, the relationships between structures can be direct or indirect.

[0162] Furthermore, the specific terms used in this specification to describe embodiments, such as "an embodiment," "one embodiment," or "some embodiments," refer to a particular feature, structure, or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. Moreover, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples, without contradiction.

[0163] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0164] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0165] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0166] The above description is merely an embodiment of this specification and is not intended to limit this document. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims in this document.< / think> < / think> < / think>

Claims

1. A method for training a user interface navigation model, the method comprising: Acquire multiple task description information, as well as the initial state and target action associated with each task description information. The task description information is used to describe the specified task that the user expects to complete within the graphical user interface. For any task description information, the task description information and the associated initial state are input into the interface navigation model, and the target action associated with the initial state is used as a prompt condition to guide the interface navigation model to generate a first exploration trajectory corresponding to the task description information. The first exploration trajectory is used to describe the complete process of the interface navigation model executing the specified task under the guidance of the target action. The interface navigation model is adjusted based on the first exploration trajectory of the multiple task description information to determine the interface navigation model after reinforcement learning. The interface navigation model after reinforcement learning is used to predict and execute task navigation actions within the graphical user interface.

2. The method according to claim 1, wherein adjusting the interface navigation model based on the first exploration trajectory of the plurality of task description information to determine the interface navigation model after reinforcement learning includes: For any task description information, the task description information and the associated initial state are input into the interface navigation model, and the interface navigation model generates a second exploration trajectory corresponding to the task description information. The second exploration trajectory is used to describe the complete process of the interface navigation model executing the specified task without the guidance of the target action. Based on the first and second exploration trajectories of the multiple task description information, the interface navigation model is adjusted in terms of strategy to determine the interface navigation model after reinforcement learning.

3. The method according to claim 2, wherein adjusting the interface navigation model based on the first exploration trajectory and the second exploration trajectory of the plurality of task description information includes: Based on a preset reward function, a reward value is determined for each exploration trajectory. The reward value is used to characterize the adaptability of the exploration trajectory to achieving the specified task. The exploration trajectory can be any first exploration trajectory or any second exploration trajectory. Based on the reward value of the exploration trajectory, a target advantage value is determined for the exploration trajectory. The target advantage value is used to describe the superiority of the action that matches the confidence level of the exploration trajectory. The address superiority is used to characterize the degree of superiority or inferiority of the exploration trajectory compared to the average expectation. A strategy optimization function is constructed based on the target advantage value of the exploration trajectory, and the strategy optimization function is used to adjust the interface navigation model.

4. The method according to claim 3, wherein determining the reward value for each exploration trajectory based on a preset reward function includes: The format reward for the exploration trajectory is determined, wherein the format reward is used to describe the compliance of the format of the output data of the interface navigation model; The action type reward for determining the exploration trajectory is used to describe the degree of matching between the type of the predicted action and the type of the target action; The action parameter reward for determining the exploration trajectory is used to describe the similarity between the action parameters of the predicted action and the action parameters of the target action; The reward value for each exploration trajectory is obtained by weighting the format reward, action type reward, and action parameter reward based on the exploration trajectory.

5. The method according to claim 3, wherein determining the target advantage value of the exploration trajectory based on the reward value of the exploration trajectory includes: Based on the reward value of the exploration trajectory, the original advantage value of the exploration trajectory is calculated, which is used to characterize the degree of superiority or inferiority of the exploration trajectory compared to the average expectation; Determine the confidence level of multiple exploration trajectories corresponding to the task description information, and determine the modulation factor of each exploration trajectory based on the multiple confidence levels; Based on the original advantage value of the exploration trajectory and the modulation factor, the target advantage value of the exploration trajectory is determined.

6. The method according to claim 5, wherein determining the confidence levels of multiple exploration trajectories corresponding to the task description information, and determining the modulation factor of each exploration trajectory based on the multiple confidence levels, comprises: Calculate the prediction entropy of multiple action words in the exploration trajectory, and determine the average entropy value of the prediction entropy of the multiple action words as the confidence level of the exploration trajectory; Identify other exploration trajectories that correspond to the same task description information and initial state as the exploration trajectory, and perform normalization processing based on the confidence of the exploration trajectory and the confidence of the other exploration trajectories to obtain the normalized entropy value of the exploration trajectory and the other exploration trajectories; Based on the normalized entropy value, the modulation factor of the exploration trajectory is calculated using an exponential decay function.

7. The method according to claim 6, wherein the normalization processing based on the confidence level of the exploration trajectory and the confidence levels of other exploration trajectories to obtain the normalized entropy value of the exploration trajectory and the other exploration trajectories includes: The confidence scores of the exploration trajectory and the other exploration trajectories are normalized to obtain the normalized entropy values ​​of the exploration trajectory and the other exploration trajectories, and the average confidence score of multiple confidence scores is calculated. Based on the confidence scores of the exploration trajectory and the other exploration trajectories, normalization processing is performed to obtain the normalized entropy values ​​of the exploration trajectory and the other exploration trajectories, and the maximum and minimum confidence scores among multiple confidence scores are determined. Based on the average confidence level, the maximum confidence level, and the minimum confidence level, the normalized entropy values ​​of the exploration trajectory and the other exploration trajectories are determined.

8. The method according to claim 3, wherein constructing a strategy optimization function based on the target advantage value of the exploration trajectory, and using the strategy optimization function to adjust the interface navigation model, comprises: Determine the importance sampling ratio of the interface navigation model, which is used to measure the change in the probability of the interface navigation model outputting predicted actions when adopting a new strategy compared to when adopting an old strategy; Based on the importance sampling ratio and the target advantage value of the exploration trajectory, a strategy optimization function is constructed; With the goal of increasing the optimization function of the strategy, the strategy of the interface navigation model is adjusted through backpropagation.

9. The method according to any one of claims 1 to 8, wherein the method further comprises: Obtain the target task description information to be processed, which describes the target task that the user expects to complete within the interface; Identify the target interface associated with the target task description information, and determine the target interface as the target initial state for executing the target task; The target task description information and the target initial state are input into the reinforcement learning-based interface navigation model to obtain the target predicted action for the target initial state through the reinforcement learning-based interface navigation model.

10. A training device for an interface navigation model, the device comprising: The trajectory generation module is used to acquire multiple task description information, as well as the initial state and target action associated with each task description information. The task description information is used to describe a specified task that the user expects to complete within the graphical user interface. For any task description information, the task description information and the associated initial state are input into the interface navigation model, and the target action associated with the initial state is used as a prompt condition to guide the interface navigation model to generate a first exploration trajectory corresponding to the task description information. The first exploration trajectory is used to describe the complete process of the interface navigation model executing the specified task under the guidance of the target action. The strategy adjustment module is used to adjust the interface navigation model based on the first exploration trajectory of the multiple task description information to determine the interface navigation model after reinforcement learning. The interface navigation model after reinforcement learning is used to predict and execute task navigation actions within the graphical user interface.

11. A training device for an interface navigation model, the device comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to: Acquire multiple task description information, as well as the initial state and target action associated with each task description information. The task description information is used to describe the specified task that the user expects to complete within the graphical user interface. For any task description information, the task description information and the associated initial state are input into the interface navigation model, and the target action associated with the initial state is used as a prompt condition to guide the interface navigation model to generate a first exploration trajectory corresponding to the task description information. The first exploration trajectory is used to describe the complete process of the interface navigation model executing the specified task under the guidance of the target action. The interface navigation model is adjusted based on the first exploration trajectory of the multiple task description information to determine the interface navigation model after reinforcement learning. The interface navigation model after reinforcement learning is used to predict and execute task navigation actions within the graphical user interface.

Images