Reinforcement learning training method, system construction method, system, equipment and product

Abstract

The embodiment of the invention relates to the technical field of information processing, and discloses a reinforcement learning training method based on graphical user interface simulation, a system construction method, a system, equipment and a product, and the method comprises the steps: modeling a graphical user interface navigation graph into a directed graph, enabling nodes of the directed graph to represent the interface state of a graphical user interface, and enabling the nodes of the directed graph to represent the interface state of the graphical user interface; the edges of the directed graph are represented as state transition between the graphical user interfaces, and a navigation graph structure is generated by controlling topological parameters of the directed graph; rendering a graphical user interface according to the navigation map structure; driving the intelligent agent to perform graphical user interface operation by using the task target, and obtaining an operation action of the intelligent agent; evaluating the operation action and the task completion degree of the intelligent agent; and calculating a reward through the operation action and the task completion degree of the intelligent agent, and feeding back the reward to the intelligent agent. And an ideal experimental environment is provided for a methodology of constructing an intelligent agent based on a graphical user interface from zero.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information software, and in particular to a reinforcement learning training method based on graphical user interface simulation, a system construction method, a system, a device and a product. BACKGROUND

[0002] With the rapid development of artificial intelligence and machine learning technology, graphical user interface (GUI) automation has gradually become a research hotspot. Researchers hope to build intelligent agents (Agents) that can autonomously complete complex operation tasks in GUI environments, thereby promoting the development of software automation testing and intelligent interaction.

[0003] Currently, common GUI simulation and training environments include Android emulators, Ubuntu virtual machines, etc. These virtual environments simulate real operating systems to provide researchers with relatively realistic interaction scenarios and are widely used in software development, testing and intelligent agent training. However, the above technologies still have obvious deficiencies in the training process of intelligent agents:

[0004] Existing virtual environments cannot provide page jump navigation maps, lack intuitive representation and system management of interface jump relationships, and limit the depth analysis and optimization of GUI environments. The complexity of the navigation map is uncontrollable, and the navigation structure of real applications is often complex and unpredictable, making it difficult to meet the needs of controllable training complexity and reproducibility. In multi-step navigation tasks, existing environments lack mechanisms for real-time evaluation of agent task completion, resulting in ambiguous reward signals and affecting learning efficiency. GUI elements, operation modes and interface layouts in virtual machine environments are mostly learned in the pre-training stage of the agent, making it difficult to meet the needs of building agents from scratch and exploring new training strategies. SUMMARY

[0005] One object of the present application is to provide a reinforcement learning training method based on graphical user interface simulation, a system construction method, a system, a device and a product, at least to solve the problems of missing navigation map, uncontrollable navigation complexity and difficulty in evaluating task completion in the prior art. To achieve the above object, some embodiments of the present application provide the following aspects:

[0006] In a first aspect, the present application provides a reinforcement learning training system construction method based on graphical user interface simulation, comprising:

[0007] modeling a navigation graph of the graphical user interface as a directed graph, wherein nodes of the directed graph represent interface states of the graphical user interface, and edges of the directed graph represent state transitions between the interface states of the graphical user interface;

[0008] rendering the graphical user interface according to the navigation graph structure;

[0009] driving an intelligent agent to perform operations on the graphical user interface according to the task target, and obtaining operation actions of the intelligent agent;

[0010] evaluating the operation actions of the intelligent agent and the task completion degree;

[0011] calculating a reward according to the operation actions of the intelligent agent and the task completion degree, and feeding back the reward to the intelligent agent.

[0012] In a second aspect, some embodiments of the present application further provide a reinforcement learning training system based on graphical user interface simulation, the system comprising:

[0013] a navigation graph generation module configured to model a navigation graph of the graphical user interface as a directed graph;

[0014] a screen rendering module configured to render the graphical user interface according to the navigation graph structure;

[0015] a simulation environment interaction module configured to drive an intelligent agent to perform operations according to a task target;

[0016] a state evaluation module configured to evaluate the operation actions and the task completion degree;

[0017] a reward calculation module configured to calculate a reward according to the operation actions and the task completion degree.

[0018] In a third aspect, some embodiments of the present application further provide a reinforcement learning training method of an intelligent agent, the method being performed in a system constructed in advance, the method comprising:

[0019] driving the intelligent agent to perform operations on the graphical user interface in the system according to a task target;

[0020] evaluating a task execution process and a result of the intelligent agent;

[0021] calculating a comprehensive reward according to the evaluation result, and feeding back the comprehensive reward to the intelligent agent for policy updating, and storing training data.

[0022] In a fourth aspect, some embodiments of the present application also provide an electronic device, comprising: one or more processors; and a memory storing computer program instructions which, when executed, cause the processor to perform the steps of the method described above.

[0023] In a fifth aspect, some embodiments of the present application also provide a computer readable medium having stored thereon computer program instructions executable by a processor to implement the method described above.

[0024] In a sixth aspect, some embodiments of the present application also provide a computer program product comprising computer program / instructions which, when executed by a processor, implement the steps of the method described above.

[0025] Compared with the related art, in the scheme provided by the embodiments of the present application, the navigation complexity of the GUI environment can be accurately controlled by the modeling method based on graph theory, and suitable training methods are provided for the agent training in different stages. The interface is rendered by using the navigation graph, and the icon allocation is automatically completed, which can quickly build a diversified interface environment, improve the authenticity and flexibility of the simulation, and avoid the dependence on the existing virtual machine environment. Then, the intelligent agent is driven by the task target to carry out interactive training, so that the training process is closer to the actual application scene, thereby ensuring that the learning process of the intelligent agent has a clear goal orientation. Further, by comprehensively evaluating the operation process in multiple dimensions such as distance, efficiency and behavior, the execution performance of the intelligent agent can be fully reflected, and the problem of single reward signal in the existing environment can be effectively overcome. Finally, the reward signal is generated according to the evaluation result and fed back to the intelligent agent, which can provide clear and accurate reinforcement learning training feedback for the intelligent agent, thereby significantly improving the learning efficiency and convergence effect of the intelligent agent, and providing an ideal experimental environment for the methodology of building an intelligent agent from scratch. BRIEF DESCRIPTION OF DRAWINGS

[0026] One or more embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements in which: the drawings do not necessarily bear a proportional relationship to each other.

[0027] Figure 1 A flowchart of a reinforcement learning training system construction method based on a graphical user interface simulation is provided for an exemplary embodiment of the present disclosure;

[0028] Figure 2 A flowchart of automatic icon allocation is provided for an exemplary embodiment of the present disclosure;

[0029] Figure 3A schematic diagram of a reinforcement learning training system based on graphical user interface simulation is provided for an exemplary embodiment of the present disclosure.

[0030] Figure 4 A flowchart of a reinforcement learning training method of an intelligent agent is provided for an exemplary embodiment of the present disclosure.

[0031] Figure 5 A flowchart of a reward calculation method in a reinforcement learning training method of an intelligent agent is provided for an exemplary embodiment of the present disclosure.

[0032] Figure 6 An exemplary structural diagram of an electronic device is provided for some embodiments of the present application. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in detail with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0034] Figure 1 A reinforcement learning training system construction method based on graphical user interface simulation is provided for an exemplary embodiment of the present disclosure, and the method comprises:

[0035] S101, model the graphical user interface navigation graph as a directed graph, wherein the nodes of the directed graph represent the interface states of the graphical user interface, and the edges of the directed graph represent the state transitions between the graphical user interfaces, and generate the navigation graph structure by controlling the topological parameters of the directed graph.

[0036] Specifically, the graphical user interface is modeled as a directed graph G=(V,E), wherein V represents a node set of interface states, each node corresponds to a specific interface state; E represents an edge set of state transitions, each edge corresponds to a transition operation from one interface state to another interface state. By controlling the topological structure parameters of the directed graph, including the number of nodes, the out-degree of nodes, and the path length, etc., navigation graph structures with different complexity levels can be generated to realize the adjustability of the complexity of the training environment. Among them, the basic complexity can be realized by adjusting the total number of nodes N and the average out-degree d, the path complexity can be realized by adjusting the shortest path length and the number of paths, and the connectivity complexity can be realized by adjusting the number of strongly connected components and the loop structure. For example, the complexity levels are 1-5, which are distinguished according to the total number of nodes. The more the total number of nodes, the higher the complexity.

[0037] S102, render a graphical user interface according to the navigation graph structure.

[0038] Specifically, the node information and hierarchical relationship in the navigation graph structure are acquired, the interface layout is generated according to the interface state corresponding to the node, and the interface elements are organized according to the hierarchical relationship. On this basis, the icon position and type are automatically assigned in combination with the attribute information of the interface state, so that the generated graphical user interface can truly reflect the structural characteristics of the navigation graph, thereby forming a simulation interface available for interaction of the intelligent agent.

[0039] S103, drive the intelligent agent to perform graphical user interface operation with a task target, and acquire operation actions of the intelligent agent.

[0040] Specifically, first, a preset task target is input to the intelligent agent, and the task target is converted into a task vector after semantic coding processing. Then, the intelligent agent generates corresponding operation actions based on the task vector and the current interface state information, and the operation actions include interactive instructions such as clicking, inputting, sliding, and returning. In this process, the intelligent agent and the rendered graphical user interface are interacted in real time, and the execution process and result of the operation actions are recorded and acquired. Through this step, the training process of the intelligent agent can be ensured to have target orientation, and basic data is provided for subsequent evaluation and reward calculation.

[0041] S104, evaluate the operation actions of the intelligent agent and the task completion degree.

[0042] Specifically, the operation actions of the intelligent agent and the task completion degree are evaluated, and the task completion is analyzed in multiple dimensions. The multiple-dimensional analysis can include: evaluating the shortest path difference between the current state and the target state; evaluating the operation step number, time consumption, and invalid operation condition; evaluating the exploration rationality, action diversity, and learning progress of the agent. Through the above evaluation, the execution performance of the agent can be reflected in real time, and a comprehensive task completion degree score is generated.

[0043] S105, calculate a reward through the operation actions of the intelligent agent and the task completion degree, and feed back the reward to the intelligent agent.

[0044] Specifically, the reward value is determined according to the effectiveness of the operation action, the task completion progress and the behavior performance. If the operation of the intelligent agent makes the current state closer to the target state, a positive reward is given. If the operation deviates from the target or produces invalid actions, a negative reward is given. At the same time, for the case of successful completion of the task, an additional completion reward can be set to enhance the goal-oriented nature of the agent. For example, the distance reward is used to reflect the degree of the agent approaching the target; the efficiency reward is used to reflect the simplicity and time rationality of the operation process; the behavior reward is used to encourage reasonable exploration and punish invalid or repeated operations. The final comprehensive reward is fed back to the intelligent agent in real time as a training signal in reinforcement learning, which is used to guide the policy update and parameter optimization of the intelligent agent, thereby improving the learning efficiency and convergence performance.

[0045] In the above embodiment, by using the modeling method based on graph theory, the navigation complexity of the GUI environment can be accurately controlled, and appropriate training methods can be provided for agent training at different stages. The navigation graph is used to render the interface and automatically complete the icon assignment, which can quickly build a diversified interface environment, improve the realism and flexibility of the simulation, and avoid dependence on existing virtual machine environments. Then, the intelligent agent is driven by the task target to carry out interactive training, so that the training process is closer to the actual application scenario, thereby ensuring that the learning process of the intelligent agent has a clear goal orientation. Further, by comprehensively evaluating the operation process in multiple dimensions such as distance, efficiency and behavior, the execution performance of the intelligent agent can be fully reflected, and the problem of single reward signal in the existing environment can be effectively overcome. Finally, the reward signal is generated according to the evaluation result and fed back to the intelligent agent, which can provide clear and accurate reinforcement learning training feedback for the intelligent agent, thereby significantly improving the learning efficiency and convergence effect of the intelligent agent, and providing an ideal experimental environment for the methodology of building intelligent agents from scratch.

[0046] In one embodiment, modeling the graphical user interface navigation graph as a directed graph specifically includes:

[0047] The graphical user interface navigation graph is modeled using a directed graph G=(V,E);

[0048] V represents a set of interface state nodes of the graphical user interface, each node representing a specific interface state of the graphical user interface

[0049] E represents a set of state transitions between graphical user interfaces, each edge representing an operation transition from one graphical user interface state to another graphical user interface state;

[0050] The attributes of each node include screen ID, interface layout, icon position, and interactive elements;

[0051] The attributes of each edge include trigger operation, transition condition, execution time, and success probability.

[0052] Specifically, a directed graph G=(V, E) is employed to model the navigation graph of a graphical user interface. Wherein, V represents a set of interface state nodes of the graphical user interface, each node represents an interface state. For example, a node can correspond to a login interface, a homepage, a settings interface, or a function sub-page. E represents a set of state transitions between interface states, each edge represents an operation transition from one interface state to another, such as a click button, an input instruction, or a swipe operation.

[0053] In the directed graph, the attributes of each node include: a screen ID for uniquely identifying an interface state; an interface layout for describing the spatial structure and hierarchical relationship of interface elements; an icon position for representing the coordinate position of an interactive icon in the interface; and interactive elements including buttons, input boxes, sliders, and other interactive components that can be triggered by an agent. The attributes of each edge include: a trigger operation indicating the type of operation required to transition from the current interface state to the next interface state, such as clicking the "Confirm" button or inputting a specific text; a transition condition indicating the precondition for triggering the operation, such as the input box must be filled in completely or the checkbox must be selected; an execution time recording the time required from the initiation of the operation to the completion of the interface switching, for subsequent efficiency evaluation; and a success probability indicating the likelihood of successful operation transition, for modeling the uncertainty in real applications.

[0054] Further, in one embodiment, the directed graph modeling method can generate navigation environments of different complexities by controlling the number of nodes, the connection mode of edges, and the attribute configuration. Different complexity levels can be set to adopt different connection strategies. Low complexity (1-2 levels) linear or tree structure; medium complexity (3 levels) small-world network structure; high complexity (4-5 levels) approximate random graph structure.

[0055] For example, in one embodiment, low complexity (1-2 levels), the number of nodes is 5, and the edges are linearly connected, representing a simple task flow such as "login -> homepage -> settings -> confirm -> complete"; while in another embodiment, high complexity (4-5 levels), the number of nodes can be expanded to more than 20, and the edges contain multiple branches and loops, forming a complex application navigation structure, suitable for training intelligent agents to explore and make decisions in a multi-path, multi-condition environment.

[0056] In the above manner, the directed graph not only accurately describes the jump relationship between interfaces, but also provides a structured basis for subsequent interface rendering, task generation, and reinforcement learning training.

[0057] In one embodiment, the step of rendering a graphical user interface according to the navigation graph structure further includes:

[0058] According to the navigation graph structure information, a node and a hierarchical relationship of a graphical user interface are acquired;

[0059] According to the node and the hierarchical relationship of the graphical user interface, icons are automatically assigned.

[0060] Specifically, the node is used to represent different interface states, and the hierarchical relationship is used to represent the up-down level jump logic between the interface states. When the navigation graph structure is parsed, the interface attribute of each node and the position of the node in the navigation hierarchy can be identified, for example, the root node corresponds to the main interface of the application, the child node corresponds to the function module interface, and the leaf node corresponds to the final operation interface.

[0061] According to the node and the hierarchical relationship of the graphical user interface, icons in the interface are automatically assigned. The icon assignment includes the following steps:

[0062] Based on the node out-degree, function icons are assigned. If the node out-degree is low (for example, 1-2), a small number of function icons are assigned to correspond to a limited jump path, and if the node out-degree is high (for example, ≥5), more function icons are assigned to support multiple navigation paths.

[0063] As Figure 2 Based on the hierarchical relationship, decorative icons are assigned. The root node can be assigned a brand logo or a theme icon, the first-level child node can be assigned a module distinguishing icon, and the leaf node can be assigned a result prompt or a decorative element, thereby enhancing the hierarchy and authenticity of the interface.

[0064] Based on the semantic attribute, appropriate icons are selected. If the edge attribute is “click confirmation”, the icons of “confirmation” or “next step” are assigned, if the edge attribute is “return operation”, the icons of “return” or “previous page” are assigned, and if the edge attribute is “input operation”, an input box and related prompt icons are automatically generated in the interface.

[0065] For example, the assignment rule for the leaf node (out-degree = 0) is to assign 0-1 decorative icons.

[0066] Low-degree node (out-degree = 1-2): 1-3 function icons are assigned

[0067] Medium-degree node (out-degree = 3-5): 3-6 navigation icons are assigned

[0068] High-degree node (out-degree > 5): 5-10 multi-type icons are assigned

[0069] Root node level (Level 0): 0-3 decorative icons can be configured, which are usually used for brand logo and theme decoration

[0070] First-level child node (Level 1): 0-2 decorative icons can be configured, which are used for visual differentiation between modules

[0071] Level 2: configurable 0-1 decorative icons, keeping interface simplicity

[0072] Level N: configurable 0-2 decorative icons, enhancing visual appeal of final interface

[0073] Decorative icon allocation is not limited by node out-degree, and can coexist with functional icons in the same interface, providing a more realistic visual complexity for the training environment.

[0074] Further, an intelligent icon allocation algorithm can be used to allocate based on node out-degree, level, and icon semantic relevance. Among them, node out-degree determines the number of functional icons; the level depth of the node in the navigation graph can be intelligently allocated in the icon material library through a hierarchical decorative icon configuration dictionary.

[0075] For example, the decorative icon configuration dictionary can be set to {0: 3, 1: 2, 2: 1, 'leaf': 2}. Among them,

[0076] 0: 3, means for root node level: up to 3 decorative icons

[0077] 1: 2, means for first-level child node: up to 2 decorative icons

[0078] 2: 1, means for second-level child node: up to 1 decorative icon

[0079] 'leaf': 2 means for leaf node: up to 2 decorative icons

[0080] Further, the icon allocation process can be combined with a pre-set material library, which contains system-level icons, application-level icons, functional-level icons, and decorative-level icons. The system automatically filters and layouts corresponding icons from the material library based on node semantics and level rules.

[0081] For example, the material library can be divided into:

[0082] System-level icons: simulate operating system native icons;

[0083] Application-level icons: simulate third-party application icons;

[0084] Functional-level icons: represent abstract icons of specific functions;

[0085] Decorative-level icons: decorative elements for interface beautification, further subdivided into:

[0086] Brand decorative icons: enterprise logo, product logo, etc., mainly used for root node level;

[0087] Theme decoration icon: color elements, geometric patterns, etc., for visual unity at each level;

[0088] Functional auxiliary decoration: small icons, indicators, etc., to enhance user experience without affecting functionality;

[0089] Background decoration elements: textures, gradients, etc., to provide visual hierarchy to the interface.

[0090] Layered configuration of decoration icons, for example:

[0091] # Brand focus display configuration

[0092] 0: 3, # Root node: brand logo + theme decoration

[0093] 1: 1, # Level 1 node: theme uniform decoration

[0094] 2: 0, # Level 2 node: keep simple

[0095] 'leaf': 1# Leaf node: a small amount of functional auxiliary decoration

[0096] # Minimalist style configuration

[0097] 0: 1, # Root node: only brand logo

[0098] 1: 0, # Level 1 node: no decoration

[0099] 2: 0, # Level 2 node: no decoration

[0100] 'leaf': 0# Leaf node: no decoration

[0101] # Rich visual configuration

[0102] 0: 3, # Root node: complete brand display

[0103] 1: 2, # Level 1 node: theme + auxiliary decoration

[0104] 2: 1, # Level 2 node: auxiliary decoration

[0105] 'leaf': 2# Leaf node: function + background decoration

[0106] In the above embodiment, the rendered interface not only reflects the navigation graph structure, but also maintains the consistency of icon semantics and operation logic, improving the intuitiveness and effectiveness of proxy interaction. And by configuring different configurations, various graphical style interfaces can be realized.

[0107] In one embodiment, the method uses a hierarchical state representation, specifically including:

[0108] Visual layer representation, used for encoding graphical user interface screenshots and extracting feature vectors through convolutional neural networks;

[0109] Structural layer representation, used for hierarchical description of elements in graphical user interfaces, combining spatial position, geometric attributes and interaction semantics for encoding;

[0110] Logical layer representation, used for encoding state nodes of different graphical user interfaces into state vectors based on preset rules, generating corresponding semantic vectors for task goals through language models, and encoding operation actions into time series vectors according to action types.

[0111] Specifically, the method adopts a hierarchical state representation method, and the visual layer is used for low-level feature encoding of graphical user interface screenshots. By collecting pixel-level image data of the interface and inputting it into a convolutional neural network (CNN), a multi-dimensional feature vector is extracted. For example, for a 1920x1080 resolution screenshot, a 512-dimensional feature vector can be generated after convolution and pooling processing. This feature vector can capture the overall layout, color distribution and visual features of the main interactive elements in the interface, providing a basis for subsequent state recognition and semantic understanding.

[0112] The structural layer is used for hierarchical description of GUI elements in the interface. Each interface element (such as buttons, input boxes, sliders, text labels, etc.) is assigned a spatial position (x, y coordinates), geometric attributes (width, height), interaction attributes (whether clickable / enterable) and semantic attributes (such as function type "back", "submit", "search"). For example, a button element can be represented as {x:100, y:200, w:80, h:30, type: "button", text: "Confirm"} Through this structured approach, not only can the existence of interface elements be expressed, but also the hierarchical relationship and interaction logic between elements can be reflected.

[0113] The logical layer is used to represent interface states and task goals at a higher abstraction level. Different graphical user interface state nodes are encoded into state vectors according to preset rules, for example, a certain interface state can be represented as a high-dimensional sparse vector containing node ID, current available operation set and historical state information. The task goal is input into a language model for semantic encoding to generate a vector representation corresponding to the task semantics. For example, "complete user login" can be encoded as a semantic vector. Operation action representation: the agent's operation actions are encoded into time series vectors according to action types. For example, the operation sequence "click username box → input username → click password box → input password → click login button" can be represented as a time series embedding vector, which reflects the operation order and behavior logic.

[0114] Through the above hierarchical state representation of the visual layer, the structural layer and the logical layer, multi-level information fusion can be achieved from low-level pixel features to high-level task semantics. Such hierarchical mechanism not only improves the understanding ability of the agent to the interface state, but also enhances the decision accuracy and generalization ability in the complex GUI environment.

[0115] In one embodiment, the step of evaluating the operation action and the task completion degree of the intelligent agent comprises evaluating based on the distance between the current graphical user interface of the intelligent agent and the target graphical user interface.

[0116] Specifically, the improved shortest path algorithm (such as Dijkstra) is used to calculate the shortest path length between the current interface state and the target interface state, and the weighted path distance is obtained by combining the weight factors of different operation types (such as the weight of click operation 1.0, the weight of input operation 2.0, the weight of sliding operation 1.5, and the weight of long press operation 3.0). If the operation sequence of the intelligent agent can make the actual path close to or equal to the optimal path, the evaluation result is higher; otherwise, if the path deviates or is redundant, the evaluation result is lower. At the same time, dynamic path re-planning is supported, and when the interface state or the navigation structure changes, the evaluation benchmark can be updated in real time to ensure the accuracy of the distance evaluation.

[0117] In one embodiment, the step of evaluating the operation action and the task completion degree of the intelligent agent further comprises evaluating based on the operation efficiency of the intelligent agent.

[0118] The operation efficiency evaluation comprises evaluating the operation efficiency based on the comparison between the actual operation steps, execution time of the intelligent agent and the optimal operation steps, standard time.

[0119] Specifically, the total operation steps and total time consumed by the intelligent agent to complete the task are recorded and compared with the preset optimal operation steps and standard execution time to generate an efficiency score. If the operation steps of the intelligent agent are close to the optimal steps and the execution time is close to the standard time, the score is higher; if the intelligent agent has a large number of redundant operations or the time consumed is significantly longer than the standard time, the score is lower.

[0120] Further, the invalid operations of the intelligent agent (such as clicking on non-interactive elements, repeatedly clicking on the same element, etc.) are also detected, and a negative penalty is applied to each invalid operation to ensure that the intelligent agent remains efficient and goal-oriented during execution.

[0121] Further, in one embodiment, the operation efficiency evaluation further comprises goal-oriented evaluation and operation fluency analysis.

[0122] Specifically, the target-oriented evaluation is used to determine whether the selected actions of the intelligent agent in the execution process help to shorten the path to the target interface state. If the continuous actions are all directed to the target direction, a positive evaluation is given. If there is deviation or invalid operation, a punitive evaluation is given.

[0123] The operation fluency analysis is used to detect the action coherence and consistency of the intelligent agent in the interaction process, including measuring the proportion of repeated operations, the number of invalid clicks, and the rationality of the action sequence. When the operation fluency is high, it indicates that the decision of the intelligent agent tends to be stable and efficient. When the operation fluency is low, it indicates that the agent may be in a situation of random exploration or unstable strategy.

[0124] By combining the target-oriented evaluation and the operation fluency analysis, the ability to characterize the operation quality of the intelligent agent can be further improved based on the original distance, efficiency, and behavior dimensions, thereby providing a more detailed and accurate basis for the design of the reward function.

[0125] In one embodiment, the step of evaluating the operation actions and task completion degree of the intelligent agent further includes evaluating based on the operation behavior of the intelligent agent.

[0126] The operation behavior is evaluated, including evaluating the operation behavior based on the rationality of the exploration behavior of the intelligent agent, repeated operation detection, and decision consistency.

[0127] Specifically, in terms of exploration behavior, the exploration quality of the agent is evaluated by calculating the state coverage, exploration depth, and breadth. Reasonable exploration (such as trying different paths or performing necessary backtracking) can obtain a positive score, while blind or purposeless operation will be punished. In terms of repeated operation, a sliding window algorithm is used to detect continuous repeated action patterns. If the threshold is exceeded, an incremental penalty is applied to encourage the agent to maintain strategy diversity. In terms of decision consistency, the action selection of the agent in similar states is compared. If the decision is stable and highly consistent with the task goal, a higher score is obtained. Otherwise, if randomness or inconsistency is shown, the score is reduced.

[0128] Further, in one embodiment, the evaluation of the operation behavior further includes quantitative evaluation of learning progress and evaluation of error recovery ability.

[0129] Specifically, in terms of quantitative evaluation of learning progress, the learning effect is quantified by statistics of the task completion rate, average path length, and reward value trend of the intelligent agent in different periods. If the intelligent agent can gradually improve the completion rate and shorten the path in the same complexity of the navigation environment, it indicates that the learning progress is good and a positive score is obtained. If there is long-term stagnation or performance decline, a negative feedback is given.

[0130] In terms of error recovery capability evaluation, the reaction of the intelligent agent after deviating from the target path or operating incorrectly during execution is monitored. If the intelligent agent can recover to a reasonable navigation direction in a limited number of steps through backtracking or re-planning the operation path, it is considered to have strong error recovery capability and obtains a higher score. If the intelligent agent appears to be stagnant for a long time, repeatedly performs invalid operations, or cannot return to the correct trajectory, it has poor error recovery capability and the score is reduced.

[0131] By combining the quantitative evaluation of learning progress and the evaluation of error recovery capability, the intelligent agent can be comprehensively evaluated from the aspects of long-term learning trend and immediate error correction capability, thereby more comprehensively reflecting the real level of the intelligent agent in reinforcement learning training, and providing a reference basis for subsequent reward signal design and navigation map complexity adjustment.

[0132] Further, in an embodiment, the evaluation result can be calculated by a comprehensive score formula:

[0133]

[0134] Wherein: D represents the evaluation score based on the distance between the current graphical user interface of the intelligent agent and the target graphical user interface, E represents the evaluation score of operation efficiency, B represents the evaluation score of operation behavior, and α, β, γ are weight parameters of the three dimensions, which can be dynamically adjusted according to the training stage or task requirements.

[0135] For example, at the beginning of training, the weight γ of the behavior dimension can be increased to encourage the agent to explore more; while at the later stage of training, the weight β of the efficiency dimension can be increased to promote the agent to improve the execution efficiency in the convergence stage.

[0136] Through the comprehensive score formula, the task completion degree of the intelligent agent can be quantitatively evaluated, and the result can be used as an input basis for reward calculation, thereby realizing accurate feedback and dynamic adjustment in the training process.

[0137] In an embodiment, the reward includes at least one of distance reward, efficiency reward, exploration reward, task-specific reward, and directional reward.

[0138] The distance reward is calculated based on the distance from the current graphical user interface state to the target graphical user interface state.

[0139] The efficiency reward is calculated based on step efficiency and time efficiency.

[0140] The exploration reward is calculated based on the coverage rate of the graphical user interface state.

[0141] The task-specific reward is a task-specific reward for different types of tasks.

[0142] The directionality reward is calculated based on whether the continuous action shortens the distance to the target graphical user interface.

[0143] Specifically, the distance reward is calculated based on the shortest path distance between the current graphical user interface state and the target graphical user interface state in the navigation graph, and a negative exponential decay function is used for normalization, so that the closer the distance, the higher the reward value; at the same time, the reward is weighted by combining the path complexity, transition cost and reachability, which supports real-time re-planning of the shortest path and updating of the reward benchmark when the environment changes dynamically.

[0144] The efficiency reward is calculated by combining the step efficiency and the time efficiency, the step efficiency being the ratio of the optimal operation steps to the actual operation steps, and the time efficiency being the ratio of the standard operation time to the actual operation time; during reward calculation, the reward value is decreased by a set proportion for invalid operations, continuous repeated operations and meaningless exploration behaviors, so as to guide the agent to improve the operation fluency and target orientation.

[0145] The exploration reward is calculated based on the state coverage, exploration depth and path diversity, a positive reward is given when the agent accesses a new unexplored state, the reward is maintained when the reasonable path is retraced, and the reward value is reduced if there is excessive repeated exploration, so as to balance the proportion of exploration and utilization and improve the generalization ability of the strategy.

[0146] The task-specific reward dynamically allocates reward factors for different task types, for example, in navigation tasks, priority is given to correct path selection, in interactive tasks, reward is given for successfully triggering target controls, and in input tasks, reward is given for the accuracy of input content, so as to improve the learning efficiency for different task targets.

[0147] The directionality reward is calculated based on the distance change of the continuous action sequence and the target state, if the current action shortens the target distance, a positive reward (e.g. +0.5) is given; if the action increases the distance, a negative penalty (e.g. -0.2) is applied, so as to avoid the agent falling into a local cycle or performing invalid operations through directional guidance.

[0148] Further, in one embodiment, the above-mentioned various rewards are weighted and summed to generate a final reward signal through a dynamic weight adjustment mechanism, the dynamic weight can be adaptively adjusted according to the training stage, task type and performance indicators (including task completion rate, average efficiency and decision consistency), so as to realize the progressive learning strategy of early exploration and late efficiency reinforcement, and ensure the stable convergence of the training process and the acquisition of the optimal strategy.

[0149] For example: according to the task context and the agent behavior, the comprehensive reward is calculated, and the training progress is adaptively adjusted: early exploration weight 0.3→0.4, late efficiency weight 0.25→0.35.

[0150] In this embodiment, a combined reward function is adopted, which can be dynamically adjusted according to the task type and difficulty, and precise behavior guidance is achieved. The reward function includes four core components: distance reward, efficiency reward, exploration reward and task-specific reward, which adapt to the needs of different training stages through a dynamic weight adjustment mechanism.

[0151] In one embodiment, the method further comprises automatically generating a dataset for agent training, which includes a graphical user interface navigation map and a graphical user interface, agent operation action trajectory, evaluation calculation and reward calculation data.

[0152] Specifically, the method further comprises automatically generating a dataset for intelligent agent training. It contains a graphical user interface navigation map generated according to a preset complexity parameter and corresponding interface rendering information, and records the operation action trajectory and state transition result generated by the intelligent agent when performing tasks in the navigation map based on a random strategy, heuristic strategy or trained strategy. In addition, the method automatically calculates evaluation indicators such as task completion rate, operation efficiency, path quality and decision consistency while the intelligent agent generates the trajectory, and generates reward values based on distance, efficiency, exploration, task-specific and directionality dimensions, thereby forming complete training samples of state-action-reward.

[0153] Further, in one embodiment, in order to ensure the effectiveness of the training data, further integrity check, logical reasonableness verification and reward distribution balance processing are performed on the trajectory data, abnormal or low-quality samples are removed, and data enhancement is performed through disturbance and randomization, etc. to improve the sample diversity and generalization ability. At the same time, incremental learning is supported, the newly generated data is fused and managed with historical data, the importance of the samples is evaluated, the key trajectories are preserved to prevent catastrophic forgetting, and the continuity and stability of the online training process are guaranteed.

[0154] In one embodiment, the method further comprises:

[0155] Based on the task completion rate, execution efficiency, path quality and decision accuracy of the intelligent agent, the performance indicators are calculated;

[0156] According to the performance indicators, the topological parameters of the navigation map are adjusted to control the complexity of the agent reinforcement learning.

[0157] Specifically, the method further comprises calculating a comprehensive performance index based on the task completion rate, execution efficiency, path quality and decision accuracy of the intelligent agent, wherein the comprehensive performance index is obtained by weighted summation, the task completion rate reflects the proportion of tasks successfully completed by the agent, the execution efficiency comprehensively considers the ratio of actual execution steps to optimal steps and the ratio of actual execution time to standard time, the path quality is used to measure the closeness of the selected path to the optimal path, and the decision accuracy is used to evaluate whether the action selection of the agent at the key node is correct.

[0158] Further, a comprehensive performance evaluation model is constructed, which takes the task completion rate, execution efficiency, path quality and decision accuracy as the core evaluation dimensions; and weights are preset for each evaluation dimension (for example, the weight of the task completion rate is 40%, the weight of the execution efficiency is 30%, the weight of the path quality is 20%, and the weight of the decision accuracy is 10%), and the scores of each evaluation dimension are calculated. Specifically, the above-mentioned dimension data can be calculated by the following parameters: the number of completed tasks, the total number of tasks, the list of execution time of each task, the list of optimal time of each task, the list of path quality score, the list of decision accuracy, etc.

[0159] The task completion rate score is calculated based on the ratio of the number of completed tasks to the total number of tasks;

[0160] For each task, the ratio of its actual execution time to the preset optimal execution time is calculated, and the minimum value of the ratio is limited to 1.0, and the execution efficiency score is calculated based on the average value of all task ratios;

[0161] The average value of all task path scores is calculated as the path quality score;

[0162] The average value of all task decision accuracy scores is calculated as the decision accuracy score;

[0163] The task completion rate score, the execution efficiency score, the path quality score and the decision accuracy score are weighted and summed according to their respective preset weights to generate a comprehensive performance evaluation value within the range of [0, 1], which is used to quantitatively represent the overall performance of the system.

[0164] Further, after obtaining the comprehensive performance index, the topological structure parameters of the navigation graph are dynamically adjusted according to the index, which is used to control the complexity of the training environment. When the comprehensive performance index is higher than the preset threshold, the number of nodes, the length of the path or more loops of the navigation graph are increased to increase the complexity, thereby providing a more challenging training environment for the intelligent agent; when the comprehensive performance index is lower than the threshold, the number of nodes or the connectivity of the graph is reduced to reduce the overall complexity of the navigation graph, helping the intelligent agent to gradually converge the strategy.

[0165] Further, in some embodiments, the complexity adjustment is a gradual adjustment, by recalculating the performance indicator after each training iteration and adapting the adjustment amplitude, so that the complexity change remains smooth, avoiding the impact of environmental changes on the convergence stability. Non-linear adjustment strategies can also be used, fine-tuning the complexity when the performance indicator is close to the threshold, and making large-scale structural adjustments when the performance indicator is significantly higher or lower than the threshold, to achieve difficulty adaptation and maximize training efficiency.

[0166] Figure 3 An example embodiment of the present disclosure provides a reinforcement learning training system based on graphical user interface simulation, which comprises:

[0167] A navigation graph generation module for modeling a graphical user interface navigation graph as a directed graph;

[0168] A screen rendering module for rendering a graphical user interface according to the navigation graph structure;

[0169] A simulation environment interaction module for driving an intelligent agent to operate with a task target;

[0170] A state evaluation module for evaluating operation actions and task completion;

[0171] A reward calculation module for calculating rewards through operation actions and task completion.

[0172] Specifically, the navigation graph generation module is used to model a graphical user interface navigation graph as a directed graph, wherein the nodes of the directed graph represent interface states, and the edges represent operation transitions between interface states; each node contains attributes such as screen ID, interface layout, icon position, and interactive elements, and each edge contains attributes such as trigger operation, transition condition, execution time, and success probability. By controlling the topological structure parameters of the directed graph (including the number of nodes, connectivity, and path length), navigation graph structures of different complexities can be flexibly generated.

[0173] The screen rendering module is used to generate a corresponding interface environment according to the navigation graph structure. Specifically, this module obtains the node and hierarchical relationship in the navigation graph, constructs the interface layout based on the attribute information of the interface state, and automatically generates interface elements in combination with an icon allocation algorithm, thereby quickly rendering a simulation interface available for interaction.

[0174] The simulation environment interaction module is used to drive an intelligent agent to operate with a task target in the interface environment. This module receives a task target, encodes it as a semantic vector, and inputs it into the intelligent agent; the intelligent agent generates operation actions such as clicking, inputting, sliding, and returning according to the task target vector and the current interface state; this module is responsible for receiving and executing these operation actions, and feeding back the execution results to the system.

[0175] The state evaluation module is configured to perform multi-dimensional evaluation on the operation action and the task completion degree. Specifically, the distance difference between the operation action and the target state is calculated based on the shortest path distance; the operation efficiency is measured based on the operation step number and the execution time; the operation behavior is analyzed based on indexes such as exploration rationality, repeated operation detection, and decision consistency; and further, the learning progress and the error recovery capability can be quantitatively evaluated, so as to comprehensively reflect the performance of the intelligent agent.

[0176] The reward calculation module is configured to generate a reward signal based on the evaluation result and feed back to the intelligent agent. The reward signal includes at least one of distance reward, efficiency reward, exploration reward, task-specific reward, and directional reward, so as to provide a clear and accurate learning signal for the intelligent agent, and improve the training efficiency and convergence effect.

[0177] Through the synergistic effect of the above-mentioned modules, the system provided by the embodiment can construct a navigation graph structure with controllable complexity in a simulation environment, generate an interactive graphical user interface, and comprehensively evaluate and reward feedback on the operation process of the intelligent agent, so as to realize efficient and stable reinforcement learning training.

[0178] Further, in an embodiment, the system further comprises a training data generation module.

[0179] Specifically, the training data generation module is configured to collect the navigation graph structure information output by the navigation graph generation module, including the interface state node, the state transition edge and the attribute parameters thereof. Record the operation action trajectory of the intelligent agent in the simulation environment interaction module, including the action sequence and the corresponding time stamp such as clicking, inputting, and sliding. Obtain the evaluation result data generated by the state evaluation module, including distance evaluation, efficiency evaluation, operation behavior evaluation, and task completion degree index. Collect the reward signal output by the reward calculation module, and store it in association with the corresponding operation trajectory. Integrate the above-mentioned data into a standardized data set format, which is used for training, verification and subsequent performance comparison analysis of the intelligent agent.

[0180] Through the training data generation module, the system can automatically generate a complete training data set covering the navigation structure, operation trajectory, evaluation index and reward signal without additional manual annotation, thereby supporting the continuous optimization of the intelligent agent and the transfer learning in different task scenarios.

[0181] Figure 4 A reinforcement learning training method of an intelligent agent is provided for an example embodiment of the present disclosure. The method is performed in the system in the above-mentioned embodiments. The method comprises:

[0182] S201, driving the intelligent agent to perform graphical user interface operation in the system according to the task target;

[0183] Specifically, a preset task target is input to the intelligent agent, and the task target is converted into a task vector by a language model or a semantic coding method; the intelligent agent generates operation actions including clicking, inputting, sliding, returning, etc. based on the task vector and a current interface state vector, and performs the operation actions on a rendered interface through a simulation environment interaction module; the system records the execution process and state transition results of the operation actions in real time.

[0184] S202, evaluating the task execution process and results of the intelligent agent.

[0185] Specifically, the evaluation includes multi-dimensional index calculation. The shortest path length between the current interface state and the target interface state in the navigation graph is calculated; the actual operation step number and execution time of the intelligent agent are compared with the optimal operation path and standard time; exploration rationality, repeated operation detection, and decision consistency analysis are performed; further including learning progress quantitative evaluation and error recovery capability evaluation, for reflecting the long-term learning trend and immediate error correction capability of the agent.

[0186] S203, calculating a comprehensive reward according to the evaluation results and feeding back to the intelligent agent for policy update, and storing the training data.

[0187] Specifically, the reward calculation module generates a comprehensive reward signal according to the evaluation results in different dimensions, and the reward signal includes at least one of distance reward, efficiency reward, exploration reward, task-specific reward, and directional reward; the comprehensive reward signal is used for policy gradient update or value function update of the intelligent agent after feedback. At the same time, the training data generation module integrates the navigation graph information, operation action trajectory, evaluation data, and reward signal to form a standardized training data set, which is used for continuous training and performance verification of the intelligent agent.

[0188] Further, in one embodiment, the evaluation result calculates a comprehensive reward, which includes:

[0189] Obtaining the current interface state and calculating the distance between the current state and the target state;

[0190] If the intelligent agent has reached the target, the completion reward is calculated and the reward value is returned;

[0191] If the target has not been reached, the progress reward is calculated based on the change of the state distance, and the directional reward is calculated based on the consistency of the action direction and the target direction;

[0192] The progress reward, directional reward, and completion reward are integrated to generate a reward signal and fed back to the intelligent agent.

[0193] Further, the calculation of the reward signal is based on the following function:

[0194] The distance metric function is used to calculate the shortest path distance between the current interface state and the target interface state.

[0195] The progress evaluation function is used to quantify the reduction in target distance during the operation process;

[0196] A directional guiding function is used to determine whether the direction of a continuous action is toward the target state.

[0197] Ultimately, the system will generate a reward signal based on the combined results of progress rewards, directional rewards, and completion rewards, and feed it back to the intelligent agent to support its strategy update and learning process.

[0198] For example, such as Figure 5 As shown, firstly, the task input is received and the target state (such as the target interface or specified operation) is parsed, while the current interface state is obtained. Then, the distance between the current state and the target state is calculated. Next, it is determined whether the intelligent agent has reached the target: if the target has been reached, the completion reward is calculated and returned to the intelligent agent; if the target has not yet been reached, progress rewards and directional rewards are further calculated.

[0199] The progress reward is used to measure the change in distance between the intelligent agent and the target state after the intelligent agent's operation. If the distance is shortened, a positive reward is given, and if the distance is lengthened, a penalty is given. The directional reward is used to determine whether the intelligent agent's action is pointing in the correct navigation direction. If the continuous actions converge toward the target, the reward value increases.

[0200] Through the above steps, the method can realize task-driven reinforcement learning training in the simulation system of the above embodiment, providing clear feedback and rich data support for intelligent agents, thereby effectively improving their learning efficiency and generalization ability in the graphical user interface environment.

[0201] Furthermore, in one embodiment, the task objective includes a stochastic strategy task objective and a heuristic strategy task objective.

[0202] Specifically, the randomized strategy task objective is used to drive the intelligent agent to freely explore the interactive elements in the current interface without setting a specific task objective, thereby generating diverse operation trajectories. An example of task input is: entering the text "this is a test" on page 10, and then proceeding to page 20. Under this randomized strategy, the intelligent agent's action space includes the following three categories:

[0203] Click: Randomly select one of the clickable icons or elements on the current screen and click it.

[0204] Input (Type): If there is an input element (such as a search box, text box) on the interface, randomly select one and input randomly generated text content;

[0205] Swipe: Perform a random swipe operation on the interface, with the direction and distance of the swipe being randomly generated.

[0206] The heuristic strategy task goal is used to guide the intelligent agent to efficiently complete the task according to the preset task flow in the case of having a clear task goal. For example, in the task scenarios of page 10, page 20, page 30, page 40 and page 50, the task flow includes:

[0207] Navigate to page 10 and find and click all elements named "Button A" in the page; after successfully completing the above operation, navigate to page 30 and input the text "goal completion" in the "input box B" of page 30; finally, return to page 10 from page 30 to complete the entire task flow.

[0208] Under the heuristic strategy, the action space of the intelligent agent also includes the following three categories:

[0209] Click: used to click a specified target element, such as Button A;

[0210] Input (Type): used to input specified content in an input box, such as inputting "goal completion" in input box B;

[0211] Swipe: used to scroll to find a target element in the interface, such as swiping up to find a control located below the screen.

[0212] In this embodiment, by introducing random strategies and heuristic strategies, the system can support exploratory training under unsupervised conditions and task-oriented training with a clear goal, thereby improving the generalization ability and task completion efficiency of the intelligent agent in different scenarios.

[0213] In addition, some embodiments of the present application also provide an electronic device. The electronic device can be various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0214] The electronic device includes one or more processors and a memory having stored computer program instructions that, when executed, cause the processor to perform the steps of the method provided by any one or more embodiments described above.Figure 6 An exemplary configuration of the electronic device is disclosed. The electronic device includes one or more processors 1101, a memory 1102, and an interface for connecting the components, including a high-speed interface and a low-speed interface. The components are interconnected through different buses, and can be mounted on a common main board or otherwise installed as needed. The processor can process instructions executed within the electronic device, including instructions stored in the memory or on the memory to display a GUI on an external input / output device, such as a display device coupled to the interface, with graphic information. In some other embodiments, a plurality of processors and / or buses can be used with a plurality of memories and a plurality of memories, if needed. Also, a plurality of electronic devices can be connected, each device providing part of the necessary operations. Among them, the components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present application described and / or claimed herein.

[0215] The electronic device can further include an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103, and the output device 1104 can be connected by a bus or otherwise, and are connected by a bus in the figure as an example.

[0216] The input device 1103 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 1104 can include a display device, an auxiliary lighting device (e.g., an LED), and a tactile feedback device (e.g., a vibration motor), etc. The display device can include, but is not limited to, a liquid crystal display, a light emitting diode display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0217] To provide interaction with the user, the electronic device can be a computer. The computer has a display device (e.g., a cathode ray tube or an LCD monitor) for displaying information to the user, and a keyboard and a pointing device (e.g., a mouse) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback); and input from the user can be received in any form (e.g., voice input or tactile input).

[0218] In the embodiments of the present application, the computer readable medium stores computer programs / instructions, and the computer programs / instructions are executed by the processor to implement the steps of the method provided by any one or more of the above embodiments. The computer readable medium can be included in the electronic device described in the above embodiments, or can exist separately and not be assembled into the device. The computer readable medium carries one or more computer readable instructions.

[0219] The memory 1102 can be used to store non-transitory software programs, non-transitory computer executable programs and modules. The processor 1101 executes various functions and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more of the above embodiments in the embodiments of the present application.

[0220] The memory 1102 can include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required by a function. The data storage area can store data created according to the use of the electronic device. In addition, the memory 1102 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 1102 can optionally include a memory disposed remotely with respect to the processor 1101, and these remote memories can be connected to the electronic device through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0221] It should be noted that the computer readable medium described in the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or component, or any combination of the above. More specific examples of computer readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical fiber, a portable compact disk read-only memory, an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component.

[0222] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technologies, read-only optical discs, digital versatile discs or other optical storage, magnetic cassette tapes, magnetic tape discs storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device.

[0223] Computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including object-oriented, such as Java, Smalltalk, C++, conventional procedural programming languages, such as the C programming language or similar programming languages. Program code can be executed entirely on a user computer, partially on a user computer, as a standalone software package, partially on a user computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network or a wide area network, or can be connected to an external computer (for example, through the Internet using an Internet service provider).

[0224] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. For example, a dedicated integrated circuit, a general-purpose computer or any other similar hardware device can be used. In some embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drive or soft disc and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit cooperating with the processor to perform each step or function.

[0225] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0226] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0227] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used only for distinguishing descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0228] The above merely provides specific examples of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims, and the above examples should be regarded as exemplary and non-limiting.

Claims

1. A method for constructing a reinforcement learning training system based on graphical user interface simulation, characterized in that, The method comprises: modeling a graphical user interface navigation graph as a directed graph, nodes of the directed graph representing interface states of the graphical user interface, edges of the directed graph representing state transitions between the graphical user interfaces, generating a navigation graph structure by controlling topological parameters of the directed graph; rendering the graphical user interface according to the navigation graph structure; driving an intelligent agent to operate the graphical user interface with a task target, and obtaining operation actions of the intelligent agent; evaluating the operation actions of the intelligent agent and task completion degree; calculating a reward based on the operation actions of the intelligent agent and the task completion degree, and feeding back the reward to the intelligent agent.

2. The method of claim 1, wherein, Modeling the graphical user interface navigation graph as a directed graph specifically comprises: modeling the graphical user interface navigation graph as a directed graph G=(V,E); V represents a set of interface state nodes of the graphical user interface, each node representing an interface state of the graphical user interface E represents a set of state transitions between the graphical user interfaces, each edge representing an operation transition from one graphical user interface state to another graphical user interface state; the attributes of each node include screen ID, interface layout, icon position, and interactive elements; the attributes of each edge include trigger operation, transition condition, execution time, and success probability.

3. The method of claim 1, wherein, The step of rendering the graphical user interface according to the navigation graph structure further comprises: obtaining node and hierarchical relationship of the graphical user interface according to the navigation graph structure information; automatically assigning icons according to the node and hierarchical relationship of the graphical user interface.

4. The method of claim 1, wherein, The method adopts a hierarchical state representation method, specifically comprising: visual layer representation, used for encoding a screenshot of the graphical user interface and extracting a feature vector through a convolutional neural network; structural layer representation, used for hierarchically describing elements in the graphical user interface, combining spatial position, geometric attribute, and interactive semantics for encoding; logical layer representation, used for encoding state nodes of different graphical user interfaces into state vectors according to a preset rule, generating corresponding semantic vectors through a language model for a task target, and encoding operation actions into time series vectors according to action types.

5. The method of claim 1, wherein, The step of evaluating the operation actions of the intelligent agent and the task completion degree specifically comprises: evaluating based on a distance between a current graphical user interface of the intelligent agent and a target graphical user interface; and / or evaluating based on operation efficiency of the intelligent agent; and / or evaluating based on operation behavior of the intelligent agent; and / or quantitatively evaluating learning progress and error recovery capability based on the operation actions of the intelligent agent and the task completion degree.

6. The method of claim 1, wherein, In the step of calculating a reward based on the operation actions of the intelligent agent and the task completion degree, the reward comprises at least one of distance reward, efficiency reward, exploration reward, task-specific reward, and directional reward; the distance reward is calculated based on a distance from a current graphical user interface state to a target graphical user interface state; the efficiency reward is calculated based on step efficiency and time efficiency; the exploration reward is calculated based on graphical user interface state coverage; the task-specific reward is for task-specific rewards of different task types. The direction reward is calculated based on whether the continuous action shortens the distance to the target graphical user interface.

7. The method of claim 1, wherein, The method further comprises automatically generating a dataset for agent training, the dataset comprising a graphical user interface navigation graph and graphical user interface, agent operation action trajectory, evaluation calculation and reward calculation data.

8. The method of claim 1, wherein, The method further comprises: calculating a performance index based on the task completion rate, execution efficiency, path quality and decision accuracy of the intelligent agent; adjusting the topology parameters of the navigation graph according to the performance index, for controlling the complexity of the agent reinforcement learning.

9. The method of claim 8, wherein, The performance index calculated based on the task completion rate, execution efficiency, path quality and decision accuracy of the intelligent agent specifically comprises: constructing a comprehensive performance evaluation model; the model takes the task completion rate, execution efficiency, path quality and decision accuracy as evaluation dimensions; presetting weights for each evaluation dimension to calculate the score of each evaluation dimension.

10. A reinforcement learning training system based on graphical user interface simulation, characterized by, The system comprises: a navigation graph generation module for modeling the graphical user interface navigation graph as a directed graph; a screen rendering module for rendering the graphical user interface according to the navigation graph structure; a simulation environment interaction module for driving the intelligent agent to operate with the task target; a state evaluation module for evaluating the operation action and task completion degree; a reward calculation module for reward calculation through operation action and task completion degree.

11. The reinforcement learning training system of claim 10, wherein, The system comprises a training data generation module; the training data generation module is used to collect and save the data generated in the system for subsequent training.

12. A reinforcement learning training method of an intelligent agent, characterized by, The method is performed in the system of claim 10 or 11, and the method comprises: driving the intelligent agent to operate the graphical user interface in the system with the task target; evaluating the task execution process and result of the intelligent agent; calculating a comprehensive reward according to the evaluation result and feeding it back to the intelligent agent for policy update, and storing the training data.

13. The method of claim 12, wherein, The evaluation of the task execution process and result of the intelligent agent specifically comprises: obtaining the current interface state and calculating the distance between the current state and the target state; if the intelligent agent has reached the target, calculating the completion reward and returning the reward value; if the target has not been reached, calculating the progress reward based on the change of the state distance, and calculating the direction reward based on the consistency of the action direction and the target direction.

14. An electronic device, comprising: The electronic device comprises: one or more processors; and a memory storing computer program instructions which, when executed, cause the processor to perform the steps of the method of any one of claims 1 to 9 and claims 12 or 13.

15. A computer readable medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the steps of the method of any one of claims 1 to 9 and claims 12 or 13.

16. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by the processor, implement the steps of the method of any one of claims 1 to 9 and claims 12 or 13.

Images