A GUI task planning method, system, device and storage medium

Abstract

The application discloses a GUI task planning method, system, device and storage medium. The GUI task planning method comprises the following steps: determining a GUI target operation, an associated entity and a constraint condition in a GUI task instruction; performing task splitting on the GUI task instruction based on the GUI target operation, the associated entity and the constraint condition, so as to obtain a first subtask; performing task splitting on each first subtask, so as to obtain a second subtask; determining a first dependency relationship between the first subtasks; determining a second dependency relationship between the first subtasks and the second subtasks; determining a third dependency relationship between the second subtasks; and constructing a first graph structure. In the case that the first graph structure is a directed acyclic graph structure, the GUI task instruction is executed based on the first graph structure, so that the efficiency and accuracy of the GUI task planning are improved.

Description

Technical Field

[0001] This application relates to the technical field of GUI task planning, and in particular to a GUI task planning method, system, device and storage medium. Background Technology

[0002] Currently, GUIs (Graphical User Interfaces) have become the primary way for people to interact with various electronic devices, and are widely used in many fields such as computer operating systems, mobile applications, and smart home appliance control panels. With the rapid development of technology, users' expectations for the interactive experience of GUIs are constantly increasing.

[0003] Currently, GUI task planning mainly relies on manual configuration of subtask processes. It requires manually setting the step order and primary dependency for specific tasks, making it difficult to automatically decompose new tasks. Moreover, in the presence of dependency conflicts, task execution is prone to failure due to logical errors in the steps, resulting in a low success rate for GUI task execution. Summary of the Invention

[0004] This application aims to at least address the technical problems existing in the prior art. To this end, this application proposes a GUI task planning method, system, device, and storage medium, which can improve the efficiency and accuracy of GUI task planning, thereby increasing the success rate of GUI task execution.

[0005] A first aspect of this application provides a GUI task planning method, comprising the following steps:

[0006] Upon receiving a GUI task instruction, the GUI target operation, associated entities, and constraints in the GUI task instruction are determined. The GUI target operation is the operation to be completed in the GUI task instruction. The associated entities are at least one of software entities, interface element entities, and data entities. The constraints are the rules to be followed to complete the GUI target operation.

[0007] Based on the GUI target operation, the associated entity, and the constraint, the GUI task instruction is split into a first subtask, wherein the first subtask corresponds to a number of mouse clicks, a number of keyboard inputs, or a number of GUI element selections.

[0008] Each of the first subtasks is further divided into second subtasks, where the second subtask corresponds to a mouse click, a keyboard input, or a GUI element selection.

[0009] Determine the first dependency relationship between the first subtasks; determine the second dependency relationship between the first subtask and the second subtask; determine the third dependency relationship between the second subtasks;

[0010] Construct a first graph structure, wherein the nodes of the first graph structure are a first subtask or a second subtask, the edges of the first graph structure are directed edges from the source node to the target node, and the attributes of the edges are the dependencies between the source node and the target node, the dependencies including the first dependency, the second dependency, or the third dependency;

[0011] When the first graph structure is a directed acyclic graph (DAG), the GUI task instructions are executed based on the first graph structure. When the first graph structure contains one or more cycles, the task type score and parallelism value of all nodes in each cycle are determined. The task type score is a scalar value representing the task type of the node, such as pop-up processing, data operation, or software startup. The parallelism value represents whether the node has dependent subtasks. Based on the task type score and parallelism value, the priority score of all nodes in each cycle is determined. Based on the priority score, the edges of the first graph structure are reduced to obtain a second graph structure. The GUI task instructions are then executed based on the second graph structure.

[0012] The GUI task planning method according to the embodiments of this application has at least the following beneficial effects:

[0013] This method, upon receiving a GUI task instruction, determines the GUI target operation, associated entities, and constraints within the instruction. The GUI target operation is the action to be performed in the GUI task instruction; the associated entities are at least one of software entities, interface element entities, and data entities; and the constraints are the rules to be followed to complete the GUI target operation. Based on the GUI target operation, associated entities, and constraints, the GUI task instruction is decomposed into first subtasks, each corresponding to a number of mouse clicks, keyboard inputs, or GUI element selections. Each first subtask is then processed... The process involves splitting the task into two subtasks: a mouse click, a keyboard input, or a GUI element selection; determining the first dependency between the first and second subtasks; determining the second dependency between the first and second subtasks; determining the third dependency between the second subtasks; and constructing a first graph structure where nodes represent either the first or second subtask, edges are directed edges from source nodes to target nodes, and edge attributes represent the dependency between the source and target nodes (first, second, or third dependency). The first graph structure is a directed acyclic graph. In the case of a first graph structure, GUI task instructions are executed. If one or more cycles exist in the first graph structure, the task type score and parallelism value of all nodes in each cycle are determined. The task type score is a scalar value representing the task type of a node, such as pop-up processing, data manipulation, or software startup. The parallelism value is a scalar value representing whether a node has dependent subtasks. Based on the task type score and parallelism value, the priority score of all nodes in each cycle is determined. Based on the priority score, the edges of the first graph structure are reduced to obtain the second graph structure. GUI task instructions are executed based on the second graph structure. This application achieves this through a GUI-based... By identifying target operations, associated entities, and constraints, GUI task instructions are broken down into first subtasks. This enables cross-scenario parsing of GUI tasks, improving the accuracy and efficiency of GUI task planning. Each first subtask is then further broken down into second subtasks, and the dependencies between subtasks are determined, achieving automatic task decomposition. Finally, a first graph structure, a directed acyclic graph (DAG), is constructed. GUI task instructions are executed based on either the first or second graph structure, avoiding dependency conflicts, improving the logical coherence of task decomposition, and enhancing the efficiency and accuracy of GUI task planning, thereby increasing the success rate of GUI task execution.

[0014] According to some embodiments of this application, determining the priority score of all nodes in each ring based on the task type score and parallelism score includes:

[0015] The task type score of each node in each ring is multiplied by a first preset weight value to obtain a first product value;

[0016] The parallelism value of each node in each ring is multiplied by the second preset weight value to obtain the second product value;

[0017] The first product value is added to the second product value to obtain the priority score of each node in each ring.

[0018] According to some embodiments of this application, the step of reducing the edges of the first graph structure based on the priority score to obtain the second graph structure includes:

[0019] Filter out the node with the lowest priority score in each ring of the first graph structure;

[0020] Delete the edge in each of the rings that has the lowest priority score as the target node to obtain the second graph structure.

[0021] According to some embodiments of this application, before determining the GUI target operation, associated entity, and constraints in the GUI task instruction, the following is included:

[0022] Obtain historical task execution logs; and based on the historical task execution logs, perform ambiguity resolution on the GUI task instructions to obtain ambiguity-resolved GUI task instructions;

[0023] The ambiguity-resolved GUI task instructions are filtered for stop words to obtain filtered GUI task instructions.

[0024] The filtered GUI task instructions are used to extract keywords using the term frequency-inverse document frequency method to obtain a keyword set;

[0025] The process of determining the GUI target operation, associated entities, and constraints in the GUI task instruction includes:

[0026] The keyword set is input into the trained LoRA model to obtain the GUI target operation, the associated entity, and the constraint conditions output by the trained LoRA model.

[0027] According to some embodiments of this application, the step of splitting the GUI task instruction into a first subtask based on the GUI target operation, the associated entity, and the constraints includes:

[0028] Based on the GUI target operation, the associated entity, and the constraints, the GUI task instruction is split into a first subtask according to a first preset splitting rule. The first preset splitting rule is to first activate the software, then operate the GUI elements, and finally save the result.

[0029] According to some embodiments of this application, determining the first dependency relationship between the first subtasks; determining the second dependency relationship between the first subtask and the second subtask; and determining the third dependency relationship between the second subtasks includes:

[0030] Based on the GUI interactive physical logic, the first dependency relationship between the first subtasks is determined through a rule engine method.

[0031] Based on the GUI interaction physical logic and the first dependency relationship, the second dependency relationship is determined through the rule engine method;

[0032] Based on the GUI interaction physical logic, the first dependency relationship, and the second dependency relationship, the third dependency relationship is determined through the rule engine method.

[0033] A second aspect of this application provides a GUI task planning system, the GUI task planning system comprising:

[0034] The data acquisition module is used to determine the GUI target operation, associated entities, and constraints in the GUI task instruction when receiving the GUI task instruction. The GUI target operation is the operation to be completed in the GUI task instruction. The associated entities are at least one of software entities, interface element entities, and data entities. The constraints are the rules to be followed to complete the GUI target operation.

[0035] The first subtask splitting module is used to split the GUI task instruction into a first subtask based on the GUI target operation, the associated entity and the constraint conditions. The first subtask corresponds to a number of mouse clicks, a number of keyboard inputs or a number of GUI element selections.

[0036] The second subtask splitting module is used to split each of the first subtasks to obtain the second subtask, wherein the second subtask corresponds to a mouse click, a keyboard input, or a GUI element selection.

[0037] A dependency determination module is used to determine a first dependency between the first subtasks; determine a second dependency between the first subtask and the second subtask; and determine a third dependency between the second subtasks.

[0038] A first graph structure construction module is used to construct a first graph structure, wherein the nodes of the first graph structure are a first subtask or a second subtask, the edges of the first graph structure are directed edges from the source node to the target node, and the attributes of the edges are the dependency relationship between the source node and the target node, the dependency relationship including the first dependency relationship, the second dependency relationship or the third dependency relationship;

[0039] The task execution module is configured to: execute the GUI task instructions based on the first graph structure when the first graph structure is a directed acyclic graph; determine the task type score and parallelism value of all nodes in each cycle when the first graph structure has one or more cycles, wherein the task type score is a scalar value representing the task type of the node, such as pop-up processing, data operation, or software startup, and the parallelism value is a scalar value representing whether the node has dependent subtasks; determine the priority score of all nodes in each cycle based on the task type score and parallelism value; reduce the edges of the first graph structure based on the priority score to obtain a second graph structure; and execute the GUI task instructions based on the second graph structure.

[0040] This system, upon receiving a GUI task instruction, determines the GUI target operation, associated entities, and constraints within the instruction. The GUI target operation is the action to be performed in the GUI task instruction; the associated entities are at least one of software entities, interface element entities, and data entities; and the constraints are the rules to be followed to complete the GUI target operation. Based on the GUI target operation, associated entities, and constraints, the GUI task instruction is broken down into first subtasks, each corresponding to a number of mouse clicks, keyboard inputs, or GUI element selections. Each first subtask is then processed. The process involves splitting the task into two subtasks: a mouse click, a keyboard input, or a GUI element selection; determining the first dependency between the first and second subtasks; determining the second dependency between the first and second subtasks; determining the third dependency between the second subtasks; and constructing a first graph structure where nodes represent either the first or second subtask, edges are directed edges from source nodes to target nodes, and edge attributes represent the dependency between the source and target nodes (first, second, or third dependency). The first graph structure is a directed acyclic graph. In the case of a first graph structure, GUI task instructions are executed. If one or more cycles exist in the first graph structure, the task type score and parallelism value of all nodes in each cycle are determined. The task type score is a scalar value representing the task type of a node, such as pop-up processing, data manipulation, or software startup. The parallelism value is a scalar value representing whether a node has dependent subtasks. Based on the task type score and parallelism value, the priority score of all nodes in each cycle is determined. Based on the priority score, the edges of the first graph structure are reduced to obtain the second graph structure. GUI task instructions are executed based on the second graph structure. This application achieves this through a GUI-based... By identifying target operations, associated entities, and constraints, GUI task instructions are broken down into first subtasks. This enables cross-scenario parsing of GUI tasks, improving the accuracy and efficiency of GUI task planning. Each first subtask is then further broken down into second subtasks, and the dependencies between subtasks are determined, achieving automatic task decomposition. Finally, a first graph structure, a directed acyclic graph (DAG), is constructed. GUI task instructions are executed based on either the first or second graph structure, avoiding dependency conflicts, improving the logical coherence of task decomposition, and enhancing the efficiency and accuracy of GUI task planning, thereby increasing the success rate of GUI task execution.

[0041] A third aspect of this application provides a GUI task planning electronic device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enables the at least one control processor to perform the GUI task planning method described above.

[0042] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the GUI task planning method described above.

[0043] It should be noted that the beneficial effects of the second to fourth aspects of this application with respect to the prior art are the same as the beneficial effects of the aforementioned GUI task planning system with respect to the prior art, and will not be described in detail here.

[0044] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0045] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0046] Figure 1 This is a flowchart illustrating an embodiment of the GUI task planning method provided in this application;

[0047] Figure 2 This is a schematic diagram of the structure of an embodiment of the GUI task planning system provided in this application;

[0048] Figure 3 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0049] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0050] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0051] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0052] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0053] Currently, GUIs (Graphical User Interfaces) have become the primary way for people to interact with various electronic devices, and are widely used in many fields such as computer operating systems, mobile applications, and smart home appliance control panels. With the rapid development of technology, users' expectations for the interactive experience of GUIs are constantly increasing.

[0054] Currently, GUI task planning mainly relies on manual configuration of subtask processes. It requires manually setting the step order and primary dependency for specific tasks, making it difficult to automatically decompose new tasks. Moreover, in the presence of dependency conflicts, task execution is prone to failure due to logical errors in the steps, resulting in a low success rate for GUI task execution.

[0055] To address the aforementioned technical deficiencies, embodiments of this application provide a GUI task planning method, system, device, and storage medium.

[0056] Please see Figure 1 This is a flowchart illustrating a GUI task planning method provided in an embodiment of this application. The method is applied to an electronic device, which may be a server, etc. Figure 1 As shown, the GUI task planning method includes:

[0057] Step S101: Upon receiving a GUI task instruction, determine the GUI target operation, associated entities, and constraints in the GUI task instruction. The GUI target operation is the operation to be completed in the GUI task instruction. The associated entities are at least one of software entities, interface element entities, and data entities. The constraints are the rules to be followed to complete the GUI target operation.

[0058] Step S102: Based on the GUI target operation, associated entities and constraints, the GUI task instruction is split into tasks to obtain the first subtask, wherein the first subtask corresponds to a number of mouse clicks, a number of keyboard inputs or a number of GUI element selections.

[0059] Step S103: Split each first subtask into a second subtask, where the second subtask corresponds to a mouse click, a keyboard input, or a GUI element selection.

[0060] Step S104: Determine the first dependency between the first subtasks; determine the second dependency between the first and second subtasks; determine the third dependency between the second subtasks.

[0061] Step S105: Construct a first graph structure, wherein the nodes of the first graph structure are the first subtask or the second subtask, the edges of the first graph structure are directed edges from the source node to the target node, and the attributes of the edges are the dependency relationship between the source node and the target node, which is the first dependency relationship, the second dependency relationship, or the third dependency relationship.

[0062] Step S106: If the first graph structure is a directed acyclic graph, execute GUI task instructions based on the first graph structure; if the first graph structure has one or more cycles, determine the task type score and parallelism value of all nodes in each cycle, where the task type score is a scalar value representing the task type of the node, such as pop-up processing, data operation, or software startup, and the parallelism value is a scalar value representing whether the node has dependent subtasks; based on the task type score and parallelism value, determine the priority score of all nodes in each cycle; based on the priority score, reduce the edges of the first graph structure to obtain the second graph structure; execute GUI task instructions based on the second graph structure.

[0063] The above GUI task commands can be user-input GUI task commands.

[0064] When the GUI task instruction is to merge cells A1 to C1 of Sheet1 in Excel and center them, the above GUI target operation can be two target operations: merging cells and centering text in the corresponding Excel file. When the GUI task instruction is to filter headphones priced below 500 yuan in Google Chrome and export them to Excel, the above GUI target operation can be three target operations: filtering the corresponding webpage, copying data, and saving the file.

[0065] The aforementioned software entities may include, but are not limited to, Excel and Google Chrome.

[0066] The aforementioned interface elements may include, but are not limited to, buttons, cells, and text boxes.

[0067] The aforementioned data entities may include, but are not limited to, headphones or 500 yuan.

[0068] When the GUI task instruction is to filter headphones priced below 500 yuan in Google Chrome and export them to Excel, the above constraints can be that the price is less than 500 yuan and the export format is Excel. The rule that must be followed to complete the GUI target operation can be that the price is less than 500 yuan and the export format is Excel.

[0069] The aforementioned GUI elements may include, but are not limited to, buttons, input fields, file upload controls, text, icons, pop-ups, and tooltips.

[0070] The above-mentioned dependency relationship is used to describe the association between two or more entities (entities are objects, components, tasks or systems) in a "mutually dependent, one party is affected by the other party" form. Its core essence is that the normal operation, state or existence of one entity (the dependent party) must be based on the existence, attributes or output of another entity (the dependent party).

[0071] The above dependency relationship can be represented by a constant of 1 to indicate that there is a dependency, and a constant of 0 to indicate that there is no dependency. When the dependency relationship in the properties of the above edge is 1, it means that the source node (dependent party) depends on the target node (dependent party).

[0072] In step S103 above, each first subtask is split into a second subtask. The second subtask corresponds to a mouse click, a keyboard input, or a GUI element selection. For example, if the first subtask is to select cells A1-C1, the selected cells A1-C1 can be divided into three second subtasks: moving the mouse to cell A1, holding down the left mouse button and dragging to cell C1, and releasing the left mouse button.

[0073] In step S106 above, executing GUI task instructions based on the first graph structure can be executing GUI task instructions according to the first graph structure.

[0074] In step S106 above, when there are one or more rings in the first graph structure, the task type score and parallelism value of all nodes in each ring can be determined by using natural language methods to confirm the task type of all nodes in each ring. If the task type of a node in the ring is pop-up processing, the task type score of that node is 10; if the task type of a node in the ring is data operation, the task type score of that node is 8; if the task type of a node in the ring is software startup, the task type score of that node is 5; if a node in the ring has dependent subtasks, the parallelism value of that node is 1; if a node in the ring has no dependent subtasks, the parallelism value of that node is 0.

[0075] In step S106 above, executing GUI task instructions based on the second graph structure can be executing GUI task instructions according to the second graph structure.

[0076] This application reduces the edges of the first graph structure by prioritizing the score, so that the second graph structure is a directed acyclic graph, thus avoiding circular dependencies.

[0077] This method, upon receiving a GUI task instruction, determines the GUI target operation, associated entities, and constraints within the instruction. The GUI target operation is the action to be performed in the GUI task instruction; the associated entities are at least one of software entities, interface element entities, and data entities; and the constraints are the rules to be followed to complete the GUI target operation. Based on the GUI target operation, associated entities, and constraints, the GUI task instruction is decomposed into first subtasks, each corresponding to a number of mouse clicks, keyboard inputs, or GUI element selections. Each first subtask is then processed... The process involves splitting the task into two subtasks: a mouse click, a keyboard input, or a GUI element selection; determining the first dependency between the first and second subtasks; determining the second dependency between the first and second subtasks; determining the third dependency between the second subtasks; and constructing a first graph structure where nodes represent either the first or second subtask, edges are directed edges from source nodes to target nodes, and edge attributes represent the dependency between the source and target nodes (first, second, or third dependency). The first graph structure is a directed acyclic graph. In the case of a first graph structure, GUI task instructions are executed. If one or more cycles exist in the first graph structure, the task type score and parallelism value of all nodes in each cycle are determined. The task type score is a scalar value representing the task type of a node, such as pop-up processing, data manipulation, or software startup. The parallelism value is a scalar value representing whether a node has dependent subtasks. Based on the task type score and parallelism value, the priority score of all nodes in each cycle is determined. Based on the priority score, the edges of the first graph structure are reduced to obtain the second graph structure. GUI task instructions are executed based on the second graph structure. This application achieves this through a GUI-based... By identifying target operations, associated entities, and constraints, GUI task instructions are broken down into first subtasks. This enables cross-scenario parsing of GUI tasks, improving the accuracy and efficiency of GUI task planning. Each first subtask is then further broken down into second subtasks, and the dependencies between subtasks are determined, achieving automatic task decomposition. Finally, a first graph structure, a directed acyclic graph (DAG), is constructed. GUI task instructions are executed based on either the first or second graph structure, avoiding dependency conflicts, improving the logical coherence of task decomposition, and enhancing the efficiency and accuracy of GUI task planning, thereby increasing the success rate of GUI task execution.

[0078] In some embodiments, the method may also include, but is not limited to, steps S201 to S204:

[0079] Step S201: In the case that there are one or more rings in the first graph structure, determine the task type score and parallelism value of all nodes in each ring. The task type score is a scalar value that represents the task type of the node. The task type is pop-up processing, data operation or software startup. The parallelism value is a scalar value that represents whether the node has dependent subtasks.

[0080] Step S202: Determine the priority score of all nodes in each ring based on the task type score and parallelism score;

[0081] Step S203: Based on the priority score, the edges of the first graph structure are reduced to obtain the second graph structure;

[0082] Step S204: Execute GUI task instructions based on the second graph structure.

[0083] In step S201 above, when there are one or more rings in the first graph structure, the task type score and parallelism value of all nodes in each ring can be determined by using natural language methods to confirm the task type of all nodes in each ring. If the task type of a node in the ring is pop-up processing, the task type score of that node is 10; if the task type of a node in the ring is data operation, the task type score of that node is 8; if the task type of a node in the ring is software startup, the task type score of that node is 5; if a node in the ring has dependent subtasks, the parallelism value of that node is 1; if a node in the ring has no dependent subtasks, the parallelism value of that node is 0.

[0084] In step S201 above, executing GUI task instructions based on the second graph structure can be executing GUI task instructions according to the second graph structure.

[0085] This application reduces the edges of the first graph structure by prioritizing the score, so that the second graph structure is a directed acyclic graph, thus avoiding circular dependencies.

[0086] In some embodiments, the priority score of all nodes in each ring determined by step S106 based on task type score and parallelism score may include, but is not limited to, steps S201 to S203:

[0087] Step S201: Multiply the task type score of each node in each ring by the first preset weight value to obtain the first product value;

[0088] Step S202: Multiply the parallelism value of each node in each ring by the second preset weight value to obtain the second product value;

[0089] Step S203: Add the first product value and the second product value to obtain the priority score of each node in each ring.

[0090] The first preset weight value mentioned above is a constant value that is preset according to actual needs.

[0091] The aforementioned second preset weight value is a constant value pre-set according to actual needs. The sum of the aforementioned first preset weight value and the second preset weight value is 1.

[0092] This application improves the accuracy of priority scores by setting different weight values ​​for task type scores and parallelism scores, thereby improving the accuracy of GUI task planning.

[0093] In some embodiments, step S106, which involves removing edges from the first graph structure based on priority scores to obtain the second graph structure, may include, but is not limited to, steps S301 to S302:

[0094] Step S301: Filter out the node with the lowest priority value in each ring of the first graph structure;

[0095] Step S302: Delete the edges in each cycle that have the lowest priority score as the target node to obtain the second graph structure.

[0096] This application avoids circular dependencies and improves the accuracy of GUI task planning by deleting edges in each cycle that target the node with the lowest priority score, making the second graph structure a directed acyclic graph.

[0097] In some embodiments, prior to step S101, the method may include, but is not limited to, steps S401 to S403:

[0098] Step S401: Obtain historical task execution logs; and resolve ambiguities in GUI task instructions based on historical task execution logs to obtain ambiguity-resolved GUI task instructions;

[0099] Step S402: Filter the GUI task instructions after ambiguity resolution by stopping words to obtain the filtered GUI task instructions;

[0100] Step S403: Extract keywords from the filtered GUI task instructions using the term frequency-inverse document frequency method to obtain a keyword set;

[0101] Step S101 may include, but is not limited to, step S404:

[0102] Step S404: Input the keyword set into the trained LoRA model to obtain the GUI target operation, associated entities and constraints output by the trained LoRA model.

[0103] In step S401 above, the GUI task instructions are ambiguously resolved based on the historical task execution log. The GUI task instructions after ambiguity resolution can be obtained by retrieving the operating software (such as Excel) from the historical task execution log when the operating software is not specified in the GUI task instructions. The operating software in the historical task execution log is then used as the operating software of the GUI task instructions.

[0104] This application improves the accuracy and efficiency of GUI task planning by transforming GUI task instructions into GUI target operations, associated entities, and constraints, providing data for subsequent task decomposition.

[0105] In some embodiments, step S102 may include, but is not limited to, steps S501 to S503:

[0106] Step S501: Based on the GUI target operation, associated entities and constraints, the GUI task instruction is split into tasks according to the first preset splitting rule to obtain the first subtask. The first preset splitting rule is to first activate the software, then operate the GUI elements, and finally save the results.

[0107] In step S501 above, based on the GUI target operation, associated entities, and constraints, the GUI task instruction is split according to the first preset splitting rule to obtain the first subtask. This subtask can be, in the case where the GUI task instruction is "Merge cells A1 to C1 of Sheet1 and center them", determining the GUI target operation, associated entities, and constraints in the GUI task instruction, and splitting the task "Merge cells in Excel and center them" into "Open Excel" according to the first preset splitting rule. Switch to Sheet1 Select A1 to C1 Click the merge button Center Save the file.

[0108] In some embodiments, step S104 may include, but is not limited to, steps S601 to S603:

[0109] Step S601: Based on the GUI interactive physical logic, determine the first dependency relationship between the first subtasks through the rule engine method;

[0110] Step S602: Based on the GUI interaction physical logic and the first dependency relationship, determine the second dependency relationship through the rule engine method;

[0111] Step S603: Based on the GUI interactive physical logic, the first dependency relationship, and the second dependency relationship, determine the third dependency relationship through the rule engine method.

[0112] The aforementioned GUI interaction physical logic refers to the principles and laws of the physical world that must be followed in graphical user interface design. For example, in the refinement of press feedback, the pressure-sensitive button on the touch screen corresponds to "stress" when pressed, and the degree of deformation (strain) of the button increases with the increase of pressure; controls such as task progress bars and integral slots show "plastic filling" as the progress increases.

[0113] In step S601 above, determining the first dependency relationship between the first subtasks based on the GUI interaction physical logic and using the rule engine method can be achieved by extracting general dependency logic from large-scale GUI task data, and then determining the first dependency relationship between the first subtasks according to the general dependency logic and the GUI interaction physical logic using the rule engine method.

[0114] In step S602 above, the second dependency relationship is determined by the rule engine method based on the GUI interaction physical logic and the first dependency relationship. This can be done by determining the second dependency relationship between the first subtask and the second subtask according to the general dependency logic, the GUI interaction physical logic and the first dependency relationship.

[0115] In step S603 above, the third dependency is determined by the rule engine method based on the GUI interaction physical logic, the first dependency relationship, and the second dependency relationship. This can be done by determining the third dependency relationship between the second subtasks according to the general dependency logic, the GUI interaction physical logic, the first dependency relationship, and the second dependency relationship.

[0116] Because the second subtask is derived from the first subtask, the completion of the first-level subtask depends on the sequential completion of all its second-level subtasks. At the same time, the prerequisites of the first-level subtask (such as "selecting A1-C1" depending on "switching to Sheet1") will be passed on to all its second-level subtasks. By confirming the first, second, or third dependency relationships, this application can achieve inter-level dependency continuity, providing data basis for subsequent task planning and improving the accuracy and efficiency of GUI task planning.

[0117] Additionally, refer to Figure 2 One embodiment of this application provides a GUI task planning system, including a data acquisition module 1100, a first subtask splitting module 1200, a second subtask splitting module 1300, a dependency determination module 1400, a first graph structure construction module 1500, and a task execution module 1600, wherein:

[0118] The data acquisition module 1100 is used to determine the GUI target operation, associated entities and constraints in the GUI task instruction when receiving the GUI task instruction. The GUI target operation is the operation to be completed in the GUI task instruction, the associated entity is at least one of software entity, interface element entity and data entity, and the constraints are the rules to be followed to complete the GUI target operation.

[0119] The first subtask splitting module 1200 is used to split GUI task instructions into first subtasks based on GUI target operations, associated entities and constraints. The first subtask corresponds to a number of mouse clicks, a number of keyboard inputs or a number of GUI element selections.

[0120] The second subtask splitting module 1300 is used to split each first subtask into a second subtask, wherein the second subtask corresponds to a mouse click, a keyboard input, or a GUI element selection.

[0121] The dependency determination module 1400 is used to determine the first dependency between the first subtasks; determine the second dependency between the first subtasks and the second subtasks; and determine the third dependency between the second subtasks.

[0122] The first graph structure construction module 1500 is used to construct the first graph structure, wherein the nodes of the first graph structure are the first subtask or the second subtask, the edges of the first graph structure are directed edges from the source node to the target node, and the attributes of the edges are the dependency relationship between the source node and the target node, which is the first dependency relationship, the second dependency relationship, or the third dependency relationship.

[0123] The task execution module 1600 is used to execute GUI task instructions based on the first graph structure when the first graph structure is a directed acyclic graph structure; when the first graph structure has one or more cycles, it determines the task type score and parallelism value of all nodes in each cycle, where the task type score is a scalar value representing the task type of the node, such as pop-up processing, data operation, or software startup, and the parallelism value is a scalar value representing whether the node has dependent subtasks; based on the task type score and parallelism value, it determines the priority score of all nodes in each cycle; based on the priority score, it reduces the edges of the first graph structure to obtain the second graph structure; and it executes GUI task instructions based on the second graph structure.

[0124] This system, upon receiving a GUI task instruction, determines the GUI target operation, associated entities, and constraints within the instruction. The GUI target operation is the action to be performed in the GUI task instruction; the associated entities are at least one of software entities, interface element entities, and data entities; and the constraints are the rules to be followed to complete the GUI target operation. Based on the GUI target operation, associated entities, and constraints, the GUI task instruction is broken down into first subtasks, each corresponding to a number of mouse clicks, keyboard inputs, or GUI element selections. Each first subtask is then processed. The process involves splitting the task into two subtasks: a mouse click, a keyboard input, or a GUI element selection; determining the first dependency between the first and second subtasks; determining the second dependency between the first and second subtasks; determining the third dependency between the second subtasks; and constructing a first graph structure where nodes represent either the first or second subtask, edges are directed edges from source nodes to target nodes, and edge attributes represent the dependency between the source and target nodes (first, second, or third dependency). The first graph structure is a directed acyclic graph. In the case of a first graph structure, GUI task instructions are executed. If one or more cycles exist in the first graph structure, the task type score and parallelism value of all nodes in each cycle are determined. The task type score is a scalar value representing the task type of a node, such as pop-up processing, data manipulation, or software startup. The parallelism value is a scalar value representing whether a node has dependent subtasks. Based on the task type score and parallelism value, the priority score of all nodes in each cycle is determined. Based on the priority score, the edges of the first graph structure are reduced to obtain the second graph structure. GUI task instructions are executed based on the second graph structure. This application achieves this through a GUI-based... By identifying target operations, associated entities, and constraints, GUI task instructions are broken down into first subtasks. This enables cross-scenario parsing of GUI tasks, improving the accuracy and efficiency of GUI task planning. Each first subtask is then further broken down into second subtasks, and the dependencies between subtasks are determined, achieving automatic task decomposition. Finally, a first graph structure, a directed acyclic graph (DAG), is constructed. GUI task instructions are executed based on either the first or second graph structure, avoiding dependency conflicts, improving the logical coherence of task decomposition, and enhancing the efficiency and accuracy of GUI task planning, thereby increasing the success rate of GUI task execution.

[0125] It should be noted that the system embodiments described above are based on the same inventive concept as the method embodiments described above. Therefore, the relevant content of the method embodiments described above is also applicable to the system embodiments described above, and will not be repeated here.

[0126] Figure 3A schematic diagram of the hardware structure for GUI task planning provided in an embodiment of this application is shown.

[0127] The GUI task planning device may include a processor 301 and a memory 302 storing computer program instructions.

[0128] Specifically, the processor 301 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0129] Memory 302 may include mass storage for data or instructions. For example, and not limitingly, memory 302 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 302 may include removable or non-removable (or fixed) media. Where appropriate, memory 302 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 302 is non-volatile solid-state memory.

[0130] In some embodiments, memory 302 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Thus, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0131] The processor 301 implements any of the GUI task planning methods described in the above embodiments by reading and executing computer program instructions stored in the memory 302.

[0132] In one example, the GUI task planning device may also include a communication interface 303 and a bus 310. Wherein, as Figure 3 As shown, the processor 301, memory 302, and communication interface 303 are connected through bus 310 and complete communication with each other.

[0133] The communication interface 303 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0134] Bus 310 includes hardware, software, or both, that couples components of a GUI task planning device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 310 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0135] This GUI task planning device can execute the GUI task planning method in the embodiments of this application based on a 3D design model, thereby achieving a combination of Figure 1 and Figure 2 The GUI task planning method and system are described.

[0136] Furthermore, in conjunction with the GUI task planning methods in the above embodiments, this application embodiment can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the GUI task planning methods in the above embodiments.

[0137] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0138] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0139] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0140] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0141] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A GUI task planning method, characterized by, The GUI task planning method comprises: In the case of receiving a GUI task instruction, determining a GUI target operation, an associated entity and a constraint condition in the GUI task instruction, wherein the GUI target operation is an operation to be completed in the GUI task instruction, the associated entity is at least one of a software entity, an interface element entity and a data entity, and the constraint condition is a rule to be followed to complete the GUI target operation; Based on the GUI target operation, the associated entity and the constraint condition, task splitting is performed on the GUI task instruction to obtain a first subtask, wherein the first subtask corresponds to a number of mouse clicks, a number of keyboard inputs or a number of GUI element selections; Task splitting is performed on each first subtask to obtain a second subtask, wherein the second subtask corresponds to one mouse click, one keyboard input or one GUI element selection; A first dependency relationship between the first subtasks is determined, a second dependency relationship between the first subtasks and the second subtasks is determined, and a third dependency relationship between the second subtasks is determined; A first graph structure is constructed, wherein the nodes of the first graph structure are the first subtasks or the second subtasks, the edges of the first graph structure are directed edges from source nodes to target nodes, the attributes of the edges are dependency relationships between the source nodes and the target nodes, and the dependency relationships include the first dependency relationship, the second dependency relationship or the third dependency relationship; In the case that the first graph structure is a directed acyclic graph structure, the GUI task instruction is executed based on the first graph structure; in the case that the first graph structure has one or more loops, a task type score and a parallelism value of all nodes in each loop are determined, wherein the task type score is a scalar value representing the task type of the node, the task type is a pop-up window processing, a data operation or a software startup, and the parallelism value is a scalar value representing whether the node has a dependent subtask; based on the task type score and the parallelism value, a priority score of all nodes in each loop is determined; based on the priority score, the edges of the first graph structure are pruned to obtain a second graph structure; and the GUI task instruction is executed based on the second graph structure.

2. The GUI task planning method of claim 1, wherein, The determination of the priority score of all nodes in each loop based on the task type score and the parallelism value comprises: Multiplying the task type score of each node in each loop by a first preset weight value to obtain a first product value; Multiplying the parallelism value of each node in each loop by a second preset weight value to obtain a second product value; Adding the first product value and the second product value to obtain the priority score of each node in each loop.

3. The GUI task planning method of claim 2, wherein, The pruning of the edges of the first graph structure based on the priority score to obtain a second graph structure comprises: Filtering out the node with the lowest priority score in each loop in the first graph structure; Deleting the edges of each loop that have the node with the lowest priority score as the target node to obtain the second graph structure.

4. The GUI task planning method of claim 1, wherein, Before the determining of the GUI target operation, the associated entity and the constraint condition in the GUI task instruction, comprising: obtaining a historical task execution log; and based on the historical task execution log, ambiguity resolution is performed on the GUI task instruction to obtain an ambiguity-resolved GUI task instruction; stop word filtering is performed on the ambiguity-resolved GUI task instruction to obtain a filtered GUI task instruction; key word extraction is performed on the filtered GUI task instruction by a term frequency-inverse document frequency method to obtain a key word set; the determining of the GUI target operation, the associated entity and the constraint condition in the GUI task instruction, comprising: the key word set is input into a trained LoRA model to obtain the GUI target operation, the associated entity and the constraint condition output by the trained LoRA model.

5. The GUI task planning method of claim 1, wherein, the task splitting of the GUI task instruction based on the GUI target operation, the associated entity and the constraint condition to obtain a first subtask, comprising: based on the GUI target operation, the associated entity and the constraint condition, the GUI task instruction is split according to a first preset splitting rule to obtain a first subtask, wherein the first preset splitting rule is to activate software first, then operate GUI elements, and finally save the results.

6. The GUI task planning method of claim 1, wherein, the determining of the first dependency relationship between the first subtasks; the determining of the second dependency relationship between the first subtask and the second subtask; the determining of the third dependency relationship between the second subtasks, comprising: based on the GUI interaction physical logic, the first dependency relationship between the first subtasks is determined by a rule engine method; based on the GUI interaction physical logic and the first dependency relationship, the second dependency relationship is determined by the rule engine method; based on the GUI interaction physical logic, the first dependency relationship and the second dependency relationship, the third dependency relationship is determined by the rule engine method.

7. A GUI task planning system characterized by, the GUI task planning system comprises: a data acquisition module configured to determine a GUI target operation, an associated entity and a constraint condition in a GUI task instruction when the GUI task instruction is received, wherein the GUI target operation is an operation to be completed in the GUI task instruction, the associated entity is at least one of a software entity, an interface element entity and a data entity, and the constraint condition is a rule to be followed to complete the GUI target operation; a first subtask splitting module configured to split the GUI task instruction based on the GUI target operation, the associated entity and the constraint condition to obtain a first subtask, wherein the first subtask corresponds to a number of mouse clicks, a number of keyboard inputs or a number of GUI element selections; a second subtask splitting module configured to split each first subtask to obtain a second subtask, wherein the second subtask corresponds to one mouse click, one keyboard input or one GUI element selection; A dependency relationship determining module is configured to determine a first dependency relationship between the first sub-tasks, determine a second dependency relationship between the first sub-tasks and the second sub-tasks, and determine a third dependency relationship between the second sub-tasks. A first graph structure constructing module is configured to construct a first graph structure, wherein a node of the first graph structure is a first sub-task or a second sub-task, an edge of the first graph structure is a directed edge from a source node to a target node, and an attribute of the edge is a dependency relationship between the source node and the target node, the dependency relationship including the first dependency relationship, the second dependency relationship, or the third dependency relationship. A task executing module is configured to execute the GUI task instruction based on the first graph structure when the first graph structure is a directed acyclic graph structure, determine a task type score and a parallelism value of all nodes in each loop when the first graph structure has one or more loops, wherein the task type score is a scalar value representing a task type of the node, the task type being a pop-up processing, a data operation, or a software starting, and the parallelism value is a scalar value representing whether the node has a dependent sub-task, determine a priority score of all nodes in each loop based on the task type score and the parallelism value, and perform pruning on edges of the first graph structure based on the priority score to obtain a second graph structure, and execute the GUI task instruction based on the second graph structure.

8. A GUI task planning device characterized by comprising: The computer readable storage medium stores computer executable instructions, and the computer executable instructions are used to make a computer execute a GUI task planning method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer readable storage medium stores computer executable instructions, and the computer executable instructions are used to make a computer execute a GUI task planning method according to any one of claims 1 to 6.

Images