A complex task process automation system and method

Abstract

The present invention discloses a complex task process automation system and method, belonging to the field of software testing. The system comprises a task recorder and distributor, a scheduler, an operation action recorder, an exception handler, a task rewriting and subtask planner, an instruction reasoner, an instruction executor, an execution state detector and a path recorder. The scheduler is a logic control unit of the entire process. The path recorder comprises three components: a page recognition model, a page id allocation and a path record. The instruction reasoner is responsible for obtaining the GUI page of the current system according to the historical operation record and the current task input by the scheduler, using a cross-modal large model trained by a domain data set for reasoning. The execution state detector is used to detect whether the action execution is successful. The instruction executor is used to interact with the system, reduce the difficulty of process automation execution, ensure stability, convert complex logic into a semantically clear instruction set, and greatly improve the system action space and execution capability.

Description

Technical Field

[0001] The present invention belongs to the technical field of software testing, and in particular relates to a complex task flow automation system and method. Background Art

[0002] In the current wave of digitalization and informatization, various complex operating systems and applications need to be operated and used in production and life. For example, mobile phones, various apps in mobile phones, car systems, counter systems in the industry, etc. Each system and each app will be regularly upgraded, and new functions will be launched from time to time. Especially for apps involving cash transactions in banks and securities, a lot of testing work is required before each functional module is launched. Taking the whole machine test of mobile phones as an example, the number of test cases is generally in the tens of thousands. Among the tens of thousands of test cases, only hundreds can be automated, and more than 90% of the test cases need to be manually executed by manpower. These testing tasks are very boring for humans, and the timeliness and reliability are not guaranteed for enterprises.

[0003] Defects and shortcomings of the existing technology: 1. When testing a certain function point in the current automated testing system, the test engineer needs to record the pseudo code test script of the execution process in advance. This recording process is very inefficient, time-consuming and labor-intensive. A mature automated testing engineer can only record 5-10 test scripts a day, while a small business system app (such as a bank app) usually has hundreds of function points to test, and each function point usually has several test cases that require writing test scripts. That is, a small system upgrade requires more than 6 man-months of test script development work; 2. With the support of big model technology, some research work on webagent and appagent (i.e. web page execution agent, app intelligent execution) has been launched recently. However, the accuracy and stability of big models are not yet sufficient to meet the needs of industrial production. For example, the accuracy of each reasoning of the big model is 90%, and a functional test generally requires multiple steps of execution. For example, if 8 steps are required, the final accuracy is only 43%. In actual testing, due to the relatively complex operations within the professional business system, gpt4 and gpt4o cannot achieve 90% reasoning accuracy. At present, these explorations are in the experimental stage, and no research has yet applied them to the automated testing of apps.

[0004] In view of the above-mentioned shortcomings of the prior art, the present invention is committed to providing a complex task process automation system and method, and solves the above-mentioned problems through innovation of the system and method. Summary of the invention

[0005] To achieve the above-mentioned purpose, the technical solution of the present invention is as follows: a complex task process automation system, including a task recorder and distributor, a scheduler, an operation action recorder, an exception handler, a task rewriting and subtask planner, an instruction reasoner, an instruction executor, an execution state detector and a path recorder, wherein the scheduler is a logic control unit of the entire process, wherein: The path recorder contains three components: page identification model, page id allocation and path recording; The instruction reasoner is responsible for inferring the current task and the historical operation records passed by the scheduler and the GUI page of the current system using a large cross-modal model trained with the domain dataset; An execution status detector is used to detect whether the action execution is successful; The command executor is used to interact with the system.

[0006] Preferably, the page recognition model is obtained by a visual classification neural network combined with ternary loss function training, wherein the path record is a directed graph, the vertices of the graph are image IDs, and the directed edges are associated elements that need to be operated from vertex 1 to vertex 2.

[0007] Preferably, the process includes: obtaining the current page screenshot passed in by the scheduler, the page recognition model compares the current page screenshot with the picture set in the path record, and calculates the similarity. If the page recognition model determines that it is a new picture, the page id allocator is called to assign an id to it, and then the path record component is called to add a vertex and edge to complete a path record.

[0008] Preferably, the instructions include not only system operation instructions, but also operation instructions with the aid of external tools and logic instructions.

[0009] Based on the above technical features, the instructions not only include system operation instructions such as clicking, inputting, and sliding, but also include operation instructions using external tools such as plugging and unplugging network cables and rotating the screen, as well as logical instructions such as process end, end if inspection passed, and end if inspection failed.

[0010] Preferably, the execution state detector is implemented by OCR target content recognition plus regular expression.

[0011] Based on the above technical features, the execution status detector is used to detect whether the action is successfully executed. It can be achieved through OCR target content recognition plus regular expression, or through other verification means.

[0012] Preferably, the task rewriting and subtask planner are composed of a large language model and a prompt word template.

[0013] Preferably, the instruction executor is an adb debugging tool or a mechanical arm physical device.

[0014] Preferably, a complex task process automation method implemented by the system comprises the following steps: In the first step, the original task is input into the scheduler. The scheduler calls the task rewriting and subtask planner to complete the semantic parsing, information completion and task decomposition of the original task, and returns the result to the scheduler to enter the second step. In the second step, the scheduler receives the return results of the task rewriting and subtask planner and sends the results to the task recorder and distributor. The task recorder and distributor stores the task queue and returns a subtask to the scheduler, and then enters the third step; In the third step, the scheduler requests relevant historical operation information from the operation action recorder and sends subtasks to the instruction reasoner. The instruction reasoner reasons about the current task and returns an execution action instruction to the scheduler, entering the fourth step; In the fourth step, the scheduler judges the received execution action instruction action. If it is an end instruction, it goes to the sixth step; otherwise, the scheduler sends the received execution action instruction to the instruction executor, and the instruction executor connects to the actual system to start executing the instruction. The scheduler synchronously sends the execution action instruction to the operation action recorder to register the instruction to be executed, and then goes to the fifth step; In the fifth step, after a set delay, the scheduler calls the execution status detector to determine the execution status of the instruction and returns the result to the scheduler; If the return result is successful, the scheduler will complete the following three actions: 1) Send the result to the operation action recorder, and the operation action recorder changes the state of the corresponding execution instruction; 2) Call path recorder; 3) Go to step 3 to continue execution; if it fails, call step 4 again to try the set number of times. If it fails after the set number of times, send the result to the operation action recorder, and the operation action recorder changes the state of the corresponding execution instruction and goes to step 7 at the same time; In the sixth step, the scheduler makes a request to the task record and distributor. If the task popped up by the task record and distributor is empty, the whole process ends. Otherwise, the scheduler obtains a new execution instruction and enters the third step. In the seventh step, the scheduler calls the exception handler. If the exception handler adopts direct external processing, the process ends. If it is not external processing, the "request other path attempts" identifier is returned to the scheduler, and the eighth step is entered; In step 8, the scheduler receives the "request other path attempts" identifier from the exception handler, searches for the path record in the path recorder, goes back N steps, and enters step 3 again.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: a complex task process automation system and method, including a task recorder and distributor, a scheduler, an operation action recorder, an exception handler, a task rewriting and subtask planner, an instruction reasoner, an instruction executor, an execution state detector, and a path recorder, a total of 9 modules, which work together to greatly improve the accuracy and stability of the model; Specifically, the present invention solves the intelligence and reliability issues of process automation of complex tasks through the following technologies: 1. The introduction of task rewriting and subtask planner can solve the problem of non-standard use case description and mixing multiple test inspection tasks in manual test cases, re-divide the problem into semantically complete subtasks, and reduce the difficulty of process automation execution; 2. The path recorder proposes a novel method for recording the internal structure tree of the system, realizing the display and construction of the spatial map, so that historical exploratory behaviors can be recorded and retrieved, thus ensuring the reliable process automation of complex tasks; 3. The execution agent is innovatively split into a command reasoner, a command executor, and an execution status detector, so that the execution effect of each step has a judgment procedure to ensure stability. The execution instructions are innovatively extended from system operation instructions to external tool call instructions and logical instructions, turning complex logic into a semantically clear instruction set, greatly improving the system's action space and execution capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a structural diagram of the complex task process automation system and method in this embodiment; Figure 2 Schematic diagram of path recording in this embodiment. DETAILED DESCRIPTION

[0017] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0018] Example: Figure 1 to Figure 2 As shown in the figure, the system consists of 9 modules: task recorder and distributor, scheduler, operation action recorder, exception handler, task rewriting and subtask planner, instruction reasoner, instruction executor, execution state detector, and path recorder. The scheduler is the logic control unit of the entire process, and the other modules are specific functional modules.

[0019] Here’s how it works: In the first step, the original task is input into the scheduler, and the scheduler calls the task rewriting and subtask planner to complete the semantic parsing, information completion and task decomposition of the original task, and returns the result to the scheduler. Enter the second step.

[0020] In the second step, the scheduler receives the return results of the task rewriting and subtask planner and sends the results to the task recorder and distributor. The task recorder and distributor stores the task queue and returns a subtask to the scheduler. Enter the third step.

[0021] In the third step, the scheduler requests relevant historical operation information from the operation action recorder and sends subtasks to the instruction reasoner. The instruction reasoner infers the current task and returns an execution action instruction to the scheduler. Enter the fourth step.

[0022] In the fourth step, the scheduler judges the received execution action instruction action. If it is an end instruction, it goes to the sixth step. Otherwise, the scheduler sends the received execution action instruction to the instruction executor, and the instruction executor connects to the actual system to start executing the instruction. The scheduler synchronously sends the execution action instruction to the operation action recorder to register the instructions to be executed. Go to the fifth step.

[0023] In the fifth step, after a set delay, the scheduler calls the execution status detector to determine the execution status of the instruction and returns the result to the scheduler.

[0024] If the return result is successful, the scheduler will complete the following three actions: 1. Send the result to the operation action recorder, and the operation action recorder changes the state of the corresponding execution instruction; 2. Call the path recorder; 3. Go to step 3 to continue.

[0025] If it fails, the fourth step is called again to try the set number of times. If it still fails after the set number of times, the result is sent to the operation action recorder, which changes the state of the corresponding execution instruction and enters the seventh step at the same time.

[0026] In the sixth step, the scheduler makes a request to the task record and distributor. If the task popped up by the task record and distributor is empty, the whole process ends. Otherwise, the scheduler obtains a new execution instruction. <j>, proceed to the third step.

[0027] In step 7, the scheduler calls the exception handler. If the exception handler is directly handled externally, the process ends. If not handled externally, the "request other paths to try" identifier is returned to the scheduler. Go to step 8.

[0028] In step 8, the scheduler receives the "request other path attempts" identifier from the exception handler, searches for the path record in the path recorder, goes back N steps, and enters step 3 again.

[0029] Furthermore, the path recorder includes three components: page identification model, page id allocation and path recording.

[0030] The page recognition model is trained by a visual classification neural network combined with a ternary loss function.

[0031] The path is recorded as a directed graph, where the vertices are the image IDs and the directed edges are the associated elements that need to be operated from vertex 1 to vertex 2.

[0032] The process is as follows: Get the current page screenshot passed in by the scheduler, and the page recognition model compares the current page screenshot with the image set in the path record to calculate the similarity. If the page recognition model determines that it is a new image, call the page id assigner to assign an id to it. Then call the path record component, add a vertex and edge, and complete a path record.

[0033] Furthermore, the command reasoner is responsible for inferring the historical operation records and the current system GUI page of the current task and scheduler using a cross-modal large model trained by the domain dataset. The command includes not only system operation commands such as click, input, and slide, but also operation commands with the help of external tools such as plugging and unplugging network cables and rotating the screen, as well as logical commands such as process end, check pass end, and check fail end.

[0034] Furthermore, the execution state detector is used to detect whether the action is successfully executed, which can be achieved through OCR target content recognition plus regular expression, or through other verification means.

[0035] Furthermore, the task rewriting and subtask planner can be composed of a large language model and a prompt word template.

[0036] Furthermore, the instruction executor is used to interact with the system, and may be an adb debugging tool, a physical device such as a robotic arm, or other analog signal devices.

[0037] In this embodiment, taking the test scenario as an example, compared with the automated test system, this embodiment avoids the tedious work of writing scripts for test case execution. On a 4-card A6000 server, it can achieve an average of 30 seconds of autonomous reasoning to execute a test case. The original average test workload of 6 man-months for a single system can be reduced to 1 day. If the machine scale is expanded, the automated test time can be further shortened. In terms of accuracy, more than 90% of the cases can be accurately executed. In the field of manual test cases that cannot be solved by the automated test system, it greatly liberates manpower.

[0038] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.< / j>

Claims

1. A complex task process automation system, characterized in that: It includes task recorder and distributor, scheduler, operation action recorder, exception handler, task rewriting and subtask planner, instruction reasoner, instruction executor, execution state detector and path recorder, among which the scheduler is the logic control unit of the whole process, including: The path recorder contains three components: page identification model, page id allocation and path recording; The instruction reasoner is responsible for inferring the current task and the historical operation records passed by the scheduler and the GUI page of the current system using a large cross-modal model trained with the domain dataset; An execution status detector is used to detect whether the action execution is successful; The command executor is used to interact with the system.

2. A complex task process automation system according to claim 1, characterized in that: The page recognition model is trained by a visual classification neural network combined with a ternary loss function. The path is recorded as a directed graph, the vertices of the graph are the image IDs, and the directed edges are the associated elements that need to be operated from vertex 1 to vertex 2.

3. A complex task process automation system according to claim 2, characterized in that: The process includes: Get the current page screenshot passed in by the scheduler. The page recognition model compares the current page screenshot with the picture set in the path record and calculates the similarity. If the page recognition model determines that it is a new picture, the page id allocator is called to assign an id to it, and then the path record component is called to add a vertex and edge to complete a path record.

4. A complex task process automation system according to claim 1, characterized in that: The instructions include not only system operation instructions, but also operation instructions with the help of external tools and logic instructions.

5. A complex task process automation system according to claim 1, characterized in that: The execution state detector is implemented through OCR target content recognition plus regular expression.

6. A complex task process automation system according to claim 1, characterized in that: The task rewriting and subtask planner are composed of a large language model and a prompt word template.

7. A complex task process automation system according to claim 1, characterized in that: The command executor is the adb debugging tool or the physical device of the robotic arm.

8. A method for automating complex task processes implemented by the system according to any one of claims 1 to 7, characterized in that: The following steps are involved: In the first step, the original task is input into the scheduler. The scheduler calls the task rewriting and subtask planner to complete the semantic parsing, information completion and task decomposition of the original task, and returns the result to the scheduler to enter the second step. In the second step, the scheduler receives the return results of the task rewriting and subtask planner, and sends the results to the task recorder and distributor. The task recorder and distributor stores the task queue and returns a subtask to the scheduler, and then enters the third step; In the third step, the scheduler requests relevant historical operation information from the operation action recorder and sends subtasks to the instruction reasoner. The instruction reasoner reasons about the current task and returns an execution action instruction to the scheduler, entering the fourth step; In the fourth step, the scheduler judges the received execution action instruction action. If it is an end instruction, it goes to the sixth step; otherwise, the scheduler sends the received execution action instruction to the instruction executor, and the instruction executor connects to the actual system to start executing the instruction. The scheduler synchronously sends the execution action instruction to the operation action recorder to register the instruction to be executed, and then goes to the fifth step; In the fifth step, after a set delay, the scheduler calls the execution status detector to determine the execution status of the instruction and returns the result to the scheduler; If the return result is successful, the scheduler will complete the following three actions: 1) Send the result to the operation action recorder, and the operation action recorder changes the state of the corresponding execution instruction; 2) Call path recorder; 3) Go to step 3 to continue execution; if it fails, call step 4 again to try the set number of times. If it fails after the set number of times, send the result to the operation action recorder, and the operation action recorder changes the state of the corresponding execution instruction and goes to step 7 at the same time; In the sixth step, the scheduler makes a request to the task record and distributor. If the task popped up by the task record and distributor is empty, the whole process ends. Otherwise, the scheduler obtains a new execution instruction and enters the third step. In the seventh step, the scheduler calls the exception handler. If the exception handler adopts direct external processing, the process ends. If it is not external processing, the "request other path attempts" identifier is returned to the scheduler, and the eighth step is entered; In step 8, the scheduler receives the "request other path attempts" identifier from the exception handler, searches for the path record in the path recorder, goes back N steps, and enters step 3 again.

Images