App automatic execution and test system based on multi-agent collaborative perception decision

Abstract

The invention relates to an app automatic execution and test system based on multi-agent collaborative perception decision, and the system comprises three agents and an upper computer which is used for collecting information and carrying out interactive execution: an execution decision agent receives the output of a planning strategy recommendation agent and a perception agent, serves as input information, generates an instruction to control the upper computer, and carries out the execution of the decision. And interacting with the to-be-tested system to complete the execution task of the to-be-tested system. The three agents are an execution decision agent, a planning strategy recommendation agent and a perception agent, and the three agents jointly form a decision system of the app automatic execution system. After a task target is given, the system can interact with a system to be tested, text recognition, icon recognition, keyboard recognition, layout recognition and anomaly recognition are carried out on a UI page where the system is located through a sensing agent, and meanwhile, system information is stored in a knowledge base after a planning strategy recommendation agent carries out offline exploration; and the execution decision-making agent autonomously reasones out an execution action according to information input of the perception agent and the planning strategy recommendation agent so as to complete execution of a task target. According to the method, multiple agents with different capabilities are combined to work cooperatively while the reasoning capability of the large model is utilized, so that the accuracy and the stability of the model are greatly improved.

Description

Technical Field

[0001] The present invention relates to a testing system, in particular to an APP automatic execution and testing system based on multi-agent collaborative perception and decision-making, belonging to the field of artificial intelligence and computer technology. Background Art

[0002] In today's information age, people's daily lives are inseparable from a variety of mobile app services. For example: Taobao and Pinduoduo for shopping, China Merchants Bank and Huatai Securities for financial management, car apps related to transportation, Ping An Good Car Owner, etc. Each app will be regularly upgraded and new features will be launched from time to time. And each app, especially banking and securities apps involving cash transactions, requires a lot of testing work before each functional module is launched, such as regression testing (regression testing refers to retesting the previous test after the software is modified to ensure the correctness of the modification).

[0003] There are already many automated testing systems (management platforms) on the market to assist the entire testing process and improve testing efficiency.

[0004] Defects and shortcomings of the existing technology:

[0005] 1. When testing a certain function point in the current automated testing system, the test engineer needs to record the pseudo-code test script of the execution process in advance. This recording process is very inefficient, time-consuming and labor-intensive. A mature automated testing engineer can only record 5-10 test scripts a day. However, a small business system app (such as a bank app) usually has hundreds of function points to be tested, and each function point usually has several test cases that require writing test scripts. That is, a small system upgrade requires more than 6 man-months of test script development work;

[0006] 2. With the support of big model technology, some research work on webagent and appagent (i.e. web page execution agent, app intelligent execution) has been launched recently. However, the accuracy and stability of big models are not yet sufficient to meet the needs of industrial production. For example, the accuracy of each reasoning of the big model is 90%, and a functional test generally requires multiple steps of execution. For example, if 8 steps are required, the final accuracy is only 43%. In actual testing, due to the relatively complex operations within the professional business system, gpt4 and gpt4o cannot achieve 90% reasoning accuracy. At present, these explorations are in the experimental stage, and no research has yet applied them to the automated testing of apps. Summary of the invention

[0007] The present invention is aimed at the technical problems existing in the prior art and provides an app automatic execution and testing system based on multi-agent collaborative perception and decision-making. The present invention is committed to providing an app automatic execution agent - Zhice. After a given task goal, the agent can interact with the system to be tested, observe the page it is on, and combine historical experience information to autonomously infer execution actions to complete the execution of the task goal. While relying on the reasoning capability of a large model, this application combines multiple agents with different capabilities to work together, greatly improving the accuracy and stability of the model.

[0008] In order to achieve the above-mentioned purpose, the technical scheme of the present invention is as follows: an app automatic execution and testing system based on multi-agent collaborative perception decision-making, characterized in that the test system includes three agents and a host computer for collecting information and interactive execution. The three agents are: an execution decision agent, a planning strategy recommendation agent and a perception agent. The three agents together constitute the decision system of the app automatic execution system. The execution decision agent receives the output of the planning strategy recommendation agent and the perception agent as input information, generates instructions to control the host computer, and interacts with the system to be tested; the test process is as follows:

[0009] Connect the mobile phone carried by the app to be tested, or the computer / cabinet carried by the system to be tested, to the host computer; the host computer captures a screenshot of the current page and sends it to "an app automatic execution and testing system based on multi-agent collaborative perception and decision-making", and at the same time, the host computer also sends the received target instructions to "an app automatic execution and testing system based on multi-agent collaborative perception and decision-making";

[0010] The perception agent gets the page screenshot and identifies the valid information set Z in the page; the planning strategy recommendation agent gets the target instruction + the output of the perception agent, searches the knowledge base, and outputs the execution path + prompt information based on the information returned by the search results; the target instruction + the output Z of the perception agent + the output of the planning strategy recommendation agent + the historical action are used as the input of the execution decision agent, that is, the prompt word of the execution decision model. The execution decision agent outputs the action to be executed and stores it in the historical action library; the action output by the execution decision agent is verified. If the verification is successful, the verified action is sent to the host computer as the final execution action and handed over to the host computer for execution. If the verification fails, it returns to the planning strategy recommendation agent and executes the action reasoning again until the verification is successful.

[0011] Repeat the above process until the termination condition is reached.

[0012] Among them, the execution decision agent is a large language model, whose input is composed of the output of the planning strategy recommendation agent and the perception agent, and the task instructions received by the host computer, and its output is the specific actions and objects that need to be executed.

[0013] The object is the page element recognized by the perception agent, which contains the text description and coordinate position of the page element, recorded as {"text":"","boundding_box":[]}. The actions include: {slide, click, double-click, complete, enter}. The output actions of the execution decision agent support custom extensions.

[0014] Among them, the planning strategy recommendation agent is responsible for generating reference information of the overall execution plan based on the task and offline exploration information. Its input is the natural language description of the task to be performed, and its output is the reference action path for performing the task. The planning strategy recommendation agent contains a knowledge base. The information in the knowledge base consists of index and metadata (text information). The text in the index is encoded by the embedding model (such as the bge model) and converted into a vector.

[0015] The perceptual agent is a large cross-modal model responsible for understanding page elements. The input is a screenshot of the software page obtained from the automation system. It performs text recognition, icon recognition, keyboard recognition, layout recognition, and exception recognition. The recognition results are recorded as: {text}\{icon}\{layout}\{error};

[0016] Among them, the identified abnormal types include: empty page, pop-up abnormal alarm,

[0017] Among them, text recognition includes text position detection in the page and OCR recognition results.

[0018] Among them, icon recognition includes the detection of the location of common icons in the app page and the corresponding meaning mapping.

[0019] Among them, keyboard recognition includes whether there is a keyboard on the current page and the recognition of elements in the keyboard.

[0020] The layout identification includes the sub-module to which the element belongs and the dependencies between the elements.

[0021] Action verification: Determine whether the action output by the execution decision agent is sliding and completing. If not, compare whether the action object is in the text set and icon set of the perception agent. If not, return verification failure.

[0022] Termination condition: Action = "Complete" or the exception recognition result {error} is not empty.

[0023] Compared with the prior art, the present invention has the following advantages: 1) The present invention provides an app automatic execution agent - Zhice. After the task goal is given, the agent can interact with the system to be tested, observe the page it is on, and combine historical experience information to autonomously infer the execution action and complete the execution of the task goal. This avoids the need to manually write pseudo-code test scripts for execution, thereby greatly reducing the manual workload. While relying on the reasoning ability of a large model, the present invention combines multiple agents with different capabilities to work together, greatly improving the accuracy and stability of the model; 2) The present invention avoids the heavy work of writing test case execution scripts. On a 4-card A6000 server, it can achieve an average of 2 minutes of autonomous reasoning and execution of a test case. The original average test workload of 6 man-months for a single system can be reduced to 2.5 days. If the machine scale is expanded, the automated testing time can be further shortened. In terms of accuracy, more than 75% of the cases can be accurately executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a block diagram of an app automatic execution and testing system based on multi-agent collaborative perception and decision-making. DETAILED DESCRIPTION

[0025] In order to deepen the understanding of the present invention, the present embodiment is described in detail below with reference to the accompanying drawings.

[0026] Example 1: See Figure 1 , an app automatic execution and testing system based on multi-agent collaborative perception decision-making, the test system includes three agents and a host computer for collecting information and interactive execution, the three agents are: an execution decision agent, a planning strategy recommendation agent and a perception agent, the three agents together constitute the decision system of the app automatic execution system, the execution decision agent receives the output of the planning strategy recommendation agent and the perception agent as input information,

[0027] Generate instructions to control the host computer and interact with the system to be tested;

[0028] The testing process is as follows:

[0029] Connect the mobile phone carried by the app to be tested, or the computer / cabinet carried by the system to be tested, to the host computer; the host computer captures a screenshot of the current page and sends it to "an app automatic execution and testing system based on multi-agent collaborative perception and decision-making", and at the same time, the host computer also sends the received target instructions to "an app automatic execution and testing system based on multi-agent collaborative perception and decision-making";

[0030] The perception agent gets the page screenshot and identifies the valid information set Z in the page; the planning strategy recommendation agent gets the target instruction + the output of the perception agent, searches the knowledge base, and outputs the execution path + prompt information based on the information returned by the search results; the target instruction + the output Z of the perception agent + the output of the planning strategy recommendation agent + the historical action are used as the input of the execution decision agent, that is, the prompt word of the execution decision model. The execution decision agent outputs the action to be executed and stores it in the historical action library; the action output by the execution decision agent is verified. If the verification is successful, the verified action is sent to the host computer as the final execution action and handed over to the host computer for execution. If the verification fails, it returns to the planning strategy recommendation agent and executes the action reasoning again until the verification is successful.

[0031] Repeat the above process until the termination condition is reached.

[0032] The object is the page element recognized by the perceptual agent, including the text description and coordinate position of the page element, recorded as {"text":"","boundding_box":[]}. Actions include: slide, click, double-click, complete, input, etc.

[0033] The planning strategy recommendation agent is responsible for generating overall execution plan reference information based on the task and offline exploration information. Its input is the natural language description of the task to be performed, and its output is the reference action path for performing the task. The planning strategy recommendation agent contains a knowledge base. The information in the knowledge base consists of index and metadata. The text in the index is encoded by the embedding model (such as the bge model) and converted into a vector.

[0034] The perceptual agent is a large cross-modal model responsible for understanding page elements. The input is a screenshot of the software page obtained from the automation system. It performs: text recognition, icon recognition, keyboard recognition, layout recognition and exception recognition. The recognition results are recorded as: {text}\{icon}\{layout}\{error},

[0035] The identified anomalies include: empty pages, pop-up window anomaly warnings (errors, NULL, anomalies, etc.), and text recognition includes text position detection in the page and OCR recognition results.

[0036] Icon recognition includes the location detection of common icons in the app page and the corresponding meaning mapping.

[0037] Keyboard recognition includes whether there is a keyboard on the current page and the recognition of elements in the keyboard.

[0038] The layout identification includes the sub-module to which the element belongs and the dependencies between the elements.

[0039] Flowchart description: Connect the **Bank app to the host computer - Jieke automated testing platform; the host computer captures a screenshot of the current page and sends it to "an app automatic execution and testing system based on multi-agent collaborative perception decision-making", and at the same time, the host computer sends the test steps in the test case to the agent for execution one by one; the perception agent gets the page screenshot and identifies the valid information set Z in the page; the planning strategy recommendation agent gets the test intention and the test steps of the i-th step + the output of the perception agent, searches the knowledge base, and outputs the execution path + prompt information based on the information returned by the search results; the target instruction + the output Z of the perception agent + the output of the planning strategy recommendation agent + the historical action are used as the input of the execution decision agent, that is, the prompt word of the execution decision model. The execution decision agent outputs the action to be executed and stores it in the historical action library; the action output by the execution decision agent is verified. If the verification is successful, the verified action is sent to the host computer as the final execution action and handed over to the host computer for execution. If the verification fails, it returns to the planning strategy recommendation agent and executes the action reasoning again until the verification is successful. The agent and the host computer continue to interact and reason until the agent infers that the current step is completed; the host computer sends the i+1th execution step, and the agent interactively executes it; until all execution steps are executed, the execution result is returned according to the final page status: passed or failed.

[0040] Example 2: Business Talk Mode

[0041] An app automatic execution and testing system based on multi-agent collaborative perception and decision-making, the system includes three agents: execution decision-making agent, planning strategy recommendation agent and perception agent.

[0042] The executive decision-making agent is a large language model that is responsible for obtaining all valid information provided by Wuneng and Wujing, reasoning according to the task, and outputting the specific actions and objects that need to be executed.

[0043] The object is the page element recognized by the perceptual agent, including the text description and coordinate position of the page element, recorded as {"text":"","boundding_box":[]}. Actions include: slide, click, double-click, complete, input, etc.

[0044] The planning strategy recommendation agent is responsible for generating overall execution plan reference information based on the task and offline exploration information. Its input is the natural language description of the task to be performed, and its output is the reference action path for performing the task. The planning strategy recommendation agent contains a knowledge base. The information in the knowledge base consists of index and metadata. The text in the index is encoded by the embedding model (such as the bge model) and converted into a vector.

[0045] The perceptual agent is a large cross-modal model responsible for understanding page elements. The input is a screenshot of the software page obtained from the automation system. It performs: text recognition, icon recognition, keyboard recognition, layout recognition and exception recognition. The recognition results are recorded as: {text}\{icon}\{layout}\{error}

[0046] The identified anomalies include: empty pages, pop-up window anomaly warnings (errors, NULL, anomalies, etc.), and text recognition includes text position detection in the page and OCR recognition results.

[0047] Icon recognition includes the location detection of common icons in the app page and the corresponding meaning mapping.

[0048] Keyboard recognition includes whether there is a keyboard on the current page and the recognition of elements in the keyboard.

[0049] The layout identification includes the sub-module to which the element belongs and the dependencies between the elements.

[0050] Flowchart description: Connect **Bank app to the host computer - adb (Android Debug Bridge); the host computer grabs the screenshot of the current page and sends it to "A system for automatic execution and testing of apps based on multi-agent collaborative perception decision-making". At the same time, the host computer sends the received target instruction (transfer 50 to Wang Nan, transaction password 147458) to "A system for automatic execution and testing of apps based on multi-agent collaborative perception decision-making"; the perception agent gets the page screenshot and identifies the valid information set Z in the page; the planning strategy recommendation agent gets the target instruction + the output of the perception agent, searches the knowledge base, and outputs the execution path + prompt information according to the information returned by the search result; the target instruction + the output Z of the perception agent + the output of the planning strategy recommendation agent + the historical action are used as the input of the execution decision agent, that is, the prompt word of the execution decision model. The execution decision agent outputs the action to be executed and stores it in the historical action library; the action output by the execution decision agent is verified. If the verification is successful, the verified action is sent to the host computer as the final execution action and handed over to the host computer for execution. If the verification fails, it returns to the planning strategy recommendation agent and executes the action reasoning again until the verification is successful. Repeat the above process until the termination condition is reached.

[0051] It should be noted that the above embodiments are not intended to limit the protection scope of the present invention, and equivalent changes or substitutions made on the basis of the above technical solutions all fall within the protection scope of the claims of the present invention.

Claims

1. An app automatic execution and testing system based on multi-agent collaborative perception decision-making, characterized in that: The test system includes three agents and a host computer for collecting information and interactive execution. The three agents are: an execution decision agent, a planning strategy recommendation agent and a perception agent. The three agents together constitute the decision system of the app automatic execution system. The execution decision agent receives the output of the planning strategy recommendation agent and the perception agent as input information, generates instructions to control the host computer, and interacts with the system to be tested. The testing process is as follows: Connect the mobile phone carried by the app to be tested, or the computer / cabinet carried by the system to be tested, to the host computer; the host computer captures the screenshot of the current page and sends it to "an app automatic execution and testing system based on multi-agent collaborative perception and decision-making", and at the same time, the host computer also sends the received target instruction to "an app automatic execution and testing system based on multi-agent collaborative perception and decision-making"; The perception agent gets the page screenshot and identifies the valid information set Z in the page; the planning strategy recommendation agent gets the target instruction + the output of the perception agent, searches the knowledge base, and outputs the execution path + prompt information based on the information returned by the search results; the target instruction + the output Z of the perception agent + the output of the planning strategy recommendation agent + the historical action are used as the input of the execution decision agent, that is, the prompt word of the execution decision model. The execution decision agent outputs the action to be executed and stores it in the historical action library; the action output by the execution decision agent is verified. If the verification is successful, the verified action is sent to the host computer as the final execution action and handed over to the host computer for execution. If the verification fails, it returns to the planning strategy recommendation agent and executes the action reasoning again until the verification is successful. Repeat the above process until the termination condition is reached.

2. The app automatic execution and testing system based on multi-agent collaborative perception decision-making according to claim 1 is characterized in that: The execution decision agent is a large language model. Its input consists of the output of the planning strategy recommendation agent and the perception agent, and the task instructions received by the host computer. Its output is the specific actions and objects that need to be executed. The object is the page element recognized by the perception agent, which contains the text description and coordinate position of the page element, denoted as {"text":"","boundding_box":[]}. The actions include: {slide, click, double-click, complete, enter}. The output actions of the decision-making agent support custom extensions.

3. The app automatic execution and testing system based on multi-agent collaborative perception decision-making according to claim 2 is characterized in that: The planning strategy recommendation agent is responsible for generating reference information for the overall execution plan based on the task and offline exploration information. Its input is the natural language description of the task to be performed, and its output is the reference action path for performing the task. The planning strategy recommendation agent contains a knowledge base. The information in the knowledge base consists of an index and metadata (text information). The text in the index is encoded by an embedding model (such as the bge model) and converted into a vector.

4. The app automatic execution and testing system based on multi-agent collaborative perception decision-making according to claim 3 is characterized in that: The perceptual agent is a large cross-modal model responsible for understanding page elements. The input is a screenshot of the software page obtained from the automation system. It performs: text recognition, icon recognition, keyboard recognition, layout recognition and exception recognition. The recognition results are recorded as: {text}\{icon}\{layout}\{error}; Among them, the identified abnormal types include: empty page, pop-up abnormal alarm, Among them, text recognition includes text position detection in the page and OCR recognition results. Among them, icon recognition includes the detection of the location of common icons in the app page and the corresponding meaning mapping. Among them, keyboard recognition includes whether there is a keyboard on the current page and the recognition of elements in the keyboard. The layout identification includes the sub-module to which the element belongs and the dependencies between the elements.

5. The app automatic execution and testing system based on multi-agent collaborative perception decision-making according to claim 4 is characterized in that: Action verification: Determine whether the action output by the execution decision agent is sliding and completing. If not, compare whether the action object is in the text set and icon set of the perception agent. If not, return verification failure.

6. The app automatic execution and testing system based on multi-agent collaborative perception decision-making according to claim 5 is characterized in that: Termination condition: Action = "Complete" or the exception recognition result {error} is not empty.

Images