Multi-agent-based full-process automatic testing method, equipment and medium

Abstract

The invention provides a full-process automatic testing method and device based on multiple agents and a medium, and belongs to the technical field of software testing. The full-process automatic testing method comprises the steps that a multi-agent cooperation framework is built in a full-process mode for a testing link; the test link whole process comprises a plurality of test links, and the multi-agent collaboration framework comprises a plurality of agents and a framework context information storage module; a user software demand is received, an intelligent agent corresponding to a current test link is used for executing the current test link, and a test result is uploaded to the framework context information storage module; according to the test result stored in the framework context information storage module, the intelligent agent corresponding to the next test link in the multiple intelligent agents is triggered to automatically execute the next test link, the test result is uploaded to the framework context information storage module, and the step is iteratively executed until a final test report corresponding to the user software requirement is obtained. The problem that the prior art is lack of full-process series connection capability can be solved.

Description

Technical Field

[0001] This application belongs to the field of software testing technology, specifically relating to a multi-agent-based full-process automated testing method, equipment, and medium. Background Technology

[0002] With the rise of large-scale AI model technology, more and more companies are beginning to explore the application of large-scale models in the field of software testing. However, the current application of large-scale models in software testing still has the following limitations: First, the process is fragmented: most existing technology applications or tools focus on a single testing link in the software testing process, such as developing test case generation, test script execution and test result analysis separately.

[0003] Specifically, regarding test case generation, existing technologies (such as patent CN202411663771.9, which provides a test case generation method) only focus on generating unit test cases based on code, and evaluate test cases through pre-defined quality models, completely neglecting the subsequent role and optimization of test cases. Regarding the test execution process, various testing tools exhibit diverse characteristics, requiring different skill levels from personnel for tool use and result analysis. For example, Selenium is used for web testing, and JMeter for stress testing, lacking cross-tool collaborative scheduling.

[0004] Because existing technologies revolve around a single testing stage in the software testing process, the related intelligent agent architectures are mostly localized innovations and lack the ability to connect the entire process. This leads to automation gaps and data silos in various stages of software testing. Inevitably, the connection between each stage requires human intervention, resulting in a fragmented testing process and low efficiency. Summary of the Invention

[0005] This application aims to provide a multi-agent-based, fully automated testing solution. It addresses the lack of end-to-end connectivity in existing technologies, which leads to automation gaps and data silos in various stages of software testing. Inevitably, manual intervention is required to connect each stage, resulting in a fragmented testing process and low efficiency.

[0006] According to a first aspect of this application, this application provides a multi-agent-based fully automated testing method, including:

[0007] For the entire testing process involved in software project testing, a corresponding multi-agent collaboration framework is built; the entire testing process includes multiple testing stages, and the multi-agent collaboration framework includes multiple agents and a framework context information storage module.

[0008] Receive user software requirements, use the agent corresponding to the current test phase from among multiple agents, execute the current test phase according to the user software requirements, and upload the test results of the current test phase to the framework context information storage module;

[0009] Based on the test results stored in the framework context information storage module, the agent corresponding to the next test stage among multiple agents is triggered to automatically execute the next test stage, and the obtained test results are uploaded to the framework context information storage module. This step is iteratively executed according to the flow order of the entire test stage until the final test report corresponding to the user's software requirements is obtained.

[0010] Preferably, the above-mentioned fully automated testing method, for the entire testing process involved in software project testing, includes the following steps for building a corresponding multi-agent collaborative framework:

[0011] Following the process sequence, the entire testing process is set up to include the requirements analysis stage, test case generation stage, termination threshold judgment stage, test case execution stage, result analysis stage, defect registration stage, and test report generation stage;

[0012] Set up a requirement analysis intelligent agent for the requirement analysis stage, a test case generation intelligent agent for the test case generation stage, a test execution intelligent agent for the test case execution stage, a result analysis intelligent agent for the result analysis stage, a defect registration intelligent agent for the defect registration stage, and a test report generation intelligent agent for the test report stage.

[0013] In the termination threshold determination phase, code coverage threshold and requirement coverage threshold are used to design test termination indicators for the termination threshold determination phase; when the test termination indicators are met, the code coverage data and requirement coverage data obtained from the test are submitted as test results to the framework's upper and lower storage modules.

[0014] Preferably, the above-mentioned fully automated testing method includes the following steps: receiving user software requirements, using the agent corresponding to the current testing stage from among multiple agents to execute the current testing stage according to the user software requirements, and uploading the test results of the current testing stage to the framework context information storage module.

[0015] When the requirements analysis agent corresponding to the requirements analysis phase listens to the user's software requirements input, it receives the user's software requirements.

[0016] The requirements analysis agent calls the large model to parse user software requirements according to pre-organized prompts, and extracts a list of requirements and corresponding verification points.

[0017] The requirement list and verification points are submitted as test results to the framework context information storage module.

[0018] Preferably, the above-mentioned fully automated testing method, based on the test results stored in the framework context information storage module, triggers the agent corresponding to the next test stage among multiple agents to automatically execute the next test stage, uploads the obtained test results to the framework context information storage module, and iteratively executes this step according to the flow order of the entire testing process until the final test report corresponding to the user's software requirements is obtained. This includes the following steps:

[0019] In the test case generation phase, the test case generation agent extracts the requirement list and verification points from the framework context information storage module; according to the pre-organized prompts, it calls the large model to generate test cases for each requirement in the requirement list and the corresponding verification points; and submits the generated test cases to the framework context information storage module.

[0020] During the test case execution phase, the test case execution agent retrieves test cases from the framework context information storage module; it calls the corresponding automated testing tools to execute the test cases according to different test case types, uses code coverage analysis tools to collect program code coverage information after the test case execution, and submits the program code coverage information to the framework context information storage module.

[0021] In the termination threshold determination stage, program code coverage information is extracted from the framework context information storage module to determine whether the program code coverage information has reached the test termination index. When the program code coverage information reaches the test termination index, the test case execution results are uploaded to the framework context information storage module. The test termination index is designed based on the code coverage threshold and the requirement coverage threshold. The test case execution results include test case execution status, code coverage data, and requirement coverage data.

[0022] In the results analysis phase, the results analysis agent extracts the test case execution results from the framework context information storage module, uses the test case execution results to test the execution log, obtains the test execution results, and submits them to the framework context information storage module.

[0023] In the test report generation phase, the test report agent extracts the requirement list, test cases, code coverage information, test case execution results, and test execution results from the framework context information storage module.

[0024] Following the test report template, use the requirement list, test cases, code coverage information, test case execution results, and test execution results to generate the final test report.

[0025] Preferably, the above-mentioned fully automated testing method, based on the test results stored in the framework context information storage module, triggers the agent corresponding to the next test stage among multiple agents to automatically execute the next test stage, uploads the obtained test results to the framework context information storage module, and iteratively executes this step according to the flow order of the entire testing process until the final test report corresponding to the user's software requirements is obtained. This includes the following steps:

[0026] In the defect registration stage, which is between the result analysis stage and the test report generation stage, a defect registration agent is used to extract the test execution results from the framework context information storage module.

[0027] Use the test execution results to organize program defect information, call the defect management system interface to register program defect information, and obtain detailed defect information;

[0028] Submit the defect details to the framework context information storage module.

[0029] Preferably, the above-mentioned fully automated testing method includes the following steps: executing test cases by calling corresponding automated testing tools according to different test case types, and using code coverage analysis tools to statistically analyze the program code coverage information after test case execution.

[0030] Extract the test case list from the framework context information storage module;

[0031] Analyze the test case type corresponding to each test case in the test case list;

[0032] According to different test case types, UI automation testing tools, interface automation testing tools, performance automation testing tools, and security automation testing tools are invoked respectively.

[0033] Use UI automation testing tools, API automation testing tools, performance automation testing tools, and security automation testing tools to test test cases of the corresponding test case types.

[0034] Preferably, in the above-mentioned fully automated testing method, the step of retrieving program code coverage information from the framework context information storage module and determining whether the program code coverage information has reached the test termination indicator in the termination threshold judgment stage includes:

[0035] In the termination threshold determination stage, according to the user's software requirements, the line coverage threshold and branch coverage threshold under the code coverage threshold are set respectively to generate multi-level requirement coverage thresholds under requirement coverage.

[0036] From the program code coverage information, extract line coverage, branch coverage, and multi-level requirement coverage corresponding to the multi-level requirement coverage threshold;

[0037] Determine whether the row coverage rate is greater than or equal to the row coverage rate threshold, whether the branch coverage rate is greater than or equal to the branch coverage rate, and whether the multi-level demand coverage rate is greater than or equal to the multi-level demand coverage rate.

[0038] If the row coverage rate is determined to be greater than or equal to the row coverage rate threshold, the branch coverage rate is greater than or equal to the branch coverage rate, and the multi-level requirement coverage rate is greater than or equal to the multi-level requirement coverage rate, then the test case execution results are uploaded to the framework context information storage module and the result analysis phase begins.

[0039] Preferably, in the above-mentioned fully automated testing method, after determining whether the row coverage rate is greater than or equal to the row coverage rate threshold, whether the branch coverage rate is greater than or equal to the branch coverage rate, and whether the multi-level requirement coverage rate is greater than or equal to the multi-level requirement coverage rate, the method further includes:

[0040] If the row coverage rate is determined to be less than the row coverage rate threshold, the branch coverage rate is less than the branch coverage rate, and the multi-level requirement coverage rate is less than the multi-level requirement coverage rate, the test case execution result will be uploaded to the framework context information storage module and the requirement analysis agent corresponding to the requirement analysis stage will be returned.

[0041] Using the requirements analysis agent, retrieve the latest test cases and program code coverage information from the framework context information storage module, organize supplementary test case prompts according to the test cases and program code coverage information, and upload the supplementary test case prompts to the framework context information storage module.

[0042] Use test cases to generate intelligent agents, use supplementary test case prompts to call the large model, generate supplementary test cases for the uncovered parts, and upload them to the framework context information storage module.

[0043] The test case execution agent calls supplementary test cases from the framework context information storage module, and executes the supplementary test cases according to the corresponding automated testing tools based on the test case type to obtain program code coverage information;

[0044] Repeat the above steps until the program code coverage information reaches the test termination index, then proceed to the results analysis stage.

[0045] According to a second aspect of this application, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-agent-based full-process automated testing method provided by any of the above technical solutions.

[0046] According to a third aspect of this application, this application also provides a computer storage medium storing a computer program thereon, which, when executed, implements the multi-agent-based full-process automated testing method provided by any of the above technical solutions.

[0047] The technical solution of this application has at least the following technical effects:

[0048] The multi-agent-based full-process automated testing solution provided in this application establishes a corresponding multi-agent collaborative framework for the entire testing process required for software project testing. This full-process testing includes multiple testing stages, and the multi-agent collaborative framework includes multiple agents corresponding to each testing stage, as well as a framework context information storage module. When a user's software requirement is received, the agent corresponding to the current testing stage can be used to execute the current testing stage according to the user's software requirement, and the test results are uploaded to the framework context information storage module. Upon receiving the test results, the framework context information storage module can trigger the agent corresponding to the next testing stage to automatically execute the next testing stage and upload the obtained test results to the framework context information storage module. This triggers the agent corresponding to the next testing stage in the full-process testing to repeat the above steps until the final test report corresponding to the user's software requirement is obtained. Through this method, the entire testing process involved in software project testing can be linked using multiple agents, and the test results from the framework context information storage module in the multi-agent collaborative framework can trigger the agent for the next testing stage, thereby achieving fully automated execution of the software testing process and realizing an end-to-end closed-loop testing process. The technical solution proposed in this application requires no manual intervention throughout the entire process, which can solve the problems of process fragmentation and unclear test termination goals in the existing technology, thereby significantly improving the automation level and efficiency of software project testing processes. Attached Figure Description

[0049] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0050] Figure 1 A flowchart illustrating the first fully automated testing method based on multi-agent systems provided in this application embodiment;

[0051] Figure 2 for Figure 1 The illustrated embodiment provides a flowchart of a method for building a multi-agent collaborative framework;

[0052] Figure 3 for Figure 1The illustrated embodiment provides a flowchart of a method for using an intelligent agent to execute the current test phase;

[0053] Figure 4 for Figure 1 The illustrated embodiment provides a flowchart of a method for using an intelligent agent to perform the next testing phase;

[0054] Figure 5 A flowchart illustrating the first fully automated testing method based on multi-agent systems provided in this application embodiment;

[0055] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0056] To more clearly illustrate the overall concept of this application, a detailed explanation is provided below with reference to the accompanying drawings.

[0057] Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below. It should be noted that, unless otherwise specified, the embodiments of this application and the features thereof can be combined with each other.

[0058] In this application, unless otherwise expressly specified and limited, the terms "above" and "below" the second feature can refer to direct contact between the first and second features, or indirect contact between the first and second features through an intermediate medium. In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0059] The existing technology has the following drawbacks:

[0060] Existing software testing solutions suffer from fragmented processes and unclear test termination objectives. Specifically, regarding process fragmentation, focusing only on a single step in the testing process (such as test case generation, test script execution, or test result analysis) lacks end-to-end connectivity, still requiring manual intervention for each step, resulting in a disjointed and inefficient testing process. Regarding unclear test termination objectives: test termination conditions rely on human experience; the coverage and usability of test cases generated using large models still require manual verification; and in cases of low coverage, manual interaction with the large model is necessary to supplement new test cases.

[0061] To solve the above problems, see [link to relevant documentation]. Figure 1 , Figure 1 A flowchart illustrating the first fully automated testing method based on multi-agent systems provided in this application is shown below. Figure 1 As shown, this application provides a fully automated testing method based on multi-agent systems, including:

[0062] S110: For the entire testing process involved in software project testing, build a corresponding multi-agent collaboration framework; the entire testing process includes multiple testing stages, and the multi-agent collaboration framework includes multiple agents and a framework context information storage module.

[0063] S120: Receives the user's software requirements, uses the agent corresponding to the current test phase from among multiple agents, executes the current test phase according to the user's software requirements, and uploads the test results of the current test phase to the framework context information storage module.

[0064] S130: Based on the test results stored in the framework context information storage module, trigger the agent corresponding to the next test stage among multiple agents to automatically execute the next test stage, upload the obtained test results to the framework context information storage module, and iteratively execute this step according to the flow order of the entire test stage until the final test report corresponding to the user's software requirements is obtained.

[0065] The multi-agent-based full-process automated testing method provided in this application establishes a corresponding multi-agent collaborative framework for the entire testing process required for software project testing. This full-process testing includes multiple testing stages, and the multi-agent collaborative framework includes multiple agents corresponding to each testing stage and a framework context information storage module. When a user's software requirement is received, the agent corresponding to the current testing stage can be used to execute the current testing stage according to the user's software requirement and upload the test results to the framework context information storage module. Upon receiving the test results, the framework context information storage module can trigger the agent corresponding to the next testing stage to automatically execute the next testing stage and upload the obtained test results to the framework context information storage module. This triggers the agent corresponding to the next testing stage in the full-process testing to repeat the above steps until a final test report corresponding to the user's software requirement is obtained. This method uses multiple agents to connect the entire testing process involved in software project testing, and triggers the agent for the next testing stage through the test results in the framework context information storage module of the multi-agent collaborative framework, thereby achieving fully automated execution of the software testing process and realizing an end-to-end closed-loop testing process. The technical solution proposed in this application requires no manual intervention throughout the entire process, which can solve the problems of process fragmentation and unclear test termination goals in the existing technology, thereby significantly improving the automation level and efficiency of software project testing processes.

[0066] Specifically, as a preferred embodiment, such as Figure 2 As shown, step S110 above, which involves building a corresponding multi-agent collaboration framework for the entire testing process of a software project, includes:

[0067] S111: Following the process sequence, the entire testing process is set up to include the requirements analysis stage, test case generation stage, test case execution stage, termination threshold judgment stage, result analysis stage, defect registration stage, and test report generation stage;

[0068] S112: Set up a requirement analysis intelligent agent for the requirement analysis stage, a test case generation intelligent agent for the test case generation stage, a test execution intelligent agent for the test case execution stage, a result analysis intelligent agent for the result analysis stage, a defect registration intelligent agent for the defect registration stage, and a test report generation intelligent agent for the test report stage.

[0069] S113: In the termination threshold judgment stage, code coverage threshold and requirement coverage threshold are used to design test termination indicators for the termination threshold judgment stage; when the test termination indicators are met, the code coverage data and requirement coverage data obtained from the test are submitted as test results to the framework upper and lower storage modules.

[0070] The technical solution provided in this application, through a multi-agent collaborative framework, seamlessly connects the requirement analysis, test case generation, test execution, termination threshold judgment, result analysis, and test report generation stages. It introduces code coverage and requirement coverage thresholds as quantitative standards for test termination in the termination threshold judgment stage, enabling an end-to-end closed-loop testing process. Furthermore, in the result analysis stage, test case generation is further optimized based on the test results. Correspondingly, agents invoke various automated testing tools for automatic execution, achieving automatic termination upon reaching the test threshold target. Through the advancement of multiple agents and the triggering of the framework context information storage module, the entire testing process requires no manual intervention, thereby significantly improving the automation level and efficiency of the process.

[0071] Specifically, as a preferred embodiment, such as Figure 3 As shown, in the above-described fully automated testing method, step S120—receiving the input user software requirements, using the agent corresponding to the current testing stage from among multiple agents, executing the current testing stage according to the user software requirements, and uploading the test results of the current testing stage to the framework context information storage module—includes the following steps:

[0072] S121: When the requirements analysis agent corresponding to the requirements analysis stage detects the user's software requirements input, it receives the user's software requirements.

[0073] S122: The requirements analysis agent calls the large model to parse user software requirements according to pre-organized prompts, and extracts the requirement list and corresponding verification points.

[0074] S123: Submit the requirement list and verification points as test results to the framework context information storage module.

[0075] After the test user inputs their software requirements into the entire testing process, the requirement analysis agent of the aforementioned multi-agent collaborative framework receives these requirements, organizes prompts, calls the large model to parse the requirements, and extracts a list of requirements and corresponding verification points. Simultaneously, the requirement list and verification points output by the large model are submitted to the framework context information storage module. This method enables rapid acquisition and parsing of user software requirements, and the submission of test results for storage in the framework context information storage module. This triggers the agent corresponding to the next testing phase, automatically executing the next testing phase, thereby achieving automated execution of the entire testing process.

[0076] In addition, as a preferred embodiment, such as Figure 4 As shown, Figure 1S130: Based on the test results stored in the framework context information storage module, trigger the agent corresponding to the next test stage among multiple agents to automatically execute the next test stage, upload the obtained test results to the framework context information storage module, and iteratively execute this step according to the entire test stage flow sequence until the final test report corresponding to the user software requirements is obtained. Specifically, this includes:

[0077] S131; In the test case generation phase, the test case generation agent extracts the requirement list and verification points from the framework context information storage module; according to pre-organized prompts, it calls the large model to generate test cases for each requirement in the requirement list and its corresponding verification point; and submits the generated test cases to the framework context information storage module. The test cases for the requirements and verification points can be uploaded to the framework context information storage module in the form of a test case list.

[0078] In this embodiment, after the requirements analysis task is completed, the test case generation agent is initiated. This agent retrieves the requirement list entries and corresponding verification points from the framework context information storage module, and generates test cases for each requirement and corresponding verification point by calling the large model according to pre-organized prompts within the system. The prompts further specify the test case design methods used, such as equivalence class partitioning and boundary value analysis, and ensure that the test case output format conforms to the latest national standard GB / T 25000.51-2016. After the test cases are generated, the (test case list) is submitted to the framework context information storage module.

[0079] S132: In the test case execution phase, the test case execution agent extracts test cases from the framework context information storage module; calls the corresponding automated testing tool to execute the test cases according to different test case types; uses the code coverage analysis tool to collect program code coverage information after the test case execution; and submits the program code coverage information to the framework context information storage module.

[0080] Specifically, the steps include: extracting the test case list from the framework context information storage module; analyzing the test case type corresponding to each test case in the test case list; calling the UI automation testing tool, interface automation testing tool, performance automation testing tool, and security automation testing tool according to different test case types; and using the UI automation testing tool, interface automation testing tool, performance automation testing tool, and security automation testing tool to test the test cases of the corresponding test case types.

[0081] In this embodiment, after test cases are generated, the test execution agent is invoked. The agent retrieves the test case list from the framework context information storage module and executes it. Depending on the test case type, the agent calls various testing tools, such as UI automation (e.g., Selenium), API automation (e.g., Postman), performance automation (e.g., Jmeter), and security automation (e.g., AppScan). Furthermore, this test execution agent supports user-defined extensions, dynamically adding other types of testing tools based on the MCP protocol to meet personalized test execution needs in different scenarios. During test execution, code coverage analysis tools (e.g., Jacoco) are used to statistically analyze the program code coverage after test case execution. Simultaneously, the execution results are submitted to the framework context information storage module.

[0082] S133: In the termination threshold judgment stage, extract program code coverage information from the framework context information storage module and determine whether the program code coverage information has reached the test termination index; when the program code coverage information has reached the test termination index, upload the test case execution results to the framework context information storage module; wherein, the test termination index is designed based on the code coverage threshold and the requirement coverage threshold, and the test case execution results include test case execution status, code coverage data and requirement coverage data.

[0083] Specifically, in a preferred embodiment, the step of retrieving program code coverage information from the framework context information storage module and determining whether the program code coverage information has reached the test termination indicator in the termination threshold determination stage includes:

[0084] In the termination threshold determination stage, according to the user's software requirements, the line coverage threshold and branch coverage threshold under the code coverage threshold are set respectively to generate multi-level requirement coverage thresholds under requirement coverage.

[0085] From the program code coverage information, extract the line coverage, branch coverage, and multi-level requirement coverage corresponding to the multi-level requirement coverage threshold.

[0086] Determine whether the row coverage rate is greater than or equal to the row coverage rate threshold, the branch coverage rate is greater than or equal to the branch coverage rate, and the multi-level demand coverage rate is greater than or equal to the multi-level demand coverage rate.

[0087] If the row coverage rate is determined to be greater than or equal to the row coverage rate threshold, the branch coverage rate is greater than or equal to the branch coverage rate, and the multi-level requirement coverage rate is greater than or equal to the multi-level requirement coverage rate, then the test case execution results are uploaded to the framework context information storage module and the result analysis phase begins.

[0088] In addition, if it is determined that the row coverage is less than the row coverage threshold, the branch coverage is less than the branch coverage, and the multi-level requirement coverage is less than the multi-level requirement coverage, the test case execution result will be uploaded to the framework context information storage module, and the requirement analysis agent corresponding to the requirement analysis stage will be returned.

[0089] Using the requirements analysis agent, retrieve the latest test cases and program code coverage information from the framework context information storage module, organize supplementary test case prompts according to the test cases and program code coverage information, and upload the supplementary test case prompts to the framework context information storage module.

[0090] Use test cases to generate intelligent agents, use supplementary test case prompts to call the large model, generate supplementary test cases for the uncovered parts, and upload them to the framework context information storage module.

[0091] The test case execution agent calls supplementary test cases from the framework context information storage module, executes the supplementary test cases according to the corresponding automated testing tools based on the test case type, and obtains program code coverage information.

[0092] Repeat the above steps until the program code coverage information reaches the test termination index, then proceed to the results analysis stage.

[0093] In this embodiment, after the test case execution is complete, a termination threshold judgment stage is entered. This stage uses clear metrics to determine the quality level of automated testing, avoiding insufficient coverage in a single test round. Specifically, the system provides default code coverage and requirement coverage thresholds, and further provides line coverage and branch coverage configurations for code coverage, and first-level, second-level, and third-level requirement coverage configurations for requirement coverage. Based on the industry standard ISTQB, the default values ​​for code coverage are: line coverage ≥ 80%, branch coverage ≥ 75%, and first-level requirement coverage 100%, second-level requirement coverage 95%, and third-level requirement coverage 90%. User-defined thresholds are supported, as well as the extension of other threshold metrics based on the MCP protocol.

[0094] When the result of this execution falls below the set threshold, the system submits the test case execution status, code coverage, and requirement coverage data to the framework context storage module and feeds it back to the requirement analysis module for re-analysis. Upon receiving the feedback, the system retrieves the latest data coverage, requirement specifications (including test execution results and coverage information), and already generated test case entries from the framework context storage module. It then reorganizes targeted supplementary test case prompts, calls the large model to regenerate test cases for the uncovered portions, and notifies the test case execution module to execute them again. If the threshold standard (i.e., the aforementioned test termination criterion) is reached after the re-execution, the system proceeds to the next step of result analysis; otherwise, the requirement analysis, supplementary test cases, and test execution steps are repeated.

[0095] To avoid potential infinite loops, the system supports 5 repetitions by default, but users can configure the maximum number of repetitions based on their needs. Once the maximum number of repetitions is exceeded, the system will stop the process and send a notification to the user, requiring the user to intervene and determine whether to re-optimize the loop or proceed with the next steps.

[0096] Additionally, in the result analysis phase, the result analysis agent extracts the test case execution results from the framework context information storage module, uses the test case execution results to test the execution log, obtains the test execution results, and submits them to the framework context information storage module.

[0097] The results analysis agent retrieves test case execution results from the framework context storage module, using test execution logs to obtain the test execution results. It analyzes scenarios where test case execution fails to determine whether the issue stems from a bug in the program itself, or from other problems such as the test environment or the test case itself, and then submits the test execution results to the framework context storage module.

[0098] S135: In the defect registration process, the defect registration agent is used to extract the test execution results from the framework context information storage module; the test execution results are used to organize the program defect information, and the defect management system interface is called to register the program defect information to obtain the defect details; the defect details are submitted to the framework context information storage module.

[0099] After the test execution results are analyzed, the defect registration stage begins. The defect registration agent retrieves the test execution results from the framework context storage module and further organizes the program bug information, clarifying the problem level, error type, severity, and responsible person. After the information processing is completed, the defect management system interface is called to automatically complete the defect registration and submit the final defect details to the framework context storage module.

[0100] S136: In the test report generation phase, the test report agent extracts the requirement list, test cases, code coverage information, test case execution results, and test execution results from the framework context information storage module; and generates the final test report according to the test report template, using the requirement list, test cases, code coverage information, test case execution results, and test execution results.

[0101] After defect registration is completed, the test report generation phase begins. The test report generation agent retrieves the requirement list, test case list, test execution results, and final defect details from the framework context storage module, organizes them according to the test report template, and generates the final test report. Once generated, the customer is notified to view or download the report, completing the entire automated testing task. The aforementioned final test report includes the requirement list, test case list, test case execution results, test execution results, and final defect details retrieved from the framework context information storage module.

[0102] join Figure 5 , Figure 5 This is a flowchart illustrating the second fully automated testing method based on multi-agent systems provided in this application. Figure 5 As shown, this fully automated testing method includes:

[0103] S201: Requirements Input.

[0104] S202: Requirements Analysis. After the user inputs their requirements, the requirements analysis agent receives the requirements, organizes the prompts, calls the large model to parse the requirements, and extracts a list of requirements and corresponding verification points. Simultaneously, the requirement list and verification points output by the large model are submitted to the framework context information storage module.

[0105] S203: Test Case Generation. After the requirements analysis task is completed, the test case generation agent is activated. This agent retrieves the requirement list entries and corresponding verification points from the framework context information storage module, organizes prompts, and calls the large model to generate test cases for each requirement and its corresponding verification point. The prompts further specify the use of equivalence class partitioning, boundary value analysis, and other test case design methods, and ensure that the test case output format conforms to the latest national standard GB / T 25000.51-2016. After the test cases are generated, the (test case list) is submitted to the framework context information storage module.

[0106] S204: Test Execution. After the test case generation task is completed, the test execution agent enters the test execution phase. The test execution agent retrieves the test case list from step 203 from the framework context information storage module and executes it. Depending on the test case type, the agent invokes testing tools such as UI automation (e.g., Selenium), API automation (e.g., Postman), performance automation (e.g., Jmeter), and security automation (e.g., AppScan). The test agent supports user-defined extensions, dynamically adding other types of testing tools based on the MCP protocol to meet personalized test execution needs in different scenarios. During test execution, code coverage analysis tools (e.g., Jacoco) are used to statistically analyze the program code coverage after test case execution. Simultaneously, the execution results are submitted to the framework's context information storage module.

[0107] S205: Termination Threshold Determination. After test case execution is complete, the termination threshold determination phase begins. This phase uses clear metrics to assess the quality level of automated testing, preventing insufficient coverage in a single test round. The system provides default code coverage and requirement coverage thresholds, and further offers configuration options for line coverage and branch coverage under code coverage, and configuration options for first-level, second-level, and third-level requirement coverage under requirement coverage.

[0108] S206: Result Analysis. After the termination threshold condition is met, the result analysis phase begins. The result analysis agent retrieves the test case execution results, test execution logs, and test execution results from the framework context storage module. It analyzes scenarios where test case execution failed to determine whether the issue stemmed from a bug in the program itself, or from other problems such as the test environment or the test case itself, and submits the analysis results to the framework context storage module.

[0109] S207: Defect Registration. After the results analysis is completed, the defect registration process begins. The defect registration agent retrieves the test results information from the framework context storage module and further organizes the program bug information, clarifying the problem level, error type, severity, responsible person, and other information. After the information processing is completed, the defect management system interface is called to automatically complete the defect registration and submit the final defect details to the framework context storage module.

[0110] S208: Generate Test Report. After defect registration is completed, the test report generation phase begins. The test report generation agent retrieves the requirement list, test case list, test execution results, defect information, etc., from the framework context storage module, organizes them according to the test report template, and generates the final test report. Once generated, the customer is notified to view or download the report, completing the entire automated testing task.

[0111] In summary, the technical solution provided by the above embodiments of this application, based on a multi-agent collaborative framework, uses multiple agents to connect the entire testing process, including requirements analysis, test case generation, test execution, result analysis, defect registration, and report generation. It uses threshold indicators such as code coverage and requirements coverage as termination conditions for each stage. This method addresses the problems of existing technologies that focus on enhancing efficiency at a single stage of the testing process, resulting in automation gaps, data silos, a lack of automated collaborative analysis, and the need for manual intervention. Therefore, the technical solution of this application not only automates the entire testing process but also significantly improves testing quality through an internal feedback mechanism.

[0112] Furthermore, the beneficial effects of the product embodiments provided in the following embodiments of this application are the same as the beneficial effects of the multi-agent-based full-process automated testing method provided in the above embodiments, and other technical features in the product embodiments are the same as the features disclosed in the methods of the above embodiments, and will not be repeated here.

[0113] See also Figure 6 The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the multi-agent-based full-process automated testing method provided in any of the above embodiments.

[0114] like Figure 6 As shown, the electronic device can include a processing unit 1001, such as a central processing unit and / or a graphics processing unit, which can perform various appropriate actions and processes according to a program stored in ROM 1002 or a program loaded from storage device 1003 into RAM 1004. RAM 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output interface 1006 is also connected to bus 1005. Typically, the following systems can be connected to input / output interface 1006: input devices 1007, such as touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, and / or gyroscopes; output devices 1008, such as liquid crystal displays (LCDs), speakers, and / or vibrators; storage devices 1003, such as magnetic tape and / or hard disks; and communication devices 1009. Communication device 1009 is capable of enabling the electronic device to exchange data with other devices wirelessly or via wired communication. Although the diagram shows a model building device with various systems, it should be understood that it is not required to implement or have all of the systems shown. It is possible to implement or have more or fewer systems alternatively.

[0115] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the multi-agent-based full-process automated testing method of the embodiments disclosed in this application.

[0116] This application provides a computer-readable storage medium having computer-readable program instructions stored thereon, namely the aforementioned computer program, which is used to execute the multi-agent-based full-process automated testing method in the above embodiments. The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a model-building device, can be written in one or more programming languages ​​or a combination thereof to perform the operations of this application. The programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, or C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. In cases involving remote computers, the remote computer can connect to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can connect to an external computer, for example, via the Internet using an Internet service provider.

[0117] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram can represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0118] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0119] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0120] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A fully automated testing method based on multi-agent systems, characterized in that, include: For the entire testing process involved in software project testing, a corresponding multi-agent collaboration framework is built; wherein, the entire testing process includes multiple testing stages, and the multi-agent collaboration framework includes multiple agents and a framework context information storage module; The system receives user software requirements, uses the agent corresponding to the current test phase among the multiple agents, executes the current test phase according to the user software requirements, and uploads the test results of the current test phase to the framework context information storage module. Based on the test results stored in the framework context information storage module, the agent corresponding to the next test step among the multiple agents is triggered to automatically execute the next test step, upload the obtained test results to the framework context information storage module, and iteratively execute this step according to the flow order of the entire test step until the final test report corresponding to the user software requirements is obtained.

2. The method as described in claim 1, characterized in that, The steps for building a corresponding multi-agent collaborative framework for the entire testing process involved in software project testing include: According to the above process sequence, the entire testing process is set to include the requirements analysis stage, test case generation stage, test case execution stage, termination threshold judgment stage, result analysis stage, defect registration stage, and test report generation stage; A requirement analysis intelligent agent is set up for the requirement analysis stage, a test case generation intelligent agent is set up for the test case generation stage, a test execution intelligent agent is set up for the test case execution stage, a result analysis intelligent agent is set up for the result analysis stage, a defect registration intelligent agent is set up for the defect registration stage, and a test report generation intelligent agent is set up for the test report stage. In the termination threshold determination process, code coverage threshold and requirement coverage threshold are used to design test termination indicators for the termination threshold determination process; when the test termination indicators are met, the code coverage data and requirement coverage data obtained from the test are submitted as test results to the framework upper and lower level storage module.

3. The method as described in claim 1 or 2, characterized in that, The steps of receiving the input user software requirements, using the agent among the plurality of agents corresponding to the current test phase, executing the current test phase according to the user software requirements, and uploading the test results of the current test phase to the framework context information storage module include: When the requirements analysis agent corresponding to the requirements analysis stage detects the user's software requirements input, it receives the user's software requirements. The demand analysis agent calls a large model to parse the user's software requirements according to pre-organized prompts, and extracts a list of requirements and corresponding verification points. The requirement list and verification points are submitted as test results to the framework context information storage module.

4. The method as described in claim 3, characterized in that, The step of triggering the agent corresponding to the next test stage among the multiple agents to automatically execute the next test stage based on the test results stored in the framework context information storage module, uploading the obtained test results to the framework context information storage module, and iteratively executing this step according to the flow order of the entire test stage until the final test report corresponding to the user software requirements is obtained includes: In the test case generation phase, the test case generation agent extracts the requirement list and verification points from the framework context information storage module; according to the pre-organized prompts, it calls the large model to generate test cases for each requirement in the requirement list and the corresponding verification points; and submits the generated test cases to the framework context information storage module. During the test case execution phase, the test case execution agent retrieves the test cases from the framework context information storage module; executes the test cases by calling the corresponding automated testing tools according to different test case types; uses code coverage analysis tools to collect program code coverage information after the test case execution; and submits the program code coverage information to the framework context information storage module. In the termination threshold determination stage, program code coverage information is extracted from the framework context information storage module, and it is determined whether the program code coverage information reaches the test termination indicator; when the program code coverage information reaches the test termination indicator, the test case execution results are uploaded to the framework context information storage module; wherein, the test termination indicator is designed based on the code coverage threshold and the requirement coverage threshold, and the test case execution results include test case execution status, code coverage data and requirement coverage data; In the results analysis phase, the results analysis agent extracts the test case execution results from the framework context information storage module, uses the test case execution results to test the execution log, obtains the test execution results, and submits them to the framework context information storage module. In the test report generation stage, the test report agent extracts the requirement list, test cases, code coverage information, test case execution results, and test execution results from the framework context information storage module. Following the test report template, the final test report is generated using the requirement list, test cases, code coverage information, test case execution results, and test execution results.

5. The method as described in claim 4, characterized in that, The step of triggering the agent corresponding to the next test stage among the multiple agents to automatically execute the next test stage based on the test results stored in the framework context information storage module, uploading the obtained test results to the framework context information storage module, and iteratively executing this step according to the flow order of the entire test stage until the final test report corresponding to the user software requirements is obtained includes: In the defect registration stage between the result analysis stage and the test report generation stage, a defect registration agent is used to extract the test execution results from the framework context information storage module. The test execution results are used to organize program defect information, and the defect management system interface is called to register the program defect information to obtain detailed defect information. The defect details are submitted to the framework context information storage module.

6. The method as described in claim 4, characterized in that, The steps of calling the corresponding automated testing tool to execute the test cases according to different test case types, and using a code coverage analysis tool to statistically analyze the program code coverage information after the test case execution include: Extract the test case list from the framework context information storage module; Analyze the test case type corresponding to each test case in the test case list; According to different test case types, UI automation testing tools, interface automation testing tools, performance automation testing tools, and security automation testing tools are invoked respectively. Use the UI automation testing tool, interface automation testing tool, performance automation testing tool, and security automation testing tool to test the test cases of the corresponding test case types.

7. The method as described in claim 4, characterized in that, In the termination threshold determination step, program code coverage information is extracted from the framework context information storage module, and it is determined whether the program code coverage information reaches the test termination index. When the program code coverage information reaches the test termination metric, the step of uploading the test case execution results to the framework context information storage module includes: In the termination threshold determination step, according to the user software requirements, the line coverage threshold and branch coverage threshold under the code coverage threshold are set respectively to generate the multi-level requirement coverage threshold under the requirement coverage. From the program code coverage information, extract the line coverage, branch coverage, and multi-level requirement coverage corresponding to the multi-level requirement coverage threshold, respectively. Determine whether the row coverage rate is greater than or equal to the row coverage rate threshold, whether the branch coverage rate is greater than or equal to the branch coverage rate, and whether the multi-level demand coverage rate is greater than or equal to the multi-level demand coverage rate, respectively. If it is determined that the row coverage rate is greater than or equal to the row coverage rate threshold, the branch coverage rate is greater than or equal to the branch coverage rate, and the multi-level requirement coverage rate is greater than or equal to the multi-level requirement coverage rate, then the test case execution results are uploaded to the framework context information storage module and the result analysis phase begins.

8. The method as described in claim 7, characterized in that, After the steps of determining whether the row coverage rate is greater than or equal to the row coverage rate threshold, whether the branch coverage rate is greater than or equal to the branch coverage rate, and whether the multi-level demand coverage rate is greater than or equal to the multi-level demand coverage rate, the method further includes: If it is determined that the row coverage rate is less than the row coverage rate threshold, the branch coverage rate is less than the branch coverage rate, and the multi-level requirement coverage rate is less than the multi-level requirement coverage rate, then the test case execution result is uploaded to the framework context information storage module, and the requirement analysis agent corresponding to the requirement analysis stage is returned to be invoked. Using the aforementioned requirement analysis agent, the latest test cases and program code coverage information are retrieved from the framework context information storage module. Supplementary test case prompts are organized according to the test cases and program code coverage information, and the supplementary test case prompts are uploaded to the framework context information storage module. The agent is generated using the test cases, and the large model is invoked using the supplementary test case prompts. Supplementary test cases corresponding to the uncovered parts are generated again and uploaded to the framework context information storage module. The test case execution agent calls supplementary test cases from the framework context information storage module, and executes the supplementary test cases according to the automated testing tool corresponding to the test case type to obtain program code coverage information; Repeat the above steps until the program code coverage information reaches the test termination index, then proceed to the result analysis stage.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-agent-based fully automated testing method as described in any one of claims 1 to 8.

10. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the multi-agent-based fully automated testing method as described in any one of claims 1 to 8.

Images