Automatic testing and document generating method and system based on AI drive

Abstract

The invention discloses an AI-driven automatic testing and document generation method and system, and the method comprises the steps: obtaining document data of test software, carrying out the preprocessing and information extraction, and forming structured data; identifying key test points from the structured data by using an agent, and generating a test case; executing the test case by utilizing a preset automatic test framework to obtain a test result; retrieving data information associated with the structured data in the knowledge base by utilizing RAG to form a retrieval result set; fusing the retrieval result set with the structured data and the test result according to a preset weight strategy to construct a complete context; and taking the context as input, calling the large language model by utilizing an agent, and generating a technical document. By means of the technical scheme, accurate and real-time technical document generation is achieved, the document generation quality can be continuously improved, and the real-time updating capacity and the data driving decision support capacity of the document are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and particularly to an AI-driven automated testing and document generation method and an AI-driven automated testing and document generation system. Background Art

[0002] The software testing technology has experienced a transformation from manual testing to automated testing. Automated testing uses scripts or specialized testing tools to automatically execute highly repetitive testing tasks, greatly improving the efficiency and accuracy of testing. In recent years, with the continuous development and maturity of artificial intelligence (AI) technology, the application of AI in the field of software testing has become increasingly widespread, especially in automated testing methods.

[0003] As software testing evolves from early manual testing and scripted automated testing to AI-based automated testing, existing technical solutions in the market have basically achieved the automatic generation and execution of test cases by an intelligent agent Agent after inputting software requirements or code. However, these solutions usually have the following defects:

[0004] (1) The generation, execution, and document generation of test cases form a "black box" as a whole, only describing what to input and what to output, lacking a detailed description of the internal data processing, information integration, and intelligent generation mechanism;

[0005] (2) Existing technologies disclose solutions for automatically generating test cases and executing tests using an Agent. However, in terms of technical document generation, most solutions only stay at directly invoking large language models for document summarization and generation, failing to solve the problems of document quality and real-time performance caused by large amounts of data and scattered information;

[0006] (3) Solutions for automatically generating technical documents by integrating the retrieval of external knowledge bases with generation models (i.e., RAG technology) are rarely disclosed in the existing literature, and the overall technical solutions are not transparent and systematic enough. Summary of the Invention

[0007] In view of the above problems, the present invention provides an AI-driven automated testing and document generation method and system. By deeply integrating the Agent with the RAG retrieval technology, vectorized retrieval technology is used to dynamically obtain supplementary information related to the current project from the knowledge base, and the context is constructed by fusing with the original data through a weight strategy to achieve accurate and real-time technical document generation. At the same time, through the self-learning and feedback iteration mechanism, the generation quality of the document can be continuously improved. The intelligent agent Agent is used to automatically complete the entire process from requirements, design to testing and document generation, reducing manual intervention while improving the real-time update ability of the document and the data-driven decision support ability.

[0008] To achieve the above object, the present invention provides an AI-driven automated testing and document generation method, including:

[0009] Obtain the document data of the software to be tested, and preprocess and extract information from the document data to form structured data;

[0010] Use an Agent to identify key test points from the structured data through machine learning algorithms, and generate test cases based on the key test points;

[0011] Use a preset automated testing framework to execute the test cases to obtain test results;

[0012] Use the Agent to retrieve relevant material information associated with the structured data in a pre-constructed knowledge base using RAG to form a retrieval result set;

[0013] Fuse the retrieval result set with the structured data and the test results according to a preset weight strategy to construct a complete context related to the test software;

[0014] Use the context as input, and use the Agent to call a large language model and generate a technical document according to a preset prompt.

[0015] In the above technical solution, preferably, the AI-driven automated testing and document generation method further includes:

[0016] Continuously collect test results, document review opinions, and code change data to form a feedback data set;

[0017] Combine the feedback data set and use a self-learning mechanism to iteratively optimize and provide feedback on the initial draft of the technical document;

[0018] Use the feedback data to fine-tune the generation model and retrieval module in the Agent, and optimize the large language model and the prompt.

[0019] In the above technical solution, preferably, the process of using a preset automated testing framework to execute the test cases to obtain test results specifically includes:

[0020] Use a preset automated testing framework to execute the test cases and record a test report;

[0021] Perform natural language processing on the test report to extract defect type and code location information, interface parameter verification results, and test coverage analysis data;

[0022] Structure the extracted data as the test results.

[0023] In the above technical solution, preferably, the intelligent agent Agent is used to retrieve the material information associated with the structured data in a pre-constructed knowledge base by using RAG to form a retrieval result set. The specific process includes:

[0024] Construct a knowledge base related to the technology of the test software based on historical data, and use vectorization technology to perform semantic encoding on the knowledge base to form a retrievable vector index, where the historical data includes historical project documents, technical standards, API descriptions, code comments, and industry public information;

[0025] Use the intelligent agent Agent to retrieve the material information associated with the structured data in the knowledge base by using RAG retrieval technology to form a retrieval result set;

[0026] When RAG retrieves a historical defect repair solution, compare the current test results with the historical test results to find similar defect patterns, and dynamically adjust the document generation strategy of the generation model in the intelligent agent Agent.

[0027] In the above technical solution, preferably, the retrieval result set, the structured data, and the test results are fused according to a preset weight strategy to construct a complete context related to the test software. The specific process includes:

[0028] Adopt a preset weight strategy to fuse the retrieval result set, the structured data, and the test results;

[0029] Construct a complete context related to the test software according to the fusion content. The context includes detailed function descriptions, interface descriptions, test results, and historical reference information.

[0030] In the above technical solution, preferably, using the context as input, the intelligent agent Agent is used to call a large language model and generate a technical document according to a preset prompt word. The specific process includes:

[0031] Use the intelligent agent Agent to call a preset large language model and use the context as the input of the large language model;

[0032] According to the clear directive words and format requirements of the preset prompt word, the large language model generates a technical document.

[0033] The present invention also proposes an AI-driven automated testing and document generation system, which applies the AI-driven automated testing and document generation method disclosed in any one of the above technical solutions, including:

[0034] A structure data extraction module, which is used to obtain the document data of the software to be tested, preprocess and extract information from the document data to form structured data;

[0035] A test case generation module, which is used to identify key test points from the structured data by using an intelligent agent Agent through a machine learning algorithm, and generate test cases according to the key test points;

[0036] An automatic test execution module, which is used to execute the test cases by using a preset automated test framework to obtain test results;

[0037] A knowledge RAG retrieval module, which is used to use the intelligent agent Agent to retrieve relevant material information associated with the structured data in a pre-constructed knowledge base by using RAG to form a retrieval result set;

[0038] A complete data fusion module, which is used to fuse the retrieval result set, the structured data and the test results according to a preset weight strategy to construct a complete context related to the test software;

[0039] A technical document generation module, which is used to use the context as input, use the intelligent agent Agent to call a large language model, and generate technical documents according to preset prompt words.

[0040] In the above technical solution, preferably, the AI-driven automated test and document generation system further includes a feedback optimization and adjustment module, which is specifically used for:

[0041] Continuously collect test results, document review opinions and code change data to form a feedback data set;

[0042] Combined with the feedback data set, adopt a self-learning mechanism to iteratively optimize and feedback the initial draft of the technical document;

[0043] Use the feedback data to fine-tune the generation model and retrieval module in the intelligent agent Agent, and optimize the large language model and the prompt words.

[0044] In the above technical solution, preferably, the knowledge RAG retrieval module is specifically used for:

[0045] Build a knowledge base related to the technology of the test software based on historical data, and use vectorization technology to perform semantic encoding on the knowledge base to form a retrievable vector index, where the historical data includes historical project documents, technical standards, API descriptions, code comments and industry public information;

[0046] Use the agent to retrieve information related to the structured data in the knowledge base using the RAG retrieval technique to form a retrieval result set;

[0047] When the RAG retrieves a historical defect repair solution, compare the current test results to find similar defect patterns based on the historical test results, and dynamically adjust the document generation strategy of the generation model in the agent.

[0048] In the above technical solution, preferably, the complete data fusion module is specifically used for:

[0049] Adopt a preset weight strategy to fuse the retrieval result set with the structured data and the test results;

[0050] Construct a complete context related to the test software based on the fusion content, where the context includes detailed function descriptions, interface specifications, test results, and historical reference information.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows: By deeply integrating the agent with the RAG retrieval technique, using the vectorized retrieval technique to dynamically obtain supplementary information related to the current project from the knowledge base, and fusing it with the original data through a weight strategy to construct a context, accurate and real-time technical document generation is achieved. At the same time, through the self-learning and feedback iteration mechanism, the generation quality of the documents can be continuously improved. The agent automatically completes the entire process from requirements, design to testing and document generation, reducing manual intervention while improving the real-time update ability of the documents and the data-driven decision support ability. Brief Description of the Drawings

[0052] Figure 1 It is a schematic flowchart of a method for AI-driven automated testing and document generation disclosed in an embodiment of the present invention;

[0053] Figure 2 It is a schematic flowchart of a framework for combining the agent with RAG retrieval disclosed in an embodiment of the present invention;

[0054] Figure 3 It is a schematic flowchart of test errors and feedback iteration disclosed in an embodiment of the present invention. Detailed Embodiments

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0056] The present invention will be further described in detail below with reference to the accompanying drawings:

[0057] As Figure 1 and Figure 2 shown, a method for AI-driven automated testing and document generation provided by the present invention includes:

[0058] Obtain the document data of the software to be tested, and perform preprocessing and information extraction on the document data to form structured data;

[0059] Use the intelligent agent Agent to identify key test points from the structured data through machine learning algorithms, and generate test cases based on the key test points;

[0060] Use a preset automated testing framework to execute the test cases and obtain test results;

[0061] Use the intelligent agent Agent to retrieve relevant material information associated with the structured data in a pre-constructed knowledge base using RAG to form a retrieval result set;

[0062] Fuse the retrieval result set with the structured data and test results according to a preset weight strategy to construct a complete context related to the test software;

[0063] Using the context as input, use the intelligent agent Agent to call a large language model and generate a technical document according to a preset prompt.

[0064] In this embodiment, through the deep combination of the Agent and RAG retrieval technology, the vectorized retrieval technology is used to dynamically obtain supplementary information related to the current project from the knowledge base, and the context is constructed by fusing with the original data through the weight strategy, realizing accurate and real-time technical document generation. At the same time, through the self-learning and feedback iteration mechanism, the generation quality of the document can be continuously improved. The intelligent agent Agent automatically completes the entire process from requirements, design to testing and document generation, reducing manual intervention while improving the real-time update ability of the document and the data-driven decision support ability.

[0065] During the implementation process, during the process of obtaining document data of the test software, the input content includes requirement documents, design documents, source code, API documents, user manuals, etc. A custom parser is used to preprocess documents in different formats, extract key information (such as functional modules, interface descriptions, data structures, change logs, etc.), and form structured data. During the process of generating test cases, the Agent identifies key test points from the preprocessed requirement and design documents to generate test cases.

[0066] Among them, the vectorized retrieval technology is used to dynamically obtain supplementary information related to the current project from the knowledge base, fuse it with the original data, construct the context, and achieve accurate and real-time technical document generation. This refined implementation process is different from the existing "black box" solution that only relies on large language models to directly generate documents. Through the weight strategy and fusion algorithm, it is ensured that the generated documents not only retain the real-time nature of the project but also combine historical knowledge, making up for the deficiencies of large language models in terms of generation accuracy and consistency. In addition, through a complete set of algorithmic process designs for information extraction, context construction, dynamic retrieval, and multiple iterative optimizations, from requirements, design to testing and document generation, it is all automatically completed by the Agent, realizing full-process automation, and providing support for test decision-making and resource allocation through data-driven, ensuring that the generated documents have high accuracy, consistency, and traceability. While reducing manual intervention, it improves the real-time update ability of the documents and data-driven decision support, significantly different from the traditional single large language model generation solution.

[0067] In the above implementation manner, preferably, a preset automated test framework is used to execute the test cases to obtain test results. The specific process includes:

[0068] Use the preset automated test framework to execute the test cases and record the test report;

[0069] Perform natural language processing on the test report to extract the defect type (configuration management), code location information (code file + line number), interface parameter verification results (expected value / actual value comparison), and test coverage analysis data;

[0070] Structurize the extracted data as the test result.

[0071] In the above implementation manner, preferably, the intelligent agent Agent uses RAG to retrieve the material information associated with the structured data in the pre-constructed knowledge base to form a retrieval result set. The specific process includes:

[0072] Construct a knowledge base related to the technology of the test software based on historical data, and use vectorization technology to perform semantic encoding on the knowledge base to form a retrievable vector index. Among them, the historical data includes historical project documents, technical standards, API descriptions, code comments, and industry public information;

[0073] When the Agent starts generating technical documents, the intelligent agent Agent uses the RAG retrieval technology in the knowledge base to retrieve the information related to the structured data. Specifically, according to the structured data output by the preprocessing module (such as the extracted API names, function descriptions, interface parameters, etc.), the vector search technology is used to retrieve the background information and supplementary materials related to the current project in the knowledge base, forming a retrieval result set;

[0074] When the RAG retrieves the historical defect repair solutions, it compares the current test results with the historical test results to find similar defect patterns, and dynamically adjusts the document generation strategy of the generation model in the intelligent agent Agent. For example: If the current test finds an error of "database connection pool overflow", and there is a solution document for this problem in the knowledge base (historical defect repair solutions), then when generating the technical document, the defect repair solution will be automatically inserted into the "Configuration Suggestions" section of the technical document.

[0075] In the above embodiment, preferably, the retrieval result set, the structured data, and the test results are fused according to a preset weight strategy to construct a complete context related to the test software. The specific process includes:

[0076] Adopt a preset weight strategy to fuse the retrieval result set, the structured data, and the test results to ensure that there is both a distinction and complementarity between the real-time data of the current project and the historical knowledge;

[0077] Construct a complete context related to the test software according to the fused content. The context includes detailed function descriptions, interface descriptions, test results, and historical reference information. This context will be used as the input for the subsequent large language model to generate technical documents, so that the generated technical documents not only reflect the latest project status but also have technical depth and historical comparability.

[0078] In the above embodiment, preferably, using the context as the input, the intelligent agent Agent calls the large language model and generates technical documents according to the preset prompt words. The specific process includes:

[0079] Use the intelligent agent Agent to call the preset large language model and use the context as the input of the large language model;

[0080] According to the clear directive words and format requirements of the preset prompt words, the large language model generates technical documents.

[0081] During the implementation process, the technical documents generated by the large language model can be adjusted by pre-designing the prompt words.

[0082] Such as Figure 3As shown above, in the above embodiments, preferably, the AI-driven automated testing and document generation method further includes:

[0083] Continuously collect information such as test results, manually marked document review opinions by users, and code change data to form a feedback dataset;

[0084] Combined with the feedback dataset, adopt a self-learning mechanism to iteratively optimize and provide feedback on the initial draft of the technical document to ensure the accuracy, consistency, and high readability of the document content;

[0085] Use the feedback data to fine-tune the generation model and retrieval module in the intelligent agent Agent, enabling the system to continuously optimize the retrieval strategy and information fusion weight, optimize the large language model and prompt words, and improve the overall performance and generation quality.

[0086] The present invention also proposes an AI-driven automated testing and document generation system, which applies the AI-driven automated testing and document generation method disclosed in any one of the above embodiments, including:

[0087] A structure data extraction module, which is used to obtain the document data of the software to be tested, preprocess and extract information from the document data to form structured data;

[0088] A test case generation module, which is used to use the intelligent agent Agent to identify key test points from the structured data through machine learning algorithms and generate test cases based on the key test points;

[0089] An automatic test execution module, which is used to execute the test cases using a preset automated test framework to obtain test results;

[0090] A knowledge RAG retrieval module, which is used to use the intelligent agent Agent to retrieve relevant material information associated with the structured data in a pre-constructed knowledge base to form a retrieval result set;

[0091] A complete data fusion module, which is used to fuse the retrieval result set with the structured data and test results according to a preset weight strategy to construct a complete context related to the test software;

[0092] A technical document generation module, which takes the context as input, uses the intelligent agent Agent to call a large language model, and generates a technical document according to a preset prompt word.

[0093] In the above embodiments, preferably, the AI-driven automated testing and document generation system further includes a feedback optimization and adjustment module, which is specifically used for:

[0094] Continuously collect test results, document review opinions, and code change data to form a feedback dataset;

[0095] In combination with the feedback data set, a self-learning mechanism is adopted to iteratively optimize and provide feedback on the initial draft of the technical document;

[0096] The feedback data is used to fine-tune the generation model and retrieval module within the intelligent agent Agent, and to optimize the large language model and prompt words.

[0097] In the above embodiment, preferably, the knowledge RAG retrieval module is specifically configured to:

[0098] Build a knowledge base related to the technology of the test software based on historical data, and use vectorization technology to perform semantic encoding on the knowledge base to form a retrievable vector index, where the historical data includes historical project documents, technical standards, API descriptions, code comments, and industry public information;

[0099] Use the intelligent agent Agent to retrieve information related to structured data in the knowledge base using RAG retrieval technology to form a retrieval result set;

[0100] When the RAG retrieves the historical defect repair solution, compare and find the similar defect patterns of the current test results based on the historical test results, and dynamically adjust the document generation strategy of the generation model in the intelligent agent Agent.

[0101] In the above embodiment, preferably, the complete data fusion module is specifically configured to:

[0102] Adopt a preset weight strategy to fuse the retrieval result set with the structured data and test results;

[0103] Construct a complete context related to the test software based on the fusion content, where the context includes detailed function descriptions, interface descriptions, test results, and historical reference information.

[0104] According to the AI-driven automated testing and document generation system disclosed in the above embodiment, the functions to be realized by each module correspond to the steps of the AI-driven automated testing and document generation method disclosed in the above embodiment respectively. During the implementation process, refer to the above embodiment for operation, and details are not described herein again.

[0105] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An AI-driven automated testing and document generation method, characterized in that: include: Obtaining document data of the software to be tested, and preprocessing and extracting information from the document data to form structured data; Using an intelligent agent to identify key test points from the structured data through a machine learning algorithm, and generating test cases based on the key test points; Execute the test cases using a preset automated testing framework to obtain test results; Using the intelligent agent to use RAG in a pre-built knowledge base to retrieve information associated with the structured data to form a search result set; The search result set is integrated with the structured data and the test result according to a preset weight strategy to construct a complete context related to the test software; The context is used as input, the large language model is called by the intelligent agent, and technical documents are generated according to preset prompt words.

2. The AI-driven automated testing and document generation method according to claim 1, characterized in that: Also includes: Continuously collect test results, document review comments, and code change data to form a feedback data set; In combination with the feedback data set, a self-learning mechanism is used to iteratively optimize and provide feedback on the draft of the technical document; The feedback data is used to fine-tune the generation model and the retrieval module in the intelligent agent, and to optimize the large language model and the prompt words.

3. The AI-driven automated testing and document generation method according to claim 1 or 2, characterized in that: The process of executing the test case using the preset automated testing framework to obtain the test result specifically includes: Execute the test cases using the preset automated testing framework and record the test reports; Performing natural language processing on the test report to extract defect type and code location information, interface parameter verification results, and test coverage analysis data; The extracted data is structured as the test result.

4. The AI-driven automated testing and document generation method according to claim 3, characterized in that: The intelligent agent is used to use RAG to retrieve information associated with the structured data in the pre-built knowledge base to form a search result set. The specific process includes: Building a knowledge base of technologies related to the test software based on historical data, and semantically encoding the knowledge base using vectorization technology to form a searchable vector index, wherein the historical data includes historical project documents, technical standards, API descriptions, code comments, and industry public information; Using the intelligent agent to use RAG retrieval technology in the knowledge base to retrieve information associated with the structured data to form a retrieval result set; When RAG retrieves historical defect repair solutions, it searches for similar defect patterns of current test results based on historical test results, and dynamically adjusts the document generation strategy of the generation model in the intelligent agent.

5. The AI-driven automated testing and document generation method according to claim 4, characterized in that: The search result set is integrated with the structured data and the test result according to a preset weight strategy to construct a complete context related to the test software. The specific process includes: Adopting a preset weight strategy to merge the search result set with the structured data and the test result; A complete context related to the test software is constructed based on the fused content, the context including detailed functional description, interface description, test results and historical reference information.

6. The AI-driven automated testing and document generation method according to claim 5, characterized in that: Taking the context as input, the agent is used to call the large language model, and a technical document is generated according to the preset prompt words. The specific process includes: Using the intelligent agent Agent to call a preset large language model, and using the context as an input of the large language model; According to the clear indicator words and format requirements of the preset prompt words, the large language model generates technical documents.

7. An AI-driven automated testing and document generation system, characterized in that: The method for automated testing and document generation based on AI driving according to any one of claims 1 to 6 is applied, comprising: A structured data extraction module is used to obtain the document data required for testing the software, and to preprocess and extract information from the document data to form structured data; A test case generation module, used to use an intelligent agent to identify key test points from the structured data through a machine learning algorithm, and generate test cases according to the key test points; An automatic test execution module is used to execute the test cases using a preset automatic test framework to obtain test results; A knowledge RAG retrieval module, used to use the agent to use RAG to retrieve information associated with the structured data in a pre-built knowledge base to form a retrieval result set; A complete data fusion module, used to fuse the search result set with the structured data and the test result according to a preset weight strategy to construct a complete context related to the test software; The technical document generation module is used to use the context as input, use the intelligent agent to call the large language model, and generate technical documents according to preset prompt words.

8. The AI-driven automated testing and document generation system according to claim 7, characterized in that: It also includes a feedback optimization and adjustment module, which is specifically used to: Continuously collect test results, document review comments, and code change data to form a feedback data set; In combination with the feedback data set, a self-learning mechanism is used to iteratively optimize and provide feedback on the draft of the technical document; The feedback data is used to fine-tune the generation model and the retrieval module in the intelligent agent, and to optimize the large language model and the prompt words.

9. The AI-driven automated testing and document generation system according to claim 7 or 8, characterized in that: The knowledge RAG retrieval module is specifically used for: Building a knowledge base of technologies related to the test software based on historical data, and semantically encoding the knowledge base using vectorization technology to form a searchable vector index, wherein the historical data includes historical project documents, technical standards, API descriptions, code comments, and industry public information; Using the intelligent agent to use RAG retrieval technology in the knowledge base to retrieve information associated with the structured data to form a retrieval result set; When RAG retrieves historical defect repair solutions, it searches for similar defect patterns of current test results based on historical test results, and dynamically adjusts the document generation strategy of the generation model in the intelligent agent.

10. The AI-driven automated testing and document generation system according to claim 9, characterized in that: The complete data fusion module is specifically used for: Adopting a preset weight strategy to merge the search result set with the structured data and the test result; A complete context related to the test software is constructed based on the fused content, the context including detailed functional description, interface description, test results and historical reference information.

Images