Test system based on multi-modal knowledge base

Abstract

The invention relates to a test system based on a multi-modal knowledge base, belongs to the technical field of software testing, and solves the problems of insufficient reuse of historical project data and expert knowledge, lack of professional domain knowledge, illusion, low test case adoption rate and low practicability. The system comprises a knowledge base construction module used for constructing a hierarchical multi-mode professional field knowledge base; the preprocessing module is used for forming a standardized project development document and a software source code file of a to-be-tested project; the test case generation module is used for generating a test case and a software test description thereof based on the multi-modal professional domain knowledge base, the project development document of the project to be tested and the software source code file; and the test execution module prepares execution conditions according to the test case and the software test description thereof, and then executes the software test steps in sequence according to the input and operation description steps described by the test process to obtain an execution result of the test case. And knowledge-based and deep structured multiplexing of test assets is realized, and the test quality and efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software testing, and in particular to a test system based on a multi-modal knowledge base. BACKGROUND

[0002] Software testing, as the core link of software quality assurance system, plays an irreplaceable role in high reliability fields such as aerospace. Through systematic design and execution of test cases, the process aims to effectively identify potential problems and defects in software, verify the compliance of its functions and non-functional requirements, and thus ensure that the software product meets the established quality standards and reliability requirements.

[0003] In the software testing process, there are high requirements for the standardization of the testing process and the quality of software testing. Test requirement analysis and disassembly, test outline writing and its review, as key links of software testing quality assurance, are the key inputs for subsequent test case design and test specification document preparation. Traditional test case design mainly relies on the personal experience and subjective judgment of testers, which requires a lot of time and labor cost. At the same time, a large amount of historical project data and expert knowledge cannot be effectively reused and utilized, and often there are problems such as non-standard test cases, incomplete test steps, poor executability, and incomplete requirement coverage designed by testers, which not only increases the maintenance cost and quality risk, but also directly affects the sufficiency and effectiveness of testing. The long cycle, low efficiency, high cost, and unstable quality of test case design and test specification document preparation seriously restrict the improvement of test coverage and the timely discovery of defects, and gradually fail to meet the testing and verification requirements of high-quality software systems.

[0004] In recent years, breakthrough progress has been made in artificial intelligence technology, especially in the evolution of large language models (LLM) based on the Transformer architecture, which provides a new technical approach for test case generation. Relying on its powerful natural language processing and reasoning capabilities, large language models can understand and summarize the key points of test case design in requirement documents, and can assist testers in quickly designing test cases in test case generation, improving work efficiency, and showing strong application potential. However, current large models are mostly trained based on general corpus, and generally lack professional domain knowledge, have prominent model hallucination problems, have poor test case executability, low adoption rate, and weak practicality. SUMMARY

[0005] In view of the above analysis, the embodiments of the present application aim to provide a test system based on a multi-modal knowledge base to solve the problems of insufficient reuse of historical projects and expert knowledge in existing test systems, lack of professional domain knowledge in large models, prominent hallucination problems, low adoption rate of test cases, and weak practicality.

[0006] In one aspect, the embodiment of the present application provides a test system based on a multi-modal knowledge base, comprising: a knowledge base construction module, configured to classify and filter and normalize project development documents, software source code files, project test documents and software test document templates of historical projects, so as to construct a hierarchical multi-modal professional field knowledge base; a preprocessing module, configured to preprocess project development documents and software source code files of a to-be-tested project, so as to form normalized project development documents and software source code files of the to-be-tested project; a test case generation module, configured to generate test cases and software test instructions based on the multi-modal professional field knowledge base, the project development documents of the to-be-tested project and the software source code files; and a test execution module, configured to prepare an execution condition according to the test cases and the software test instructions, and then execute software test steps in sequence according to input and operation instruction steps described in a test process description, so as to obtain an execution result of the test cases.

[0007] The technical scheme has the following beneficial effects: a high-quality hierarchical multi-modal professional field knowledge base is constructed, knowledge of test assets is structured and reused in depth, and test design efficiency is significantly improved. The present application first systematically standardizes and deeply structures scattered, heterogeneous and multi-modal data assets (requirement specification, source code, test outline, test case, etc.) in the field of software testing. By constructing a multi-modal associated knowledge base (test case text library, test case associated graph library, software code library and test document template library), originally isolated documents that need to be manually reviewed and understood are converted into structured knowledge that can be understood, searched and reasoned by machines. When designing new test cases, testers can efficiently recall and reuse high-quality test assets and field knowledge in historical projects.

[0008] The technical schemes described above can be combined with each other to achieve more preferred combination schemes. Other features and advantages of the present application will be described in the following description, and some advantages will become apparent from the description, or will be understood by those skilled in the art through implementation of the present application. The purposes and other advantages of the present application can be achieved and obtained from the contents specifically pointed out in the description and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0009] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings: Figure 1 A block diagram of a test system based on a multi-modal knowledge base according to an embodiment of the present application; Figure 2 A first-stage test tree diagram according to an embodiment of the present application; Figure 3A tree graph for second stage testing according to an embodiment of the present application; Figure 4 A block diagram of a test case generation system according to an embodiment of the present application; Figure 5 A system architecture diagram of a multi-modal knowledge base based testing system according to an embodiment of the present application. DETAILED DESCRIPTION

[0010] Preferred embodiments of the present application are described in detail below with reference to the attached drawing figures, wherein the present application is shown as being implemented in the particular embodiments, by way of illustration and not by way of limitation. The following detailed description is presented in order to describe the preferred embodiments and should not be taken as limiting the present application.

[0011] Reference Figure 1 In one embodiment of the present application, a multi-modal knowledge base based testing system is disclosed, comprising a knowledge base construction module 101, a preprocessing module 102, a test case generation module 103, a test execution module 104 and a cause analysis and traceability module 105.

[0012] The knowledge base construction module 101 is configured to classify, filter and normalize the project development documents, software source code files, project test documents and software test document templates of historical projects to construct a hierarchical multi-modal professional domain knowledge base. The multi-modal professional domain knowledge base can include a test case text library, a test case correlation graph library and a software code library.

[0013] The knowledge base construction module can include an identification system construction module, a data asset construction module, a test case text file construction module, a test case text library construction module, a test case correlation graph library construction module, a software code library construction module and a software test document template library construction module.

[0014] An identification system construction module is configured to construct a unique identification (ID) system by using a hierarchical combination coding method. A data asset construction module is configured to collect and organize project development documents, software source code files, project test documents, and software test document templates of a plurality of projects accumulated historically to form data assets. A test case text file construction module is configured to classify and normalize the project development documents and the project test documents of each project in the data assets to form test case text files. A test case text library construction module is configured to segment, embed, vectorize, and index the test case text files to form a test case text library and perform retrieval and evaluation on the test case text library. A test case association graph library construction module is configured to analyze the development documents, the project test documents, and the software source code files of the plurality of projects, extract test case association relationships, construct a hierarchical test case tree, store the hierarchical test case tree in a graph database, and establish a test case association graph library. A software code library construction module is configured to establish a code file metadata model and a function metadata model, initialize a code library directory structure according to a unified standard format, and perform metadata annotation and storage processing on the software source code files and the functions included therein according to the software type, the professional field, the project name, and the unique identification (ID) of the project to form a software code library. A software test document template library construction module is configured to classify and normalize the software test document templates in the data assets to construct a software test document template library.

[0015] The project development documents include software requirement specification, software design specification, development general requirement, technical scheme, and other project document materials provided by a software development unit.

[0016] The project test documents include software test outline, software test specification, software test record, software test report, and other software test process documents.

[0017] The software test document templates include software test outline template, software test specification template, software test record template, software test report template, and other document templates in the software test process.

[0018] The software test outline includes test object, test environment, name of a software configuration item to be tested, test type, test item name, test item identification, tracking relationship with the development documents, test subitem, and a description thereof.

[0019] The software test description includes: a summary of the software under test (including software name, software type, programming language, development environment, running environment, development unit, etc.), test environment, configuration item name of the software under test, test type, test case name, test case identifier, tracking relationship with test outline, test description, test case initialization, premise and constraint, termination condition, test process (including input and operation description, expected test result, evaluation criterion given according to detailed test steps), etc.

[0020] The data assets are collected, classified and screened according to corresponding standards and standardized, and a high-quality hierarchical multi-modal professional field knowledge base is constructed, mainly including: (1) A unique identification ID system is constructed by using a hierarchical combination coding method.

[0021] 1) The asset type (identification object) is defined as follows: The document assets (document type abbreviation) include: SRS requirement specification (Software Requirements Specification); SDS software design specification (Software Design Specification); TSL technical solution (Technical Solution); TSD test outline (Test Specification Document); TDD test description (Test Description Document); TRP test report (Test Report).

[0022] The code assets include: SRC source code (Source Code); CLA class (Class); FUC function (Function); CDS code snippet (Code Snippet).

[0023] The knowledge units include: DEP document paragraph (Document Extracted Paragraph); TBL table (Table); PIC picture (Picture); TRP test requirement point (Test Requirement Point); SCI software configuration item (Software Configuration Item); FNT functional test (Functional Test); PFT performance test (Performance Test); INT interface test (Interface Test); SCT security test (Security Test); TST test item (Test Item); TSI test subitem (Test Subitem); STC software test case (Software TestCase).

[0024] 2) ID coding standard specification: uniform hierarchical combination coding method is adopted to ensure the uniqueness, readability and analyzability of ID. Among them, the hierarchical serial number of software configuration item, test item, test subitem and test case can reflect its position in the test tree.

[0025] Exemplarily, the ID coding standard specification can be defined as: ① Project ID (Project_ID): PRJ{software type number (2 bits)}{professional field number (2 bits)}{project year and month YYYYMM (6 bits)}{serial number (3 bits)}.

[0026] Among them, the software type number rule is: Embedded software: 01; Non-embedded software: 02; FPGA software: 03.

[0027] Among them, the professional field number is the professional field number uniformly developed by the unit for different professional fields, for example, the satellite control system professional number is 01, the aircraft inertial navigation professional number is 02, etc.

[0028] Among them, the serial number is the unique serial number under the same level.

[0029] Exemplarily, the ID of the embedded software test project of the 16th aircraft inertial navigation professional in June 2025 is: PRJ0102202506016.

[0030] ② Document ID (Doc_ID): {Project_ID}-{document type abbreviation}-{version number}.

[0031] Exemplarily, the second software configuration item ID in the test outline document of the 16th aircraft inertial navigation professional embedded software test project in June 2025: PRJ0102202506016-TSD-V1.2-SCI02.

[0032] ③Software configuration item ID (SCI_ID): {Doc_ID}-SCI{Serial number (2 digits)}.

[0033] Exemplarily, the second software configuration item ID in the test outline document of the 16th aircraft inertial navigation professional embedded software test project in June 2025: PRJ0102202506016-TSD-V1.2-SCI02.

[0034] ④Test item ID (TestItem_ID): {SCI_ID}-{Test type (FNT / PFT / INT / SCT, etc.)}-TST{Hierarchical serial number (3 digits)}.

[0035] Exemplarily, the fifth test item ID of the function test of the second software configuration item in the test outline document of the 16th aircraft inertial navigation professional embedded software test project in June 2025: PRJ0102202506016-TSD-V1.2-SCI02-FNT-TST005.

[0036] ⑤Test subitem ID (TestSubitem_ID): {TestItem_ID}-TSI{Hierarchical serial number (3 digits)}.

[0037] Exemplarily, the second test subitem ID of the fifth test item of the function test of the second software configuration item in the test outline document of the 16th aircraft inertial navigation professional embedded software test project in June 2025: PRJ0102202506016-TSD-V1.2-SCI02-FNT-TST005-TSI002.

[0038] ⑥Test case ID (TestCase_ID): {TestSubitem_ID}-STC{Hierarchical serial number (3 digits)}.

[0039] Exemplarily, the first test case ID of the second test subitem of the fifth test item of the function test of the second software configuration item in the test outline document of the 16th aircraft inertial navigation professional embedded software test project in June 2025: PRJ0102202506016-TSD-V1.2-SCI02-FNT-TST005-TSI002-STC001.

[0040] ⑦Code file ID (File_ID): {Project_ID}-SRC-{Hash value (8 bits) of the relative path of the code file in the version library}, for example: PRJ2024100101-SRC-a1b2c3d4.

[0041] ⑧Class ID (Class): {File_ID}-CLA-{Class name}-{Starting line number}_{Ending line number}.

[0042] ⑨Function ID (Function): {File_ID}-FUN-{Function name}-{Starting line number}_{Ending line number}.

[0043] ⑩Code snippet ID (Code Snippet): {File_ID}-CDS-{Starting line number}_{Ending line number}.

[0044] ⑪Test requirement point ID (TRP_ID): {Doc_ID}-{Chapter number}-TRP{Sequence number (3 digits)}.

[0045] For example, the first requirement point ID of section 3.2.2 of the requirement specification document of the 16th aircraft inertial navigation professional embedded software test project in June 2025: PRJ0102202506016-SRS-V1.2-3.2.2-TRP001.

[0046] 3) Identification processing: all the documents and code assets stored in the library are parsed, and their unique IDs are automatically generated according to the above ID coding standard, and embedded as metadata in the corresponding data model of the knowledge base.

[0047] (2) Collect and organize the development documents and software codes of multiple historical projects, project test documents, test document templates, and other materials to form data assets; (3) After classifying and screening and standardizing the project development documents and project test documents of each project in the data assets according to the first set of standards, test case text files are formed.

[0048] Among them, the first set of standards specifies a standard specification for hierarchical classification and standardization processing of project data materials (project development documents, project test documents) closely related to test cases. Specifically, it includes: 1) The first level classification is performed according to the type of software involved in the project, including embedded software, non-embedded software, and FPGA software. 2) The second level classification is performed according to the professional field of the measured object involved in the project, such as satellite control system professional, aircraft inertial navigation professional, etc. 3) Third level classification according to test type, such as function test, performance test, interface test, security test, margin test, boundary test, capacity test, etc. 4) Screening typical representative project complete document materials (project development documents, project test documents) with consistent expert recognition of review to form a high-quality test case data asset set after standardized preprocessing; Among them, the standardized preprocessing mainly includes error checking and correction, professional term unification, file format unification, hierarchical classification annotation, unique ID generation and embedding, etc.

[0049] 5) For each project in the high-quality test case data asset set, extract the key core data related to the test case from the project file, and construct a test case text candidate file in a unified format.

[0050] Among them, the key core data related to the test case is extracted from the project file in order to store structured and clean test case text and its associated metadata in the test case text library, eliminate irrelevant redundant data, and support efficient and accurate semantic retrieval.

[0051] Among them, the test case candidate file in a unified format is a MARKDOWN format file named according to a unified naming format, which mainly extracts and standardizes the test case related data in the project (in particular, all tables in the document are converted to MARKDOWN format and added with titles and description texts, then replaced in situ; for the pictures (flowchart, timing diagram, etc.) in the document, tools combined with OCR are used for recognition and key information extraction, and then converted to corresponding text description and replaced in situ; for the formulas in the document, the formulas are recognized and key information is extracted, and then converted to LaTeX code format and replaced in situ).

[0052] Exemplarily, each test case related data is structured and processed according to the following standard format: { Test case name: [test case name in test case design single in test description] Test case ID: [test case unique ID number] Test requirement trace: [requirement document name and ID]-[position annotation] -[test requirement point name and ID] -[test requirement point description] Test outline trace: [test outline document name and ID]-[position annotation] -[software configuration item name] -[test type] - [Test item name and ID] - [Test item description] - [Test sub-item name and ID] - [Test method] Software source code tracking: [Code file name and ID] - [Function name and ID] - [Corresponding code comments] Test case description: [Corresponding test description] Test case initialization: [Initialization of test case] Preconditions and constraints: [Preconditions and constraints for test case execution] Termination conditions: [Normal termination conditions and abnormal termination conditions] Test process description (including input and operation description, expected test results, evaluation criteria for each test step): - [Step number 1] [Input and operation description] [Expected test results] [Evaluation criteria] - [Step number 2] [Input and operation description] [Expected test results] [Evaluation criteria] - [......] [......] [......] [......] - [Step number n] [Input and operation description] [Expected test results] [Evaluation criteria] Test case keyword identification: [Test case keywords] } The uniform naming format of the test case candidate file is: file name + unique ID. The file name is: software type _ professional field _ project software name _ test type.

[0053] For example: Embedded software _ satellite control system professional _ attitude control software _ function test _ PRJ0102202506016-TDD-V1.0.docx.

[0054] 6) For each project in the high-quality test case data asset set, respectively add comments to the project software source code files according to the second set of standards to form software code files.

[0055] The second set of standards for adding comments to project software source code files is to parse the project software source code files, and rewrite the original code comments of each source code file according to the file-level code comment standard format and function-level code comment standard format and add them to the beginning of the file and in front of the function. In particular, for the original source code files without comments, comments need to be added at the corresponding positions of the code files according to the code comment standards.

[0056] Exemplarily, the file-level code comment standard format is as follows: { [filename (with extension)] - [file function overview] Copyright (c) [date (YYYY-MM-DD)] [unit / author]. All rights reserved. Module Description: [Detailed description of the main function of the file, positioning in the system, core business logic] [Explanation of the file's position in the architecture and upstream dependencies] Dependencies: - Internal dependencies: [List of internal modules / classes called] - External dependencies: [List of third-party libraries / external system dependencies] Modification History: [YYYY-MM-DD] [Author] [Change Description] [YYYY-MM-DD] [Author] [Change Description] Examples: [Simple usage examples / call scenarios] } Exemplarily, the function-level code comment standard format is as follows: { [Function function summary] Detailed description: [Function specific behavior, algorithm implementation points, boundary condition description, etc.] Parameters: param1 (type): [Parameter description, including value range / constraints] param2 (type): [Parameter description, optional parameters must be marked with default value] Return: return_type: [Return value description, including special return value meaning] Exceptions: [ExceptionType]: [Trigger condition description] Call relationship: - Caller: [Which modules / functions call this function] - Called: [Key functions called inside this function] Examples: >>> function_name(arg1, arg2) [Expected output] } 7)Vectorization of test cases in the test case text candidate file, similarity calculation and analysis, screening of test cases with similarity higher than the threshold, manual analysis and expert review confirmation, deletion of redundant test cases, formation of high-quality test case text file.

[0057] (4) After segmenting (Chunking), embedding vectorization and index construction of the test case text file, a test case text library is formed, and it is retrieved and evaluated.

[0058] Among them, the segmentation (Chunking) of the test case text file is to divide each complete test case data in the test case text file into a knowledge block for text segmentation, form a text block, and add corresponding keywords and metadata (such as professional field, test type, identification ID, etc.).

[0059] Among them, the embedding is to use the Embedding model (such as bge-large-zh-v1.5, etc.) to convert the text block into a vector form, which is convenient for calculating the similarity.

[0060] Among them, the index construction is to store the vectorized text block into an efficient vector database (such as Milvus, Pinecone, etc.). At the same time, the metadata of each text block (test case) is stored in the database for quick retrieval.

[0061] Among them, the retrieval and evaluation of the test case text library include: 1)Formulate test case text library retrieval strategy: mixed retrieval. Combine semantic search (calculate the cosine similarity of query vector and Chunk vector) and keyword search (BM25 algorithm). After weighted fusion (such as 0.7 × semantic score + 0.3 × keyword score) of the results of the two, reordering is performed.

[0062] 2)Rerank: Use cross-encoder model (such as bge-reranker-large) to perform fine sorting on the preliminary retrieval results, recall the top K (such as Top 10) Chunk, and improve the relevance of the top N results.

[0063] 3)Evaluation criteria ① Retrieval accuracy: build a test query set, manually annotate the standard answer, calculate MRR@K (Mean Reciprocal Rank) and Hit Rate@K.

[0064] Among them, MRR@K (Mean Reciprocal Rank) and Hit Rate@K (Hit Rate) are the core indicators of retrieval accuracy evaluation, both of which are calculated based on "test query set + artificial annotation standard answer" and are used to measure the accuracy and recall effect of retrieval results.

[0065] Among them, MRR@K (Mean Reciprocal Rank) and Hit Rate@K (Hit Rate) are the core indicators of retrieval accuracy evaluation, both of which are calculated based on "test query set + artificial annotation standard answer" and are used to measure the accuracy and recall effect of retrieval results.

[0066] MRR@K calculation steps and formula are as follows: First, calculate the reciprocal rank (RR_q) of a single query: ; Among them, r q is the ranking order of the first relevant result.

[0067] Second, take the average of RR q of all N queries, which is MRR@K. The calculation formula is: ; Among them, the Hit Rate@K focuses on "whether there is at least one relevant result in the Top-K results of each query", and the core logic is "score 1 if there is, and score 0 if there is not", without considering the ranking of relevant results.

[0068] Hit Rate@K calculation steps and formula are as follows: First, calculate the hit value (Hit q ) ; R q indicates the results related to the query.

[0069] Second, take the average of Hit q of all N queries, which is Hit Rate@K. The calculation formula is: ; ② Association integrity: random sampling inspection to evaluate whether the retrieved text blocks carry 100% correct association metadata.

[0070] ③ Domain relevance: domain experts evaluate the domain relevance of retrieval results and queries and score (1-5 points).

[0071] (5) Analyze the development documents, software test documents and software source code files of each project, extract the test case association relationship, construct a hierarchical test case tree, store it in a graph database, and establish a test case association graph library.

[0072] The test case association relationship mainly includes the software configuration item, test item, test sub-item to which the test case belongs, the tracking relationship between the test case and the test requirement, the tracking relationship between the test case and the test outline, the tracking relationship between the test case and the software source code, the association relationship between the test cases, the association relationship between the test requirements, etc., effectively realizing bidirectional traceability.

[0073] The test case association graph library is a hierarchical structure graph database, which represents the hierarchical association relationship between the software configuration item, test type, test item, test sub-item, test case, source code fragment / function and test requirement point in the form of a tree graph, effectively revealing the multi-level association and tracking relationship between "test requirement-test item-test case-software code". Its core function is to display the visual association relationship and structured hierarchy through graphics, thereby assisting the test personnel in quickly designing test cases, ensuring the completeness of test case coverage and improving the efficiency of test case design.

[0074] The test case association graph library construction module includes: graph model design, knowledge and relationship extraction, storage and retrieval, and evaluation criteria.

[0075] 1) Graph model design. Node types: Requirement, TestItem, TestSubItem, TestCase, SourceFile, Function, TestType, ConfigurationItem.

[0076] Relationship types: validates, contains, belongs_to, depends_on, maps_to. Examples are as follows: (Requirement:TRP-001)-[:VALIDATED_BY]->(TestCase:STC-005), (TestItem:TST-001)-[:CONTAINS]->(TestSubItem:TSI-001), (TestCase:STC-005)-[:MAPS_TO]-> (Function:FUC-789).

[0077] 2) Knowledge and relationship extraction: Extract structured relationships and related knowledge from requirement specification (SRS), test outline (TSD), test description (TDD), etc. documents, and build a test tree.

[0078] Among them, the test tree is a hierarchical structure tree relationship diagram. For example, through the contains relationship, the software configuration item (ConfigurationItem) -> test type (TestType) -> test item (TestItem) -> test sub-item (TestSubItem) -> test case (TestCase) is organized into a hierarchical tree structure, and the test case (TestCase) node will be connected to the requirement (Requirement) node through the validates relationship and to the source code file (SourceFile) / function (Function) node through the maps_to relationship.

[0079] 3) Storage and retrieval: Use a graph database (such as Neo4j, etc.) to store all nodes and relationships, and provide a Cypher query language interface to support complex retrieval (for example: "Find all test cases that validate requirement TRP-001, and expand the code functions corresponding to these cases").

[0080] 4) Evaluation criteria: ① Relationship accuracy: manually sample and verify the correctness of the relationships in the graph. ② Graph completeness: check whether all core entities (such as test cases in all test description documents) are 100% extracted and imported into the graph database. ③ Query performance: test the response time of common traceability queries.

[0081] (6) Design code file metadata model and function metadata model, initialize code library directory structure according to unified standard format, and perform metadata annotation and database processing on software code files and functions according to project software type, professional field, project name, and identification ID, etc. to form a software code library.

[0082] Among them, the code file metadata model and function metadata model are used to describe software code files and functions according to their software type, professional field, project name, etc. attributes, facilitating their query, management, and reuse, etc.

[0083] For example, the code file metadata model: { File name: [file name (including extension)] File ID: [unique file identification ID] Function summary: [main function summary] Project: [project name] Software Type: [Embedded Software / FPGA Software / Non-embedded Software] Professional Field: [XXX Professional] File Path: [e.g., service / auth] Creation Time: [YYYY-MM-DD] Version Number: [Vxx.xx.xx.xx] Programming Language: [C or C++ or Java or Python, etc.] Development Environment: [Software Development Environment, e.g., Python 2.7.18, etc.] Included Function IDs: ["FUNC-1001", "FUNC-1002",...] Included Class IDs: ["CLASS-2001", "CLASS-2002",...] Running Environment: [Software Running Environment] Development Unit: [Development Unit Name] Last Modification Time: [YYYY-MM-DD] Requirement Trace: [Associated Requirement Document Name and ID] - [Requirement Point Description] [Associated Requirement Document Name and ID] - [Requirement Point Description] Use Case Trace: [Associated Test Case Name and ID] - [Test Case Description] [Associated Test Case Name and ID] - [Test Case Description] File Annotation: [Corresponding File-Level Code Annotation] } Exemplarily, Function Metadata Model: { Function ID: [Unique Function ID] Function Name: [Defined Function Name] Belonging Code File: [Belonging Code File ID] [Start Line Number]: [Function Code Start Line Number] [End Line Number]: [Function Code End Line Number] [Parameter List]: {Parameter Name: [Corresponding Parameter 1 Name], Parameter Type: [Parameter Type], Default Value: [Corresponding Parameter Default Value]}, {Parameter Name: [Corresponding Parameter 2 Name], Parameter Type: [Parameter Type], Default Value: [Corresponding Parameter Default Value]}, {Parameter Name: [Corresponding Parameter N Name], Parameter Type: [Parameter Type], Default Value: [Corresponding Parameter Default Value]}, ], Return Type: [Function Return Type] Function Annotation: [Corresponding Function-Level Code Annotation] } Among them, the code library directory structure is initialized according to the unified standard format, which specifically includes source code file directory, test code file directory, document directory, tool script directory, metadata storage directory, configuration file directory, etc.

[0084] Exemplarily, the code library directory structure is as follows: code_repo / ├── src / # Source code ├── tests / # Test code ├── docs / # Document ├── scripts / # Tool script │ ├── ast_parser.py # AST parser │ └── index_builder.py # Index builder ├── metadata / # Metadata storage │ ├── file_index.json # File index │ └── function_index.json # Function index └── config.py # Configuration file Among them, the software code library adopts structured storage (source code, documents, metadata, etc. are stored separately, and source code is parsed and stored by file, function, class, and code block), supports keyword and semantic similarity retrieval after metadata slicing and vectorization, supports source code parsing of multiple languages (for example, the tree-sitter tool can be used, which supports syntax parsing of multiple languages such as Python, Java, C, C++, Go, and JavaScript, provides syntax definitions for each language, generates syntax trees (AST), and the specific process is: traverse project source code -> use tree-sitter to parse each file -> traverse AST -> identify function definitions, class definitions, etc. -> extract its name, parameters, starting line number, ending line number, comments, etc. -> store in a structured code library), supports ID retrieval (each code file, class, function, and code snippet has a unique ID), supports text search (keyword search function name, variable name, comment, etc.), provides API linkage interface for test case text library and test case graph library to call, and realizes cross-library linkage between multiple libraries.

[0085] (7) The test document template in the data asset is classified, sorted and standardized according to the third set of standards to build a test document template library.

[0086] Among them, the third set of standards specifies the standard specifications for hierarchical classification and standardized processing of various test document templates. Specifically, it includes: 1) The first level classification is performed according to the type of software of the measured object involved in the project, including embedded software, non-embedded software, and FPGA software; 2) The second level classification is performed according to the professional field of the measured object involved in the project, such as satellite control system professional and aircraft inertial navigation professional; 3) The various test document templates of each professional field are standardized according to the relevant requirements to form standard test document templates; Among them, the standardized processing is mainly to unify and standardize the cover, signature page, table of contents, chapter architecture, main content of each chapter, template file format (such as.docx format) of various test document templates (including software test outline template, software test description template, software test record template, software test report template, etc.) according to the characteristics and requirements of each professional field, and to perform hierarchical classification and annotation, forming standard test document templates.

[0087] The test document template library provides various standardized test document templates of each professional field, which are used for generating software test outline documents, software test description documents, software test record documents, software test report documents, and other software test process documents.

[0088] After the hierarchical multi-modal professional domain knowledge base is constructed, the requirement specification, software design specification, development document and software source code file of the to-be-tested project are preprocessed to form the standardized to-be-tested project text file and software source code file.

[0089] The preprocessing of the development document of the to-be-tested project mainly includes: 1) All tables in the document are converted into MARKDOWN format and added with title and description text, and then replaced in situ; 2) The pictures in the document need to be recognized and key information extracted by using tools combined with OCR, and then replaced in situ after being converted into corresponding text description; 3) For the formulas in the document, the formulas are recognized and key information is extracted, and then replaced in situ after being converted into LaTeX code format; 4) Error checking and correction, and professional terminology unification are performed on the document; 5) The document is reviewed whether it meets the requirements of GJB438C and other standards, and the contents that do not meet the requirements need to be communicated and confirmed with the development party, and then revised into a document that meets the requirements of GJB438C and other standards; 6) The document is classified and labeled in two levels according to the type of the software to be tested and the professional field; 7) The file is named according to the unified naming format (software type_professional field_project software name); 8) The unique identification ID of each document is generated; 9) The format of the document is standardized to the.docx WORD document.

[0090] The preprocessing of the software source code file of the to-be-tested project mainly includes: 1) The project software source code file is parsed, and the original code comments of each source code file are rewritten according to the file-level code comment standard format and the function-level code comment standard format, and are added to the beginning of the file and in front of the function, respectively. In particular, for the original source code file without comments, comments need to be added at the corresponding position of the code file according to the code comment standard.

[0091] 2) The syntax errors, logical errors and the like in the software source code file are checked; 3) The code file ID, class ID, function ID and code snippet ID are generated and labeled.

[0092] The preprocessing module 102 is configured to preprocess the project development document and the software source code file of the to-be-tested project to form the standardized project development document and the software source code file of the to-be-tested project. Specifically, the preprocessing module includes a development document preprocessing module and a code processing module.

[0093] The development document preprocessing module comprises a table conversion submodule, a picture conversion submodule, a formula conversion submodule, a standard conversion submodule, a classification annotation submodule, a naming submodule, an identification generation submodule, and a format standardization submodule.

[0094] Specifically, the table conversion submodule is configured to convert all tables in the project development document of the to-be-tested project, including the requirement specification, the software design specification, the development general requirement, and the technical scheme of the to-be-tested project, into the MARKDOWN format, add a title and a description text, and then replace the original tables in situ. The picture conversion submodule is configured to replace the original pictures in the project development document of the to-be-tested project in situ after recognizing and extracting key information from the pictures and converting the key information into corresponding text descriptions. The formula conversion submodule is configured to replace the original formulas in the project development document of the to-be-tested project in situ after recognizing and extracting key information and converting the key information into LaTeX code format. The standard conversion submodule is configured to review whether the project development document of the to-be-tested project meets the requirements of the GJB438C standard and the like. For the content that does not meet the standard requirements, the content is recompiled into a project development document that meets the standard requirements after being communicated and negotiated with the developer. The classification annotation submodule is configured to perform two-level hierarchical classification annotation on the project development document of the to-be-tested project according to the software type and the professional field of the to-be-tested object. The naming submodule is configured to name the project development document of the to-be-tested project according to a unified naming format, wherein the unified naming format comprises software type_professional field_project software name. The identification generation submodule is configured to generate a unique identification ID of the project development document of the to-be-tested project. The format standardization submodule is configured to standardize the format of the project development document of the to-be-tested project into a.docx WORD document.

[0095] The code processing module comprises an annotation submodule, an error checking submodule, and an annotation submodule.

[0096] Specifically, the annotation submodule is configured to parse the software source code file of the to-be-tested project. The original code annotations of each software source code file are rewritten according to the file-level code annotation standard format and the function-level code annotation standard format, and are added to the beginning of the software source code file and in front of the function, respectively. The software source code file that originally has no annotations is added with annotations at the corresponding positions of the software source code file according to the code annotation standard. The error checking submodule is configured to check the syntax errors and logic errors in the software source code file of the to-be-tested project. The annotation submodule is configured to generate and annotate the software source code file ID, the class ID, the function ID, and the code snippet ID of the to-be-tested project.

[0097] The test case generation module 103 is configured to generate test cases and software test instructions based on the multi-modal professional knowledge base, the project development document of the to-be-tested item and the software source code file.

[0098] The test case generation module can include a test requirement set construction module 401, a test requirement-code segment / function set construction module 402, a first test tree graph construction module 403, a test outline generation module 404, a second test tree graph construction module 405 and a use case generation module 406.

[0099] The test requirement set construction module 401 is configured to call a large model to segment the project development document of the to-be-tested item, and extract the test requirement points and their descriptions involved in the software configuration item under test, respectively, and include them in {requirement specification test requirement set}, {software test instruction test requirement set}, {development general requirement test requirement set} and {technical scheme test requirement set}, and perform similarity analysis, merging and labeling on the requirement points in each requirement set to form an {original test requirement set} of the to-be-tested item.

[0100] The large model (LLM) is a model in the field of artificial intelligence (AI) based on deep learning technology, which focuses on processing and generating human language. Its core feature is to understand language semantics, grammar rules, and generate text content that conforms to human expression habits through large-scale text data training.

[0101] The large model is called to segment and analyze the development document of the to-be-tested item in detail, taking into account the context window (i.e. maximum token) limitation of the large model, to effectively avoid token explosion. The document is divided into multiple {document segments} according to chapter structure, paragraph structure, semantic combination and keyword similarity, and then the large model is called to analyze and process each segment one by one.

[0102] The extraction of the test requirement points related to the functions, performance, interfaces, security, etc. of each software configuration item under test mainly includes: 1) Analyze the document as a whole and generate a document summary to form {context background knowledge}; 2) Input the {context background knowledge} and the corresponding {document segment} into the dynamic prompt word construction engine, which generates new complete prompt words by assembling tasks according to the current test requirement points; 3) Input the prompt words output by the dynamic prompt word construction engine into the large model to generate test requirement point names, test requirement point descriptions and location annotations related to the corresponding document segments; The description of the test requirement should be as consistent as possible with the description in the original document, and the text length should be controlled to facilitate subsequent processing. For example, for test requirement descriptions in the original document that are less than 200 words long, the original content can be directly copied; for requirement descriptions that are more than 200 words long, a summary compression process should be performed to extract the core elements (subject, action, parameters, etc.) to form a specific test requirement description.

[0103] The location annotation is used to mark the test requirement point at the corresponding position of the {document fragment} in the document (such as chapter number, paragraph number, and page range) to facilitate subsequent tracking.

[0104] 4) Generate a unique identifier (ID) for each test requirement; 5) Establish a unified standard format (structured) for test requirements; For example, the unified standard format for test requirement points can be set as follows: { Test requirement name: [Test requirement name] Test Requirement ID: [Unique identifier ID number for the test requirement] Test requirement tracking location annotation: [Document Name and ID] - [Chapter Number] - [Paragraph Number] - [Page Range] Test requirement description: [Detailed description of the test requirement] } 6) The generated test requirements are included in test point sets such as {functional test requirement set}, {performance test requirement set}, {interface test requirement set}, and {security test requirement set} according to the test type. 7) Repeat steps 3) through 6) until all document fragments have been parsed and the corresponding test requirements have been generated.

[0105] The analysis, merging, and annotation mainly involve vectorizing the test requirement points in each test requirement set, calculating and analyzing their similarity, merging test requirement points with high similarity (similarity value greater than threshold k) that appear in different test requirement sets, and annotating their source before merging to improve the traceability of multiple sources of requirement points and reduce the redundancy of test requirement sets.

[0106] Test Requirements - Code Snippets / Function Set Construction Module 402 is used to generate a mapping relationship between requirement point IDs and code snippet / function IDs based on the software source code files of the project to be tested and the original test requirement set, so as to form a test requirement - code snippet / function set.

[0107] The large model is called to parse the software source code files and the {original test requirement set} of the test project and generate a "requirement point ID-code snippet / function ID" mapping relationship. The code understanding ability and requirement matching ability of the large model are used to establish the mapping relationship between the code and the requirements, that is, the task is dynamically assembled according to the current test requirement point-code snippet / function mapping relationship to generate a new complete prompt word input into the large model to generate a "requirement point-function / code snippet" correspondence table, and then the possible missed requirement points are found.

[0108] The first test tree graph construction module 403 generates a first-stage test tree graph based on each test requirement point in the test requirement-code snippet / function set and the corresponding code snippet / function. Specifically, the first test tree graph construction module is configured to take each test requirement point in the test requirement-code snippet / function set and the code snippet / function corresponding to each test requirement point as the first input, and simultaneously perform parallel retrieval from the test case text library, the test case association graph library, and the software code library to obtain a first retrieval result; the first retrieval result is weighted and fused, and then input into the dynamic prompt word construction engine, and the dynamic prompt word construction engine dynamically assembles a task according to the current test tree graph to form a first complete prompt word; the first complete prompt word is input into the large model to generate a corresponding test tree graph, which is audited and confirmed through a human-computer interaction interface to form a first-stage test tree graph.

[0109] The test case text library adopts a hybrid retrieval strategy, combining semantic search (calculating the cosine similarity of query vectors and Chunk vectors) and keyword search, reordering (Rerank) the results after weighted fusion (such as 0.7 x semantic score + 0.3 x keyword score), and recalling the top K (such as Top 5) most relevant results. Rerank uses a cross-encoder model (such as bge-reranker-large) to fine-sort the preliminary retrieval results, recall the top K Chunks, and improve the relevance of the top N results.

[0110] The dynamic prompt word construction engine can automatically select different roles and modules according to different task requirements, and dynamically assemble the complete large model prompt word in MARKDOWN format that is adapted to the current task according to the standardized prompt word template architecture. The specific steps include: 1) Select the main roles, auxiliary roles, and corresponding ability / behavior criteria required to complete the task from the system role library according to the task requirements, and insert them into the corresponding positions of the prompt word template.

[0111] The system role library is a database of various different role definitions and their corresponding ability / behavior criteria associations built and maintained in the system, each role is a "domain expert" with specific skills and perspectives.

[0112] Exemplarily: ① Role: Test Requirements Analyst.

[0113] Corresponding task requirements: Test requirements analysis and related tasks.

[0114] Ability / behavior criteria: Good at interpreting and analyzing software requirement specification, software design specification, development of technical scheme, etc. development documents, can accurately identify functional requirements, non-functional requirements (performance, interface, security), business rules, input and output constraints, potential business states and transitions. Output structured, unambiguous requirements and their descriptions.

[0115] ② Role: Test Case Designer.

[0116] Corresponding task requirements: Test case generation related tasks.

[0117] Ability / behavior criteria: Good at writing high-quality, executable specific test cases, proficient in various test case design methods (equivalence class partitioning, boundary value analysis, decision table, cause-effect diagram method, etc.), can design detailed test steps, test data, preconditions and expected results, etc.

[0118] ③ Role: Code Analysis Expert.

[0119] Corresponding task requirements: Code analysis, code comment completion / correction, etc. code related tasks.

[0120] Ability / behavior criteria: Good at static code analysis, can understand code logic, identify key functions, branch conditions, exception handling logic, interface calls and potential risk code segments, can complete / modify file-level and function-level code comments.

[0121] 2) Insert the task input (Task Input) into the corresponding position of the prompt word template.

[0122] 3) Insert the contextual knowledge (Contextual Knowledge) into the corresponding position of the prompt word template.

[0123] 4) Task thinking and execution planning. This part requires intention recognition and thinking on specific input tasks, and splits the task into subtasks in the business workflow, and selects the execution-adapted prompt word submodule for different subtasks.

[0124] Among them, the prompt word submodule (Prompt Submodule) is a prompt word module (including task output requirement module) that can dynamically adapt to the selection and assembly of different task execution plans. Each prompt word submodule defines how to subdivide and analyze tasks and step-by-step reasoning according to different tasks / subtasks in the form of thought chain (Chain-of-Thought, COT) and thought tree (Tree-of-Thought, TOT), and gives detailed reasoning requirements and process descriptions for intermediate steps.

[0125] 5) Association and traceability: This part is to associate the generated content with the requirement ID, code ID, etc. provided in the background knowledge, and ensure effective tracking.

[0126] 6) Review and optimization: Review whether the generated content meets the input task requirements, whether it fully considers multiple scenarios in combination with background knowledge, whether it uses professional and unambiguous test terms, and optimize and improve the content that does not meet the requirements, etc.

[0127] 7) Output requirements: Select the appropriate task output requirement prompt word submodule (Output Prompt Submodule) for different task inputs (Task Input), and insert the prompt word template into the corresponding position. For different tasks, specify the output / non-output content, output format (such as JOSN format, etc.), ID identification according to unified specifications, and prohibit output of irrelevant content, etc.

[0128] 8) Specific example: List one or two typical instances of the corresponding task (Specific Example).

[0129] Exemplarily, the prompt word template architecture is as follows: { ## You are a senior {role}, {ability / behavior standard} ## Task input: {Task _Input} ## Contextual knowledge: {Contextual_Knowledge} ## Task thinking and execution planning: Please think and plan according to the following steps, and finally complete {Task _Input}.

[0130] ### 1.{Prompt Submodule}......; ### 2.{Prompt Submodule}......; ### 3.{Prompt Submodule}......; ## Association and Traceability: - Requirement Association: Requirement ID: xxxx - Code Fragment / Function Association: Code ID: xxxx ## Review and Optimization: Please review the generated content to ensure it meets the requirements of the input task, considers various scenarios (such as normal and abnormal, boundary conditions in test case generation tasks), uses professional and unambiguous test terminology, and optimizes any discrepancies.

[0131] ## Output Requirements: {Output Prompt Submodule} ## Specific Example: {Specific Example} } Reference Figure 2 , the first-stage test tree graph is a hierarchical tree structure that includes test items, test sub-items, test types, software configuration items under test, and the traceability between test items and test requirements, code fragments / functions.

[0132] Among them, the human-computer interaction interface is the window for user and system interaction, which can effectively pass the user input to the system and display the system output to the user in various forms. The human-computer interaction interface can display the test tree graph generated by the large model, support the test personnel to mark whether it meets the requirements in the graph, support the addition, deletion and modification of each node in the graph, and support the test personnel to give feedback scores on the generated content. For test items or test sub-items that do not meet the requirements, they can be directly edited and modified by manual intervention or returned to the previous step to be regenerated by the large model until all test items and test sub-items in the test tree meet the requirements. In particular, the test personnel can adjust the prompt words input into the large model through the human-computer interaction interface, so that the large model can better complete the tasks of test tree graph generation, test outline generation, test case generation, etc. according to the latest intentions of the test personnel.

[0133] The test outline generation module 404 is configured to generate a software test outline based on the test items, test sub-items, and corresponding test requirement points and code segments / functions in the first-stage test tree graph. Specifically, the test outline generation module is configured to take each test item, test sub-item, and corresponding test requirement point and code segment / function in the first-stage test tree graph as a second input, perform parallel retrieval from the test case text library, test case association graph library, and software code library to obtain a second retrieval result, perform weighted fusion on the second retrieval result, input the second retrieval result into the dynamic prompt word construction engine, dynamically assemble the dynamic prompt word construction engine according to the current test outline generation task to form a second complete prompt word, input the second complete prompt word into the large model to generate a preliminary draft test outline, and perform review and confirmation through the human-computer interaction interface (in particular, for large model generated content that does not meet the requirements in the preliminary draft test outline, manual editing and modification can be performed on the human-computer interaction interface, or the large model can be selected to regenerate the content, and then the regenerated content can be selected and reviewed by a human being). A test document template library corresponding to the professional field of the test outline template is called to generate a software test outline document of the to-be-tested project, and the software test outline document is reviewed, reviewed, and modified through the human-computer interaction interface to form a final version of the software test outline.

[0134] The test outline includes the names of the software configuration items to be tested, the names of the test items, the test item IDs, the test item descriptions, the names of the test sub-items, the test sub-item IDs, the test method descriptions corresponding to the test sub-items, and the tracking relationships between the test sub-items and the development documents.

[0135] The second test tree graph construction module 405 is configured to generate a second-stage test tree graph based on the first-stage test tree graph, the test item descriptions in the software test outline, and the test method descriptions corresponding to the test sub-items. Specifically, the second test tree graph construction module is configured to take the first-stage test tree graph, the test item descriptions extracted from the final version of the software test outline, and the test method descriptions corresponding to the test sub-items as a third input, perform parallel retrieval from the test case text library, test case association graph library, and software code library to obtain a third retrieval result, perform weighted fusion on the third retrieval result, input the third retrieval result into the dynamic prompt word construction engine, dynamically assemble the dynamic prompt word construction engine according to the test tree graph generation task of the to-be-tested project to form a third complete prompt word, input the third complete prompt word into the large model to generate a corresponding test tree graph, and perform review and confirmation through the human-computer interaction interface to form a final version of the second-stage test tree graph.

[0136] Reference Figure 3 The second-stage test tree graph is generated based on the first-stage test tree graph and further includes test case nodes corresponding to each test sub-item and the tracking relationships between the test case nodes and the test requirement points and code segments / functions.

[0137] A use case generation module 406 is configured to form corresponding test cases based on the second-stage test tree graph, the test item description in the software test outline, and the test method description corresponding to the test sub-item, and then evaluate the quality of the test cases to generate the final version of the test cases. Specifically, the test case generation module is configured to take the second-stage test tree graph, the test item description extracted from the software test outline, and the test method description corresponding to the test sub-item as the fourth input, perform parallel retrieval from the test case text library, the test case association graph library, and the software code library to generate a fourth retrieval result, perform weighted fusion on the fourth retrieval result, and then input the dynamic prompt word construction engine. The dynamic prompt word construction engine assembles the test case generation task of the to-be-tested project dynamically to form a fourth complete prompt word. The fourth complete prompt word is input into the large model to generate corresponding test cases, and subjective evaluation (i.e., test personnel scoring and judgment), review and confirmation (in particular, test cases generated by the large model that do not meet the requirements can be directly edited and modified by manual intervention in the human-computer interaction interface or selected for re-generation by the large model and then manually selected and confirmed), and objective evaluation (i.e., evaluating the quality of the generated test cases from multiple dimensions such as the adoption rate of the generated test cases, the requirement coverage rate, and the executability) are performed through the human-computer interaction interface to form the final version of the second-stage test tree graph. The test description template of the corresponding professional field in the software test document template library is called to generate the software test description document of the to-be-tested project, which is manually reviewed and expertly reviewed, and modified according to the expert review opinions to form the final version of the software test description.

[0138] In the test description, the software configuration item name, the test type, the test case name, the test case identifier, the test outline tracking relationship, the test description, the test case initialization, the premise and constraint, the termination condition, the test process (including input and operation description, expected test result, evaluation criterion), and the like are included.

[0139] The test personnel scoring and judgment and the review and confirmation are subjective evaluations of the generated test cases from the correctness, relevance, standardization, logical rationality, scene applicability, context consistency, and the like of the generated content, which lay a data foundation for subsequent collection of human feedback data for reinforcement learning to further optimize and improve the effect and capability of the large model in test case generation in different professional fields.

[0140] The quality of the generated test cases is evaluated from multiple dimensions such as the adoption rate of the generated test cases, the requirement coverage rate, and the executability, including: 1) Calculate the test case adoption rate. The adoption rate is the proportion of the number of directly adopted and slightly modified adopted test cases to the total number of generated test cases, and the calculation formula is: ; wherein N1 represents the number of directly adopted test cases, N2 represents the number of slightly modified adopted test cases, and N represents the total number of generated test cases.

[0141] 2) Calculate the requirement coverage rate. The requirement coverage rate is calculated by tracking the proportion of the number of test requirement points that can be tracked and covered in the generated test cases to the total number of test requirement points, and the calculation formula is: ; wherein R1 represents the number of test requirement points that can be tracked and covered in the generated test cases, and R represents the total number of test requirement points.

[0142] 3) Calculate the executability. The executability is the proportion of the number of test cases with direct executable conditions in the total number of test cases, and the calculation formula is: ; wherein M represents the number of test cases with direct executable conditions in the generated test cases, and N represents the total number of test cases.

[0143] 4) Comprehensive evaluation. The evaluation results of the three dimensions of test case adoption rate, requirement coverage rate, and executability are weighted and fused to realize comprehensive evaluation of the quality of test cases, and the specific calculation formula is: ; wherein w1 represents the weight of the test case adoption rate index, w2 represents the weight of the requirement coverage rate index, and w3 represents the weight of the executability index. Exemplarily, w1 can take a value of 0.3, w2 can take a value of 0.4, and w3 can take a value of 0.3.

[0144] Compared with the prior art, the test case generation module provided in the embodiment enhances the ability of the large model to generate test cases based on development documents and software source codes by introducing a high-quality hierarchical multi-modal professional domain knowledge base combined with a dynamic prompt word construction engine. The construction of a high-quality hierarchical multi-modal professional domain knowledge base realizes the knowledge-based and deep structured reuse of test assets, significantly improving the test design efficiency. The construction of a full-link bidirectional tracking relationship of "test requirement-test item-test case-software code" significantly improves the completeness and pertinence of test coverage. Through dynamic hybrid parallel retrieval and dynamic adaptive prompt word engineering, the professional, accuracy and explainability of the generated test cases are enhanced.

[0145] The test execution module 104 is configured to prepare an execution condition according to the test case and the software test description, and then execute the software test steps in sequence according to the input and operation description steps of the test process description to obtain the execution result of the test case.

[0146] The cause analysis and tracing module 105 includes a retrieval submodule, a dynamic prompt word construction engine, a large model, and a report generation submodule. The retrieval submodule is configured to input the test cases that fail in the execution results of the test cases into the test case text library, the test case association graph library, and the software code library respectively and perform parallel retrieval. The dynamic prompt word construction engine is configured to input the retrieval results after weighted fusion into the dynamic prompt word construction engine, dynamically assemble complete result prompt words according to the current test case failure cause analysis and tracing task, and form a human-computer interaction interface. The large model is configured to input the complete result prompt words into the large model to generate corresponding analysis results. The report generation submodule is configured to enable the test personnel to judge and make auxiliary decisions on the analysis results, summarize the overall execution of the test cases, and form a problem report and a test report.

[0147] Each test case in the final version of the test specification document is prepared with an execution condition such as test case initialization, prerequisites, and constraints, and then executed in sequence according to the input and operation description steps of the test process description. The actual test results of each step are recorded, and the execution of each actual test result is judged (pass or fail) according to the evaluation criteria. Finally, the execution result of the entire test case is obtained.

[0148] For example, the execution of one of the test cases designed for the "user login authentication function" requirement point described in the software requirement specification is shown in Table 1.

[0149] Table 1

[0150] The test case that fails is input, and parallel retrieval is performed from the test case text library, the test case association graph library, and the software code library. The retrieval results are input into the dynamic prompt word construction engine after weighted fusion. The dynamic prompt word construction engine dynamically assembles new complete prompt words according to the current test case failure cause analysis and tracing task, inputs the new complete prompt words into the large model to generate corresponding analysis results (including problem name, problem source, problem category, problem level, problem description, etc.), and displays the analysis results to the test personnel through the human-computer interaction interface for judgment and auxiliary decision making. The overall execution of the test cases is summarized, and a problem report and a test report are formed.

[0151] The test work problems mainly include test case problems, test data problems, execution step problems, and test environment problems.

[0152] The problem categories mainly include code problems, document problems, and design problems.

[0153] Among them, the problem level usually includes: fatal problem (A-level problem), serious problem (B-level problem), general problem (C-level problem), and slight problem (D-level problem).

[0154] The cause analysis and tracing module takes the failed test case as input, simultaneously performs parallel retrieval from the test case text library, the test case association graph library, and the software code library, and inputs the retrieval results into the dynamic prompt word construction engine after weighted fusion. The dynamic prompt word construction engine dynamically assembles new complete prompt words according to the current test case failure cause analysis and tracing task to form a new complete prompt word input into the large model to generate a corresponding analysis result.

[0155] After quality assessment and industry expert review of the data assets (development documents and software source code, test outline, test description, etc.) of the project, the data assets that meet the requirements are included in the high-quality hierarchical multi-modal professional knowledge base, and the knowledge is continuously accumulated and deposited.

[0156] Among them, quality assessment is an objective and subjective assessment of the quality of documents, test cases, professional field representation, and expert recognition in data assets. Objective assessment is calculated and evaluated by a large model combined with related tools, and subjective assessment is scored and evaluated by test personnel and field experts.

[0157] Reference Figure 5 The test system based on the multi-modal knowledge base includes software and hardware, wherein the software mainly includes: a data asset preprocessing module, a test case text library construction module, a test case association graph library construction module, a software code library construction module, a test document template library construction module, a test requirement-code fragment / function set construction module, a dynamic prompt word construction engine module, a test tree graph construction module, a test outline generation module, a test case generation module, a cause analysis and tracing module, a test case quality evaluation module, and a human-computer interaction interface module. The hardware mainly includes a server end and a client end.

[0158] The data asset preprocessing module is used to classify and filter and normalize the historically accumulated project development documents and test documents according to a first set of standards to form test case text files, and to uniformly add annotations to software source code files according to a second set of standards to form software code files.

[0159] The test case text library construction module is used to segment (Chunking), embed (Embedding), vectorize, and index the test case text files to form a test case text library.

[0160] The test case association graph library construction module is used for analyzing and extracting test case association relationships from software test document files and software source code files, constructing a hierarchical test case tree, storing in a graph database, establishing a test case association graph library, and supporting efficient relationship query and traceability.

[0161] The software code library construction module is used for initializing software code directories according to a unified standard format, and performing metadata annotation and library processing on software code files and functions included therein according to the software type, professional field, project name and identification ID of the project, to form a software code library.

[0162] The test document template library construction module is used for constructing a test document template library after classifying, sorting and standardizing test document templates according to a third set of standards.

[0163] The test requirement-code snippet / function set construction module is used for calling a large model to separately segment and analyze software requirement specification documents, software design specifications, development requirements, technical solutions and other development documents of a to-be-tested project, and extract test requirement points and their descriptions of functions, performance, interfaces, security and the like involved in each software configuration item of the to-be-tested software, to construct a {requirement specification test requirement set}, a {software design specification test requirement set}, a {development requirement test requirement set} and a {technical solution test requirement set}, and perform similarity analysis, merging and annotation of requirement points in the requirement sets to construct an {original test requirement set} of the project. The large model is called to analyze the software source code files of the to-be-tested project and the {original test requirement set} and generate a "requirement point ID-code snippet / function ID" mapping relationship, and combined with manual verification, the test requirement points missing in the {original test requirement set} are completed and annotated to form a complete {test requirement-code snippet / function set} of the project.

[0164] The dynamic prompt word construction engine module is used for dynamically assembling each part to construct a large model prompt word according to different task requirements. The prompt word template component includes roles and ability / behavior criteria, task input, context background knowledge, task thinking and execution planning, association and traceability, review and optimization, output requirements, specific examples and the like.

[0165] The test tree graph construction module takes each test requirement point and its corresponding code function / snippet in the {test requirement-code snippet / function set} as input, or takes the first-stage test tree graph, test item descriptions extracted from the software test outline, and test method descriptions corresponding to test sub-items as input, simultaneously performs parallel retrieval from the test case text library, the test case association graph library and the software code library, and inputs the retrieval results after weighted fusion to the dynamic prompt word construction engine, which generates a new complete prompt word input large model according to the current test tree graph to generate a corresponding test tree graph.

[0166] The test outline generation module extracts each test item, test sub-item, and its corresponding test requirement point and code segment / function in the first-stage test tree graph as input, simultaneously performs parallel retrieval from the test case text library, test case association graph library, and software code library, and inputs the weighted fusion of the retrieval results into the dynamic prompt word construction engine, which dynamically assembles new complete prompt words according to the current test outline generation task to form a new complete prompt word input into the large model to generate the corresponding test outline.

[0167] The test case generation module inputs the second-stage test tree graph, test item description extracted from the software test outline, and test sub-item corresponding test method description as input, simultaneously performs parallel retrieval from the test case text library, test case association graph library, and software code library, and inputs the weighted fusion of the retrieval results into the dynamic prompt word construction engine, which dynamically assembles new complete prompt words according to the current test case generation task to form a new complete prompt word input into the large model to generate the corresponding test case.

[0168] The cause analysis and tracing module inputs the failed test case as input, simultaneously performs parallel retrieval from the test case text library, test case association graph library, and software code library, and inputs the weighted fusion of the retrieval results into the dynamic prompt word construction engine, which dynamically assembles new complete prompt words according to the current test case failure reason analysis and tracing task to form a new complete prompt word input into the large model to generate the corresponding analysis result.

[0169] The test case quality evaluation module evaluates the quality of the generated test case from multiple dimensions such as test case adoption rate, requirement coverage, and executability, and through flexible setting of the weights of each indicator, the evaluation results of test case adoption rate, requirement coverage, and executability are weighted and fused to realize comprehensive evaluation of test case quality.

[0170] The human-computer interaction interface module is the window for user and system interaction, which can effectively pass the user input to the system and display the system output to the user in various forms. The human-computer interaction interface module can display the test tree graph generated by the large model, support the test personnel to mark whether it meets the requirements in the graph, support the addition, deletion, and modification of each node in the graph, and support the test personnel to give feedback on the generated content. For the generated content that does not meet the requirements, it can be directly edited and modified by manual intervention or regenerated by the large model according to the situation, until the relevant content meets the requirements. In particular, the test personnel can adjust the prompt words input into the large model through the human-computer interaction interface, so that the large model can better complete the tasks of test tree graph generation, test outline generation, test case generation, etc. according to the latest intentions of the test personnel.

[0171] The server end includes computing power resources (CPU, GPU card, etc.), storage resources (memory, disk array, etc.), software resources (operating system, middleware, database, etc.), network resources (firewall, switch, router, etc.), etc. The test case generation system software adopts a B / S architecture, which is deployed on a remote server together with the multi-modal professional knowledge base resources. In addition, considering the data security problem, a large model and its inference framework are also privately deployed on the server. Test personnel remotely access the human-computer interaction interface of the system through a browser on the internal local area network, and the data generated in the interaction process is stored in the database on the server.

[0172] A browser is installed on the client, which is a user terminal for test personnel to access the test case generation system on the remote server and realize effective interaction through the human-computer interaction interface. Compared with the prior art, the present application can at least realize one of the following beneficial effects: 1. A high-quality hierarchical multi-modal professional knowledge base is constructed, which realizes the knowledgeization and deep structured reuse of test assets, and significantly improves the test design efficiency. The present application first systematically standardizes and deeply structures the scattered, heterogeneous, and multi-modal data assets (requirement specification, source code, test outline, test case, etc.) in the software testing field. By constructing a multi-modal associated knowledge base (test case text library, test case association graph library, software code library, and test document template library), the originally isolated, manually reviewed, and understood document materials are transformed into structured knowledge that can be understood, retrieved, and reasoned by machines. When designing new test cases, test personnel can efficiently recall and reuse high-quality test assets and domain knowledge from historical projects.

[0173] 2. A full-link bidirectional tracking relationship of "test requirement-test item-test case-software code" is constructed, which greatly improves the completeness and pertinence of test coverage. The association between traditional test cases and requirements and code is often one-way, loose, or even broken. The present application establishes a bidirectional traceable association relationship between test cases and requirement points and source code segments / functions through fine-grained metadata model design and knowledge extraction technology during the knowledge base construction phase. During test case generation, the large model can automatically bind the newly generated test case with specific requirement ID and function ID based on the retrieved associated knowledge. The direct benefits are: a) the coverage of test cases on requirements and code can be quantitatively evaluated to ensure no omissions; b) when requirements or code change, the affected test cases that need to be modified can be accurately located, assisting regression testing; c) the generated test cases are more targeted and can directly correspond to specific code logic, improving defect detection rate; d) it can help detect redundant code and find potential software problems such as code backdoors.

[0174] 3. The problem of knowledge extraction and utilization of multi-modal test documents (such as figures, tables, and formulas) is solved, and the completeness and accuracy of the domain knowledge base are improved. For tables, flowcharts, timing diagrams, formulas, and other non-text elements rich in information in test documents, the invention innovatively uses a combination of OCR recognition and additional text description to convert them into standardized structured text descriptions and replace them in situ. This technical means solves the pain point that traditional text vectorization methods cannot effectively process image information, enabling subsequent semantic retrieval to fully utilize these key information, avoiding knowledge loss due to neglecting charts and formulas, and ensuring that the constructed knowledge base can more completely and accurately represent the entire logic of the software system.

[0175] 4. By dynamically mixing parallel retrieval and dynamically adaptive prompt word engineering, the professionalism, accuracy, and interpretability of generated test cases are enhanced. The invention proposes simultaneous parallel retrieval from the test case text library, test case association graph library, and software code library, improving retrieval efficiency. In addition, different retrieval strategies and recall mechanisms are designed according to the characteristics of the test case text library, test case association graph library, and software code library. Furthermore, a dynamic prompt word construction engine that can adapt to different large model generation tasks is designed, embedding the retrieved relevant knowledge as contextual background knowledge into the corresponding positions in the prompt words, effectively guiding and constraining the large model to generate related content within the professional field. The benefits are: a) the generated test cases conform to relevant standards and project practices, and are highly professional; b) the generation of each test case has a clear and traceable knowledge source, improving the credibility and interpretability of the test activity; c) effectively reducing the "illusion" of the large model, and the generation result is more accurate and reliable.

[0176] 5. An interactive visualization confirmation interface is provided, realizing a human-machine collaborative test case iteration optimization closed loop, and ensuring the quality of the generated results. The invention visualizes the test cases generated by the system in a tree diagram form to the test personnel, allowing interactive confirmation, modification, supplementation, and optimization. The optimized results can be fed back to the knowledge base, realizing the continuous evolution of knowledge. This human-machine collaborative closed loop process perfectly combines the efficiency of machines with the experience and judgment of humans, utilizing AI to significantly improve the efficiency of preliminary design, and ensuring the high quality and high usability of the generated test cases through final human review, achieving dual improvement in efficiency and quality.

[0177] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium includes a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0178] The above merely describes preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A multi-modal knowledge base based testing system, characterized in that, Comprise: A knowledge base construction module for classifying and screening project development documents, software source code files, project test documents and software test document templates of historical projects and normalizing them to construct a hierarchical multi-modal professional domain knowledge base; A preprocessing module for preprocessing project development documents and software source code files of a to-be-tested project to form normalized project development documents and software source code files of the to-be-tested project; A test case generation module for generating test cases and their software test instructions based on the multi-modal professional domain knowledge base, the project development documents and the software source code files of the to-be-tested project; A test execution module for preparing execution conditions according to the test cases and their software test instructions, and then executing software test steps in sequence according to input and operation instruction steps described in the test process to obtain execution results of the test cases.

2. The multi-modal knowledge base based testing system of claim 1, wherein, The multi-modal professional domain knowledge base comprises a test case text library, a test case correlation graph library, and a software code library, The test system further comprises a cause analysis and tracing module, wherein the cause analysis and tracing module comprises a retrieval submodule, a dynamic prompt word construction engine, a large model, and a report generation submodule, The retrieval submodule is configured to input test cases that fail in the execution results into the test case text library, the test case correlation graph library, and the software code library respectively and perform parallel retrieval; The dynamic prompt word construction engine is configured to input the retrieval results after weighted fusion into the dynamic prompt word construction engine, dynamically assemble complete result prompt words according to the current test case failure cause analysis and tracing task, and input the complete result prompt words into the large model to generate corresponding analysis results and provide a human-computer interaction interface; and The report generation submodule is configured to enable a test personnel to judge and make auxiliary decisions on the analysis results through the human-computer interaction interface, summarize overall execution of the test cases, and form a problem report sheet and a test report. The analysis results comprise problem names, problem sources, problem categories, problem levels, and problem descriptions; 3. The multi-modal knowledge base based testing system of claim 2, wherein, The problem names comprise test case problems, test data problems, execution step problems, and test environment problems; The problem categories comprise code problems, document problems, and design problems; The problem levels comprise fatal problems defined as A-level problems, serious problems defined as B-level problems, general problems defined as C-level problems, and slight problems defined as D-level problems. The preprocessing module comprises a development document preprocessing module and a code processing module, wherein 4. The multi-modal knowledge base based testing system of claim 2, wherein, The development document preprocessing module comprises: A table conversion submodule configured to convert all tables in the project development documents of the to-be-tested project into MARKDOWN format, add title and description text, and replace in situ, wherein the project development documents of the to-be-tested project comprise requirement specification, software design specification, development general requirements, and technical solutions of the to-be-tested project; ​ a picture conversion submodule, configured to replace in situ the pictures in the project development document of the to-be-tested item after recognizing and extracting key information and converting the key information into corresponding textual descriptions by using OCR; a formula conversion submodule, configured to replace in situ the formulas in the project development document of the to-be-tested item after recognizing and extracting key information and converting the key information into LaTeX code format; a standard conversion submodule, configured to review whether the project development document of the to-be-tested item meets the standard requirements, and, for the content that does not meet the standard requirements, to negotiate with the developer and confirm the content, and then to revise the content into the project development document that meets the standard requirements; a classification and labeling submodule, configured to classify and label the project development document of the to-be-tested item according to two levels of software types and professional fields of the to-be-tested item; a naming submodule, configured to name the project development document of the to-be-tested item according to a unified naming format, wherein the unified naming format includes software type_professional field_project software name; an ID generation submodule, configured to generate a unique ID of the project development document of the to-be-tested item; a format standardization submodule, configured to standardize the format of the project development document of the to-be-tested item into a.docx WORD document; the code processing module includes: an annotation submodule, configured to parse the software source code file of the to-be-tested item, wherein the original code annotations of each software source code file are rewritten according to a file-level code annotation standard format and a function-level code annotation standard format, and are added to the beginning of the software source code file and in front of the function, respectively; and the software source code file that has no original annotations is added with annotations at corresponding positions in the software source code file according to a code annotation standard; an error checking submodule, configured to check syntax errors and logic errors in the software source code file of the to-be-tested item; a labeling submodule, configured to generate and label an ID of the software source code file of the to-be-tested item, a class ID, a function ID, and a code snippet ID.

5. The multi-modal knowledge base based testing system of claim 2, wherein, The knowledge base construction module further includes: an ID system construction module, configured to construct a unique ID system by using a hierarchical combination coding method; a data asset construction module, configured to collect and sort the project development documents, the software source code files, the project test documents, and the software test document templates of a plurality of historical projects to form data assets; a test case text file construction module, configured to classify and filter the project development documents and the project test documents of the projects in the data assets, and to perform standardization processing to constitute test case text files; a test case text library construction module, configured to segment, embed, vectorize, and index the test case text files to form a test case text library, and to perform retrieval and evaluation on the test case text library; a test case association graph library construction module, configured to analyze the development documents, the project test documents, and the software source code files of a plurality of projects, extract test case association relationships, construct a hierarchical test case tree, store the test case tree in a graph database, and establish the test case association graph library. The software code library construction module is configured to establish a code file metadata model and a function metadata model, initialize a code library directory structure in a unified standard format, and perform metadata labeling and database processing on the software source code files and functions included therein according to the software type, professional field, project name and unique identification ID of the project, so as to form the software code library. The software test document template library construction module is configured to sort and standardize software test document templates in the data asset, so as to construct a software test document template library.

6. The multi-modal knowledge base based testing system of claim 2, wherein, The test case generation module includes: The test requirement set construction module is configured to call a large model to segment the project development document of the to-be-tested project, extract test requirement points and descriptions involved in the software configuration item to be tested, and respectively include the test requirement points and descriptions in a requirement specification, a software test specification, a development general requirement and a technical scheme test requirement set, and perform similarity analysis, merging and labeling on the test requirement points in each requirement set to form an original test requirement set. The test requirement-code snippet / function set construction module is configured to generate a mapping relationship between requirement point IDs and code snippet / function IDs based on the software source code files of the to-be-tested project and the original test requirement set, so as to form a test requirement-code snippet / function set. The first test tree graph construction module is configured to generate a first-stage test tree graph based on each test requirement point in the test requirement-code snippet / function set and the corresponding code snippet / function. The test outline generation module is configured to form a software test outline based on each test item, test sub-item, test requirement point and code snippet / function in the first-stage test tree graph. The second test tree graph construction module is configured to form a second-stage test tree graph based on the first-stage test tree graph, test item descriptions in the software test outline and test method descriptions corresponding to test sub-items. The test case generation module is configured to form corresponding test cases based on the second-stage test tree graph, test item descriptions in the software test outline and test method descriptions corresponding to test sub-items, and then evaluate the quality of the test cases to generate final test cases.

7. The multi-modal knowledge base based testing system of claim 6, wherein, The first test tree graph construction module is configured to: use each test requirement point in the test requirement-code snippet / function set and the corresponding code snippet / function as a first input, and simultaneously perform parallel retrieval from the test case text library, the test case association graph library and the software code library to obtain a first retrieval result; perform weighted fusion on the first retrieval result, and then input the first retrieval result into a dynamic prompt word construction engine to form a first complete prompt word through the dynamic prompt word construction engine according to a current test tree graph generation task; input the first complete prompt word into a large model to generate a corresponding test tree graph, and perform review and confirmation through a man-machine interaction interface to form a first-stage test tree graph.

8. The multi-modal knowledge base based testing system of claim 7, wherein, The test outline generation module is configured to: Take each test item, test sub-item and its corresponding test requirement point and code segment / function in the first stage test tree graph as the second input, and perform parallel retrieval from the test case text library, the test case association graph library and the software code library to obtain a second retrieval result; The second retrieval result is weighted and fused, and then input into a dynamic prompt word construction engine. The dynamic prompt word construction engine generates task dynamic assembly according to the current test outline to form a second complete prompt word; The second complete prompt word is input into a large model to generate a preliminary draft test outline, and the human-computer interaction interface is used for auditing and confirming; The test document template library corresponding to the professional field of the test outline template is called to generate the software test outline document of the to-be-tested project, and the human-computer interaction interface is used for auditing, reviewing and modifying to form a final version of the software test outline.

9. The multi-modal knowledge base based testing system of claim 8, wherein, The second test tree graph construction module is configured to: Take the first stage test tree graph, the test item description extracted from the final version of the software test outline, and the test method description corresponding to the test sub-item as the third input, and perform parallel retrieval from the test case text library, the test case association graph library and the software code library to obtain a third retrieval result; The third retrieval result is weighted and fused, and then input into a dynamic prompt word construction engine. The dynamic prompt word construction engine generates task dynamic assembly according to the test tree graph of the to-be-tested project to form a third complete prompt word; The third complete prompt word is input into a large model to generate a corresponding test tree graph, and the human-computer interaction interface is used for auditing and confirming to form a final version of the second stage test tree graph.

10. The multi-modal knowledge base based testing system of claim 9, wherein, The test case generation module is configured to: Take the second stage test tree graph, the test item description extracted from the software test outline, and the test method description corresponding to the test sub-item as the fourth input, and perform parallel retrieval from the test case text library, the test case association graph library and the software code library to generate a fourth retrieval result; The fourth retrieval result is weighted and fused, and then input into a dynamic prompt word construction engine. The dynamic prompt word construction engine generates task dynamic assembly according to the test case of the to-be-tested project to form a fourth complete prompt word; The fourth complete prompt word is input into a large model to generate a corresponding test case, and the human-computer interaction interface is used for subjective evaluation, auditing and confirmation and objective evaluation to form a final version of the second stage test tree graph; The test document template library corresponding to the professional field of the test explanation template is called to generate the software test explanation document of the to-be-tested project, and the human-computer interaction interface is used for subjective evaluation, auditing and confirmation and objective evaluation to form a final version of the software test explanation.

Images