Method for automatically generating test case for domain knowledge intensive rule

Abstract

The invention discloses a method for automatically generating a test case for a domain knowledge-intensive rule, which is characterized by comprising the following steps of: constructing a standard test case library, constructing a context of the domain knowledge-intensive rule by using a hybrid retrieval algorithm, and automatically generating the test case of the domain knowledge-intensive rule by using a large language model. The method specifically comprises the steps of constructing a standard test case library, performing context retrieval, generating test cases and the like. Compared with the prior art, the method has the advantages that the rule automatic retrieval domain knowledge is combined with the large language model, the more accurate and comprehensive test case is generated, the accuracy and coverage rate of the test case are greatly improved, the method is simple and convenient, and the method is particularly suitable for various application scenes in which the test rule contains dense domain knowledge; the system can be used for testing financial security transaction systems, verifying medical diagnosis systems, testing legal compliance inspection systems and the like, and has good application prospects and commercial development values.

Description

Technical Field

[0001] The invention relates to the technical field of automatic test case generation, in particular to a method for automatically generating test cases for domain knowledge-intensive rules by utilizing a large language model and context retrieval. Background Art

[0002] In software engineering, automated test case generation technology is an important means to ensure the stability and correctness of software systems. Especially in complex fields such as finance, securities, medical care, and law, the generation and verification of test cases are crucial to software quality. Traditional test case generation methods mainly include manual writing, rule-based automatic generation, and model-based generation methods. These methods have certain effects when dealing with simple scenarios, but often show obvious limitations in domain knowledge-intensive rules.

[0003] Manually writing test cases relies on the domain knowledge and experience of the tester. Although it can generate relatively accurate test cases, the process is time-consuming and it is difficult to cover all possible scenarios, especially when dealing with boundary conditions. In addition, when faced with domain knowledge-intensive rules, testers often find it difficult to fully understand and handle these complex rules. Rule-based automatic generation methods generate test cases through a pre-defined rule base. Although they perform well in systems with simple and clear rules, their flexibility and adaptability are greatly reduced when dealing with complex business rules, and the maintenance and updating of the rule base requires a lot of human resources. Model-based generation methods generate test cases by building an abstract model of the system. However, in complex domains, building an accurate and complete model is itself a challenging task. Inaccurate or incomplete models will result in the generated test cases failing to fully cover all scenarios of domain knowledge-intensive rules.

[0004] In recent years, with the rapid development of natural language processing technology, large language models have shown great potential in the field of automated test case generation. With their excellent natural language understanding and generation capabilities, these models are able to generate test cases with high coverage. However, existing research shows that large language models still have obvious shortcomings when processing rules containing intensive domain knowledge. The main problem is that large language models lack a deep understanding of domain-specific terminology and complex rule logic, resulting in insufficient accuracy and coverage of generated test cases, making it difficult to effectively cover all scenarios and boundary conditions of the rules.

[0005] Therefore, when faced with domain knowledge-intensive rules, these methods have significant limitations and are difficult to meet the needs of high-standard testing.

[0006] The test case generation methods of the prior art have obvious limitations when dealing with rules that contain intensive domain knowledge. Manually written test cases are difficult to cover all scenarios efficiently, especially when dealing with complex boundary conditions and domain knowledge-intensive business rules. In addition, rule-based and model-based automatic generation methods lack flexibility and accuracy when dealing with complex domain knowledge, making it difficult to ensure that the generated test cases fully cover all possible scenarios. Although large language models show the potential to generate test cases with high coverage, they still have significant deficiencies in understanding and processing complex domain knowledge and rule logic, resulting in the generated test cases being difficult to meet high standards in terms of accuracy and comprehensiveness. Summary of the invention

[0007] The purpose of the present invention is to provide a method for automatically generating test cases for domain knowledge-intensive rules in response to the deficiencies of the prior art. The method adopts the method of building a standard test case library, using a hybrid retrieval algorithm, constructing the context of the domain knowledge-intensive rules, and using a large language model to automatically generate test cases for the domain knowledge-intensive rules. The method effectively combines the rule automatic retrieval of domain knowledge with the large language model to generate more accurate and comprehensive test cases to improve the accuracy and coverage of the test cases, so as to better adapt to the testing needs of domain knowledge-intensive fields and further improve the reliability and security of the system. The method is simple and is particularly suitable for various application scenarios in which test rules contain intensive domain knowledge, such as testing of financial securities trading systems, verification of medical diagnosis systems, testing of legal compliance inspection systems, etc. It can automatically generate high-coverage and high-precision test cases and has good application prospects and commercial development value.

[0008] The specific technical solution for achieving the purpose of the present invention is: a method for automatically generating test cases for domain knowledge-intensive rules, which is characterized by constructing a standard test case library, using a hybrid retrieval algorithm, constructing the context of the domain knowledge-intensive rules, and using a large language model to automatically generate test cases for the domain knowledge-intensive rules. The method mainly includes the following steps:

[0009] Step 1: Build a standard test case library

[0010] Extract relevant rules from the domain rule document, and filter these rules through semantic analysis and clustering technology to remove redundant and similar rules. In order to achieve this goal, the present invention adopts K-Means++ clustering algorithm to classify the rules, and selects the most representative rules from each category as the basic rules of the standard test case library. Then, a large language model (such as GPT-4) is used to generate preliminary test cases for the selected representative rules. These test cases cover the main scenarios and logic of the rules to ensure the verification of basic functions. To ensure the accuracy of the test cases, the present invention further introduces domain experts to proofread and optimize the preliminary generated test cases, and correct the possible understanding errors and generation deviations of the large language model. In addition, the present invention also deeply analyzes the rule documents, extracts and collects domain knowledge, and stores these terms together with rules and test cases in the standard test case library. In order to improve retrieval efficiency, the present invention vectorizes all rules, optimized test cases and domain knowledge, and stores them in a database with a multi-map structure so that subsequent association and retrieval can be performed quickly.

[0011] Step 2: Contextual Retrieval

[0012] When the user inputs a new rule, the present invention first performs grammatical and semantic analysis on the rule to identify key terms and important logical points. Based on these key terms, keyword matching is performed in the standard test case library to quickly find the domain knowledge and associated information related to the input rule. Subsequently, the input rule is vectorized through vector retrieval technology, and the cosine similarity measurement method is used to calculate the similarity between the input rule and the rule and test case vectors stored in the standard test case library. Several groups of rules and test cases that are most similar to the input rule are selected. Using the Few-Shot Learning technology, a small number of samples similar to the input rule are extracted from the standard test case library. These samples, together with the keyword matching results and system prompts, constitute a complete context, laying the foundation for the subsequent generation of test cases.

[0013] Step 3: Test case generation

[0014] After completing the domain-specific retrieval, the present invention combines the domain knowledge, a small number of samples and system prompts retrieved in step 2 to build a context environment with intensive domain knowledge. The context environment not only includes detailed information of the input rules, but also includes explanations of relevant domain knowledge and reference cases of a small number of similar rules. On this basis, a large language model (such as GPT-4) is used to automatically generate test cases that match the input rules. The generated test cases will cover the main scenarios, boundary conditions and extreme cases of the rules to ensure the accuracy and coverage of the test cases. In order to further improve the quality of the test cases, the present invention performs preliminary automatic verification of the generated test cases to ensure that they comply with the use specifications of the rule logic and domain knowledge. Users can also provide feedback on the generated test cases according to actual test requirements. The present invention will continue to optimize the generation strategy based on these feedbacks and adjust the standard test case library.

[0015] The present invention ensures the dynamic adaptability of the standard test case library and the synchronization of the latest rules by incorporating new domain knowledge, rules and their test cases into the standard test case library regularly or on demand. Through this continuous updating, the content in the library is always consistent with the latest business rules. The present invention can enhance the adaptability to new rules and changes by updating the rules, test cases and domain knowledge in the standard test case library when the business rules change, ensuring that the generated test cases always have high coverage and high accuracy.

[0016] Compared with the prior art, the present invention has the following significant beneficial effects:

[0017] 1) The present invention effectively combines rule-based automatic retrieval of domain knowledge with a large language model to generate more accurate and comprehensive test cases to improve the accuracy and coverage of test cases, thereby better adapting to the testing needs of domain knowledge-intensive fields and further improving the reliability and security of the system;

[0018] 2) The present invention can deeply understand and process complex domain knowledge, ensuring that the generated test cases fully cover all possible rule scenarios. It is particularly suitable for various application scenarios such as testing of financial securities trading systems, verification of medical diagnosis systems, and testing of legal compliance inspection systems. It can automatically generate high-coverage and high-precision test cases, and has good application prospects and commercial development value.

[0019] 3) Support the continuous updating of the standard test case library, so that the system can dynamically adapt to changes in business rules, reduce dependence on human resources, and reduce labor costs during the testing process. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Build flow charts for standard test case libraries;

[0021] Figure 2 This is the structure diagram of the standard test case database;

[0022] Figure 3 Retrieve schematic diagrams for context;

[0023] Figure 4 Retrieve flowcharts for context;

[0024] Figure 5 Flowchart generated for the test case. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below through specific implementations.

[0026] Example 1

[0027] The specific steps for generating large language model test cases based on domain knowledge intensive context learning are as follows:

[0028] Step 1: Construction of standard test case library

[0029] See also Figure 1 First, you need to provide the data to be stored. This data includes two parts: one is the representative rule-standard test case pairs extracted from the documents; the other is the domain-specific knowledge collected from the documents and the Internet. The purpose of storing representative rule-standard test case pairs is to enrich the diversity of examples in the standard test case database, thereby increasing the possibility of having examples similar to the input rules. On the contrary, if the stored rule-standard test case pairs are very similar, then even if the input rules change, the provided examples may remain consistent, resulting in poor prompting when used for large language model context learning. Collecting domain-specific knowledge helps large language models better understand the rules. These two parts together constitute the standard test case library.

[0030] See also Figure 2 , the constructed standard test case database is structured as a multiple mapping of domain knowledge consisting of n terms and n domain knowledge interpretations, and rules represented by a vector consisting of n rules.

[0031] See also Figure 1 , the construction process of the standard test case library is as follows:

[0032] 1.1: Representative rule screening

[0033] In semantic similarity analysis, similar sentences are close in position in the vector space, while dissimilar sentences are far apart. Similarly, encoded similar rules are also close in the vector space. The goal of the present invention is to filter out rules that are too similar and identify representative rules. Inspired by Auto-CoT i, the present invention applies semantic clustering to achieve the goal of obtaining representative rules. Auto-CoT proves that the vector representation of clustering problems helps to distinguish different problems.

[0034] For representative rules, we first use Sentence-BERTii to encode the entire rule and convert it into a vector representation. Then, the K-Means++iii clustering algorithm is applied to divide these vector representations into k clusters. For each cluster i, the rules are sorted according to the Euclidean distance of each pair from the cluster center. The process of representative rule screening is shown in Table 1 below:

[0035] Table 1 Representative rule screening algorithms

[0036]

[0037] 1.2: Standard test case writing

[0038] For the representative rules extracted by the screening algorithm, the goal of the present invention in this step is to write corresponding standard test cases (standard answers) for these rules. The present invention first uses GPT-4 to generate preliminary test cases for the representative rules. Then, the generated test cases are proofread by domain experts to correct any inaccuracies to ensure the accuracy of the standard test cases corresponding to the representative rules. Having domain experts proofread rather than directly write standard test cases significantly reduces their workload.

[0039] 1.3: Domain knowledge collection

[0040] Domain-specific knowledge is extracted by analyzing rule documents, Baidu Encyclopedia, online glossaries, etc. Taking the financial industry as an example, rule documents usually contain explanations of specific terms, and Baidu Encyclopedia and financial terminology websites also have explanations of domain-specific terms. These complex term explanations are collected and used together with the representative rule-standard test case pairs constructed in the standard test case writing step as the data content for building the database.

[0041] 1.4: Data Storage

[0042] See also Figure 2 From the above two tasks, the present invention obtains representative rule-standard test case pairs and domain-specific knowledge. In order to facilitate the next step of retrieval, the present invention uses the following method to construct a standard test case library:

[0043] 1.4.1: Representative rule-standard test case pairs are vectorized and represented in vector format. These vector formats capture the semantic relationship between rules and test cases, thus enabling efficient subsequent retrieval.

[0044] 1.4.2: Store domain-specific knowledge in a multi-key single-value (multi-map) format as a string. This approach effectively avoids information fragmentation or inconsistency caused by different expressions of proper nouns (e.g. different ways of saying or spelling the same proper noun). By unifying different expressions of the same concept, this approach ensures consistency of information during retrieval and use.

[0045] 1.4.3: Store the vector format and key-value pairs together in a comprehensive database. This joint storage method combines the advantages of vector retrieval and key-value retrieval, ensuring the efficiency of semantic retrieval and the consistency of proper noun expression.

[0046] Step 2: Contextual Retrieval

[0047] A standard test case library is obtained through step 1, which is denoted as C. In step 2, the present invention will focus on how to retrieve context P related to input rule r from the standard test case library C. R is used to represent the hybrid retriever, and F r represents the re-ranking function, the first N documents can be expressed as: D = R (r, C), and the retrieved context can be defined as: P = F r (D).

[0048] See also Figure 3 In order to retrieve a small number of sample examples and domain-specific information, the domain-specific retrieval process is divided into two parts: keyword matching of potential terms and hybrid retrieval of a small number of sample examples. This approach ensures that the retrieved data includes both relevant domain knowledge and examples that are instructive for the input rules, thereby more effectively supporting large language models for test case generation.

[0049] See also Figure 4 , the process of context retrieval is as follows:

[0050] 2.1: Keyword matching of potential terms

[0051] For input rules, it is crucial to determine the position of domain knowledge in the rules. Domain knowledge has a specific meaning and is used to describe and express specific concepts or phenomena in the domain. Compared with the traditional named entity recognition and entity classification combination algorithm, the present invention uses a large language model to identify the potential domain knowledge in the rules. This method greatly reduces the labor required for annotation while maintaining recognition accuracy. After determining the position of domain knowledge in the rules, the present invention extracts the domain knowledge and performs strict string matching with the key-value pairs in the multi-mapping part of the standard test case library. The matched key-value pairs will be used as context materials for the subsequent step three.

[0052] 2.2. Hybrid retrieval of a small number of samples

[0053] Another key component of the material used to form the knowledge-intensive context is the examples of the input rules (a small number of sample examples). The present invention constructs a hybrid retrieval algorithm to retrieve rules similar to the input rules from the standard test case library. The retrieved rule-standard test case pairs provide useful generation examples for the test case generation process. The hybrid retrieval mainly includes two parts: keyword search and vector search.

[0054] 2.2.1: Vector Search

[0055] Vector search aims to find semantically similar documents based on vector embeddings, given an input rule r and embedding it into a vector In the example, a set of rule-test case pairs is represented by a vector P, and the input rule vector and the rule-test case pair vector P i The cosine similarity of is defined by the following formula (a):

[0056]

[0057] in: is the vectorized representation of the input rule, and P is the vector representation of the rule-test case pair.

[0058] After calculating the cosine similarity of all vectors in the vector representation P, the first k vectors with the highest cosine values ​​are selected as candidates.

[0059] 2.2.2: Keyword search

[0060] Due to the importance of domain information in rules, the present invention combines keyword search with pure semantic vector search to better retrieve a small number of sample examples similar to the input rules. Keyword search identifies relevant content by matching specified keywords. The present invention uses BM25 as a keyword search method. The score of BM25 for query r and document (rule-test case pair) P is calculated by the following formula (b):

[0061]

[0062] in, is the term r in query r i The inverse document frequency of f(r i ,P) is the term r i is the term frequency in document P; |P is the length of document P (i.e. the number of terms in P); avgdl is the average document length in the corpus; k1 and b are free parameters.

[0063] The present invention needs to add more domain terms by expanding the vocabulary to facilitate keyword search. This ensures that domain-specific terms (such as "one-time declaration") are not split into "one-time" and "declaration", but remain as the complete term "one-time declaration". In this way, similar examples can be better retrieved in the standard test case library.

[0064] 2.2.3 Implementation of hybrid retrieval

[0065] The score of the hybrid search is calculated by the following formula (c):

[0066]

[0067] Among them, α and β are user-defined weighting factors whose sum is 1, Cosine is the score of the vector search semantic similarity mentioned above, and BM25 is the score of the keyword search document relevance mentioned above. After standardizing them respectively and multiplying them by the corresponding weighting factors, the final score of the hybrid retrieval can be obtained, and the final score is guaranteed to be in the range of [0,1], ensuring that the search results are more comprehensive and accurate.

[0068] Step 3: Test case generation

[0069] See also Figure 5, through step 2, rule-standard test case pairs similar to the input rules are obtained as few sample examples, and domain knowledge related to these rules is obtained. The next step is to combine these materials, input rules and system prompts to build a context. The present invention adopts a modular approach inspired by LangGPT to perform prompt engineering. This method builds a complete context based on the following modules: roles, goals, constraints on goals, workflows for achieving goals, and output styles assigned to large language models. This modular approach significantly improves the performance of large language models, thereby producing higher quality responses. This complete context is used as input to allow large language models to clarify their work goals and generate more accurate and comprehensive test cases. In addition, the present invention also adopts a reflection strategy to allow the large language model to judge the quality of the test cases it generates until it is satisfied, thereby ensuring the accuracy and coverage of the test cases. For example, in the field of financial securities, the present invention assigns a role of a securities trading rule testing expert to a large language model, whose goal is to generate test cases for given securities trading rules. By setting constraints such as covering all possible test scenarios and ensuring comprehensive coverage of equivalence classes and boundary values, the present invention demonstrates the workflow and specifies the output format, thereby creating a high-quality, modular context covering domain knowledge. After filling this module, it can be directly passed as input to the large language model to obtain the test cases generated by the large language model. For details, see the financial domain context template shown in Table 2 below:

[0070] Table 2 Financial domain context template

[0071]

[0072] Through the above-mentioned specific implementation in the financial field as an example, the present invention can deeply understand and process complex domain knowledge, ensuring that the generated test cases fully cover all possible rule scenarios. At the same time, it supports the continuous updating of the standard test case library, so that the system can dynamically adapt to changes in business rules, reduce dependence on human resources, and reduce labor costs during the testing process. The specific beneficial technical effects and significant technical advances are as follows:

[0073] 1) Dataset settings

[0074] Due to the commercial nature and high privacy requirements in the financial field, there is currently no publicly available benchmark dataset. The present invention classifies rules according to the density of domain knowledge: rules containing more than or equal to three domain knowledge are called domain knowledge intensive rules. The present invention created five datasets from official documents, each containing 200 test rules, and each dataset has a different ratio of domain knowledge intensive rules to simple rules. The density ranges from low to high, and these datasets are divided into three groups:

[0075] Group A is sparse domain knowledge:

[0076] Dataset a1: 20% domain knowledge-intensive rules;

[0077] Dataset a2: 40% domain knowledge-intensive rules.

[0078] Group B is the actual proportion:

[0079] Dataset b1: 50% domain knowledge intensive rules.

[0080] Group C is intensive domain knowledge:

[0081] Dataset c1: 60% domain knowledge-intensive rules;

[0082] Dataset c2: 80% of domain knowledge-intensive rules.

[0083] 2) Experimental methods

[0084] The experiments are divided into two categories. The first category is contextual learning. The main idea of ​​this category is to use different prompting methods to maximize the capabilities of the large language model. Based on the same large language model (ChatGPT-3.5), this paper compares some of the current mainstream methods, including Zero-shot, Few-shot, Chain-of-Thought, and Reflection. Chain-of-Thought has a good effect in dealing with complex problems.

[0085] The second category is to compare with the fine-tuned model. The main idea of ​​this category is to fine-tune the large language model to enhance its ability in specific tasks. In order to ensure the fairness of the experiment, the standard test case library used in this method has been fine-tuned into other large language models in advance. This method is based on the large language model (ChatGPT-3.5) and compares it with the four best performing models in the financial field, namely FinGPT, FinGLM, InternLM2-Instruct, and LLaMA3-Instruct.

[0086] 3) Experimental evaluation indicators

[0087] The evaluation index of this experiment is the coverage of the generated test cases under the correct premise. This experiment invited 3 experts who have been researching in the field of rule document testing in the financial field for three years to evaluate the experimental results. The full score of each rule is 1 point, and the generated test cases cover all possible situations. 1 point, otherwise it is scored according to the coverage. The total score of each data set is the average of all rule scores of the data set. In order to reduce the error of expert evaluation, the final experimental result takes the average of the scores of the three experts. The experimental results of the prompt-based method and the fine-tuning method and the present invention (DKI-ICL) are shown in Table 3 below:

[0088] Table 3 Comparison of hint-based methods and fine-tuning methods with DKI-ICL

[0089]

[0090] From Table 3 above, we can see that the present invention (DKI-ICL) shows obvious advantages in all five test data sets, and the coverage rate does not decrease with the enhancement of domain knowledge-intensive rules, and always maintains a high coverage rate range of 0.88-0.92. Therefore, it can be proved that the present invention can significantly improve the accuracy and coverage rate of automatic generation of test cases in domain knowledge-intensive fields.

[0091] The above specific implementations are only for further explanation of the present invention, and are not intended to limit the patent of the present invention. All equivalent implementations of the present invention should be included in the scope of the claims of the patent of the present invention.

Claims

1. A method for automatically generating test cases for domain knowledge-intensive rules, characterized in that: By building a standard test case library, using a hybrid retrieval algorithm, constructing the context of domain knowledge-intensive rules, and using a large language model to automatically generate test cases for domain knowledge-intensive rules, the method specifically includes the following steps: Step 1: Construction of standard test case library Extract relevant rules from domain rule documents, filter and classify the rules through semantic analysis and clustering algorithms, select representative rules, and use large language models to generate preliminary test cases. After proofreading and correction by domain experts, they are combined with domain knowledge to form a standard test case library; Step 2: Contextual Retrieval Perform grammatical and semantic analysis on the input new rules, identify key terms and important logical points, perform keyword matching in the standard test case library, use vector retrieval technology to calculate the similarity between the input rules and the rules and test cases stored in the standard test case library, obtain a small number of samples similar to the input rules, and combine them with the keyword matching results and system prompts to form a complete context; Step 3: Test case generation The context constructed above is used as the input for test case generation, and the large language model is used to automatically generate test cases for input rules.

2. The method for automatically generating test cases for domain knowledge-intensive rules according to claim 1, characterized in that: In the step of constructing the standard test case library, the K-Means++ clustering algorithm is used to classify the rules, and the initially generated test cases are proofread and optimized by domain experts to correct the understanding errors and generation deviations of the large language model.

3. The method for automatically generating test cases for domain knowledge-intensive rules according to claim 1, characterized in that: In the context retrieval step, keyword matching and hybrid retrieval algorithms are used to introduce the context of the retrieved domain knowledge-intensive rules into a scoring mechanism to limit the number of retrieved similar rules and the degree of similarity to the original rules within a set range.

4. The method for automatically generating test cases for domain knowledge-intensive rules according to claim 1, characterized in that: In the test case generation step, the large language model adopts a reflection strategy to automatically verify the generated test cases, correct the understanding errors and generation deviations of the large language model, and make them conform to the usage specifications of rule logic and domain knowledge.

5. The method for automatically generating test cases for domain knowledge-intensive rules according to claim 2, characterized in that: In the step of constructing the standard test case library, all rules, optimized test cases and domain knowledge are vectorized and stored in a database with a multi-mapping structure.

6. The method for automatically generating test cases for domain knowledge-intensive rules according to claim 1 or claim 3, characterized in that: The word list is expanded in the keyword matching so that the domain-specific terms are not segmented and remain complete terms.

7. The method for automatically generating test cases for domain knowledge-intensive rules according to claim 1 or claim 4, characterized in that: In the test case generation step, the user provides feedback on the generated test cases according to actual test requirements, and the feedback is used to continuously optimize the generation strategy to adjust the standard test case library.

8. The method for automatically generating test cases for domain knowledge-intensive rules according to claim 1 or claim 7, characterized in that: The standard test case library ensures the dynamic adaptability of the standard test case library and the synchronization with the latest rules by incorporating new domain knowledge, rules and their test cases into the standard test case library regularly or as needed.

Images