Software test data generation method, device, equipment, medium and program product
By optimizing the initial prompt text and embedding data constraints through a large language model (LLM) and knowledge base, test data that conforms to business logic is generated, solving the problems of low efficiency and low coverage in existing technologies and achieving efficient and low-cost test data generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA UNIONPAY
- Filing Date
- 2026-01-23
- Publication Date
- 2026-06-02
Smart Images
Figure CN122132300A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of software testing, and in particular relates to a method, apparatus, equipment, medium and program product for generating software test data. Background Technology
[0002] In the software development lifecycle, software testing is a crucial step in ensuring software quality, reliability, and security. Among these, test data preparation is the foundation and core of testing activities; high-quality, high-coverage test data is essential for discovering potential defects and verifying business logic. However, as software systems become increasingly complex and business scenarios become more diverse, traditional or mainstream test data generation methods face increasingly severe challenges.
[0003] Currently, the industry primarily acquires test data through manual construction, extraction from production databases, and generation using data generation tools. Manual construction suffers from low efficiency and poor test coverage. Extracting data from production databases is limited by data security and privacy regulations, severely restricting the range of real-world data available for testing and making it difficult to meet the needs of a wide range of testing scenarios. While data generation tools improve efficiency and coverage to some extent, they typically generate random or simple data, failing to simulate the diversity of real-world business scenarios and easily generating data that is illogical or invalid. Furthermore, they require the development of specific functionalities for each test scenario, leading to complex initial setup and high maintenance costs in the long run. Summary of the Invention
[0004] This application provides a software test data generation method, apparatus, device, medium, and program product that can automatically generate test data with high coverage and conforming to business logic at a low cost.
[0005] In a first aspect, embodiments of this application provide a method for generating software test data, including: Get a test data generation request. The test data generation request includes an initial prompt text, which is used to indicate the application system to be tested and the test scenario. The initial prompt text is optimized at least once using a large language model (LLM) to obtain an optimized prompt set, which includes the optimized prompts obtained in each round of optimization. Data constraint information related to each optimization suggestion is retrieved from the pre-defined knowledge base, and the data constraint information is embedded into the relevant optimization suggestions to obtain a set of constraint suggestions; Input the set of constraint hints into the LLM, and you will get the set of data generation scripts generated by the LLM; Based on the data generation script set, generate test data corresponding to the test data generation request.
[0006] Secondly, embodiments of this application provide a software test data generation apparatus, comprising: The request acquisition module is used to acquire test data generation requests. The test data generation requests include initial prompt text, which is used to indicate the application system to be tested and the test scenario. The prompt optimization module is used to perform at least one round of optimization on the initial prompt text using a large language model (LLM) to obtain an optimized prompt set, which includes the optimized prompts obtained in each round of optimization. The constraint embedding module is used to retrieve data constraint information related to each optimization suggestion from a preset knowledge base, and embed the data constraint information into the relevant optimization suggestions to obtain a set of constraint suggestions; The script generation module is used to input a set of constraint hints into the LLM and obtain a set of data generation scripts generated by the LLM. The data determination module is used to generate test data corresponding to the test data generation request based on the data generation script set.
[0007] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions; The processor implements a software test data generation method as described in the first aspect when executing computer program instructions.
[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the software test data generation method of the first aspect.
[0009] Fifthly, embodiments of this application provide a computer program product, wherein instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the software test data generation method as described in the first aspect. Fourthly, embodiments of this application provide a computer storage medium.
[0010] In this embodiment, a test data generation request is obtained. Based on LLM and a knowledge base, the initial prompt text in the test data generation request is optimized, and data constraints of the test data to be generated are embedded, resulting in a constraint-enabled prompt. Then, based on the powerful generalization capability of LLM, the data constraint information in the constraint-enabled prompt is understood, thereby generating a data generation script that can batch generate test data. The data generation script is used to batch generate the test data required by the request. According to this embodiment, test data can be automatically generated, achieving higher generation efficiency and test coverage compared to manual construction and data extraction methods. Furthermore, data constraints are defined during data generation, which improves the effectiveness of the generated test data and ensures that the test data conforms to the business logic of the application system under test and the test scenario. In addition, using LLM and a knowledge base for test data generation eliminates the need for specific development for different test scenarios, reducing development and maintenance costs. Attached Figure Description
[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a flowchart of a software test data generation method provided in some embodiments of this application; Figure 2 This is a schematic diagram of the workflow of the embedder provided in some embodiments of this application; Figure 3 This is a schematic diagram of the workflow of the generator and verifier provided in some embodiments of this application; Figure 4 This is a flowchart illustrating a software test data generation method provided in some embodiments of this application; Figure 5 This is a schematic diagram of the structure of a software test data generation device provided in some embodiments of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation
[0013] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0014] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.
[0015] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.
[0016] LLM (Large Language Model): A large language model.
[0017] Prompt: A prompt word. In the field of artificial intelligence, it is a command or keyword used to guide AI (Artificial Intelligence) models in generating output. In natural language processing and large models, it plays a core role in passing on input information context and parameters.
[0018] RAG (Retrieval-Augmented Generation): Retrieval enhancement generation. Used for retrieval enhancement of large models.
[0019] RaR (Rephrases and Response): A large-scale model prompt optimization process.
[0020] Embedding is a technique that converts data such as text and images into vectors.
[0021] Before providing a detailed description of the embodiments of this application, the relevant technologies are introduced. In the current computer software development ecosystem, there are three main software test design patterns: keyword-driven, object-driven, and data-driven. With the changing nature of business, software testing patterns have increasingly shifted to a new data-driven pattern, which parameterizes test cases by using accurate and meaningful combinations of real-world test input data. Test data has become a crucial infrastructure in the software testing field, and the industry employs various methods to acquire test data, broadly categorized as follows: manual acquisition, production data extraction, and conventional test data generation tools.
[0022] Manual data acquisition offers the advantages of direct control and the ability to precisely construct data for specific needs. However, it is inefficient in large-scale data testing scenarios. Furthermore, the preparation of test data often relies on the tester's experience and intuition, making it difficult to ensure that the test data comprehensively covers all functions and scenarios of the software system, resulting in poor test coverage.
[0023] Extracting data from production databases directly yields authentic and valid production data, accurately reflecting actual business scenarios. However, limitations in data volume and privacy protection make it difficult to clearly define sensitive data and standardize data anonymization methods during extraction. Furthermore, data applicable to testing scenarios is extremely limited due to data security and privacy regulations.
[0024] Generating test data using conventional tools offers advantages such as large data volume, broad test coverage, and high data diversity. However, these tools primarily rely on random data generation, which cannot effectively simulate the diversity of real user behavior and may generate data that is inconsistent with business logic or extremely invalid. Furthermore, these tools lack versatility, requiring the development of specific functions for each test scenario, which can lead to complex initial setup and high maintenance costs in the long run.
[0025] Therefore, the test data generation methods in related technologies suffer from low efficiency, low test coverage, and poor data quality. In view of this, to address the technical problems existing in related technologies, embodiments of this application provide a software test data generation method, apparatus, device, medium, and program product. The aim is to acquire test data for specified scenarios based on a Large Language Model (LLM) and a knowledge base, by constructing a cross-system, cross-business knowledge base and leveraging the generalization capabilities of LLM.
[0026] The software test data generation method provided in this application can be applied to software testing scenarios, including those applicable to online or batch software systems. The software test data generation method provided in this application will be described in detail below with reference to the accompanying drawings. It should be noted that the execution entity of the software test data generation method provided in this application can be a data generation platform. This application uses the example of a data generation platform executing the software test data generation method to illustrate the software test data generation method provided in this application.
[0027] See Figure 1 The above is a flowchart illustrating a software test data generation method provided in some embodiments of this application, such as... Figure 1 As shown, the method includes steps 110-150, which will be described in detail below.
[0028] Step 110. Obtain the test data generation request. The test data generation request includes initial prompt text, which is used to indicate the application system to be tested and the test scenario.
[0029] In some embodiments of this application, when software testing is required, a user can initiate a test data generation request to the data generation platform based on testing needs. The test data generation request includes initial prompt text. This initial prompt text is text input by the user (tester or related system) in natural language, describing the basic requirements of the test data based on test data requirements, test scenarios, and other criteria.
[0030] In some embodiments of this application, the initial prompt text is used to indicate the application system to be tested and the test scenario. For example, the initial prompt text may be "Please generate test sample data for system A in a regression test scenario", where "system A" is the application system to be tested.
[0031] The data generation platform responds to the test data generation request by retrieving the initial prompt text from the test data generation request.
[0032] Step 120. Optimize the initial prompt text at least once using a large language model (LLM) to obtain an optimized prompt set, which includes the optimized prompts obtained in each round of optimization.
[0033] LLM (Local Level Management) exhibits varying degrees of sensitivity to the format and content of prompts. Research indicates that even if different prompts are semantically similar, the final performance of LLM may differ depending on the prompt. Furthermore, the initial prompt text provided by the user is often arbitrary, vague, or incomplete. If LLM directly uses this initial prompt text to generate test data, the generated results will be unreliable and ineffective. Therefore, to mitigate the uncertainty of the final generated results, in this embodiment, after obtaining the initial prompt text from the request, LLM is first used to optimize the initial prompt text, resulting in an optimized prompt set. This transforms the unstructured and potentially low-quality initial prompt text input by the user into a standardized, structured set of instructions suitable for machine processing, capable of stably triggering LLM capabilities, and efficiently interfacing with knowledge bases.
[0034] In some embodiments of this application, step 120 may include steps 1201-1202.
[0035] Step 1201. Input the initial prompt text into the LLM, and perform N rounds of re-description on the initial prompt text through the LLM to obtain the optimized prompt generated in each round of re-description, where N≥1.
[0036] In some embodiments of this application, after obtaining the initial prompt text, the initial prompt text can be input into an LLM (Limited Language Management) system. The LLM then performs N rounds of optimization on the initial prompt text based on Rephrases and Response (RaR) in the prompt word engineering. RaR instructs the LLM to re-describe the initial prompt text after obtaining it, thereby enhancing its semantic clarity and reducing ambiguity.
[0037] In some embodiments of this application, to increase the diversity of optimized prompts, N can be an integer greater than 1. When performing N rounds of re-description on the initial prompt text using LLM, LLM can first perform a first round of re-description on the initial prompt text to obtain the first round of optimized prompts. Then, using the first round of optimized prompts as input, iteratively perform at least one round of re-description until the number of optimization rounds reaches N. Finally, all the optimized prompts obtained are combined into an optimized prompt combination.
[0038] For example, the initial prompt text is {{ prompt}}. When optimizing the initial prompt text using an LLM, the following prompt words can be input into the LLM to instruct it to perform RaR on the initial prompt text: "{{ prompt}} Rephrase and expand the question, and respond ” Step 1202. Take the set of N optimization suggestions generated by N rounds of re-description as the optimization suggestion set.
[0039] In the process of repeatedly describing the initial prompt text N times using LLM, after each round of optimization, the optimized prompt obtained in that round is denoted as . ,in Indicates the number of rounds. Finally, after N rounds of optimization, N optimization suggestions are obtained. The set of these N suggestions is denoted as the optimization suggestion set P. .
[0040] For each optimization suggestion in the set P, subsequent steps such as constraint embedding and script generation are performed concurrently.
[0041] Step 130. Retrieve data constraint information related to each optimization hint from the preset knowledge base, and embed the data constraint information into the relevant optimization hints to obtain a set of constraint-bearing hints.
[0042] Here, the knowledge base is a pre-built repository for storing test data constraint information for different application systems and different test scenarios. Data constraint information refers to a series of rules, conditions, specifications, or restrictions that the data must adhere to when generating software test data. These constraints ensure that the generated test data is meaningful, valid, and meets real business and technical requirements.
[0043] While LLM boasts strong generalization capabilities, it also suffers from high training costs and difficulty in tailoring training to each specific test scenario. Therefore, to enable LLM to generate corresponding test data for different application systems and test scenarios, data constraint information from the test data to be generated is used to guide the model in data generation.
[0044] The optimized prompts obtained above typically only contain basic descriptions of test data constraints. To further clarify the complex data constraints on the required test data in different test scenarios, this application constructs a knowledge base to store test data constraints for different test scenarios. Thus, during the generation of test data through LLM, data constraint information related to the test data to be generated can be obtained from the knowledge base using RAG technology, and this data constraint information can be embedded into the optimized prompts to obtain constraint-enabled prompts. These constraint-enabled prompts then guide LLM to generate valid test data that conforms to the business logic of the application system under test. Here, constraint-enabled prompts refer to optimized prompts that carry data constraint information.
[0045] In some embodiments of this application, the knowledge base may be constructed in the following manner: Receive test data constraint documents input by the user; after vectorizing the test data constraint documents, a knowledge base is formed.
[0046] Vectorizing the test data constraint document can involve converting each constraint description in the document into a high-dimensional vector. This places semantically similar constraints close together in the vector space. Based on this, when searching the knowledge base, the input question (such as optimization hints) can be converted into a query vector. The query vector can then be matched with the vectorized constraints stored in the knowledge base using similarity (e.g., cosine similarity). This allows for the precise retrieval of all data constraints related to the input question (e.g., those with similarity greater than a similarity threshold). Compared to methods like keyword matching, semantic matching using vector similarity avoids the problem of missed detections.
[0047] In some embodiments of this application, the test data constraint document may be a document compiled from prior experience gained in previous historical testing work for different application systems and different test scenarios.
[0048] In some embodiments of this application, embedding technology can be used to generate a high-dimensional vector for each constraint description in the test data constraint document.
[0049] In some embodiments of this application, data constraint information can be embedded in the optimization prompts using an embedder. See also Figure 2 This is a schematic diagram of the embedder's workflow. Based on the embedder, in step 130 above, steps 1301-130 can be executed in parallel for each optimization suggestion in the set of optimization suggestions, thereby obtaining the constraint-encoded suggestions corresponding to each optimization suggestion.
[0050] Step 1301 identifies the test intent of the optimization prompts to obtain the test scenarios and application systems to be tested corresponding to the optimization prompts.
[0051] In some embodiments of this application, LLM can be used to identify the test intent of optimization prompts in order to determine the application system and test scenario to be tested indicated by the optimization prompts.
[0052] Step 1302. Based on the test scenario and the application system under test corresponding to the optimization suggestion, search the knowledge base to obtain the data constraint information related to the optimization suggestion.
[0053] In some embodiments of this application, semantic matching is used to retrieve the knowledge base, thereby retrieving data constraint information related to the test scenario and the application system under test that corresponds to the optimization prompts from the knowledge base.
[0054] In some embodiments of this application, the knowledge base is a vectorized knowledge base, meaning each constraint in the knowledge base is a vectorized constraint. Based on this, when retrieving the knowledge base, the application system to be tested and the test scenario corresponding to the optimization prompt can be converted into query vectors. The similarity between the query vector and each vectorized constraint in the knowledge base (e.g., using cosine similarity) is calculated. Vectorized constraints with a similarity greater than a similarity threshold are identified as data constraint information related to the optimization prompt. The similarity threshold can be set according to the actual required retrieval accuracy.
[0055] Step 1303. Embed the data constraint information related to the optimization hint into the optimization hint to obtain the constraint hint corresponding to the optimization hint.
[0056] In some embodiments of this application, for each optimization suggestion in the optimization suggestion set, relevant data constraint information can be obtained from the knowledge base. Each optimization suggestion is then integrated with the relevant data constraint information to obtain a constrained suggestion corresponding to each optimization suggestion. The set of constrained suggestions corresponding to all optimization suggestions is defined as the constrained suggestion set. The constrained suggestion set is denoted as... , Where L represents the optimization hints obtained from the knowledge base. Relevant data constraint information.
[0057] Step 140. Input the set of constraint hints into the LLM to obtain the set of data generation scripts generated by the LLM.
[0058] The data generation script is a code script that can generate test data in batches.
[0059] In some embodiments of this application, for each constraint hint in the constraint hint set, a data generation script corresponding to that constraint hint can be generated using LLM based on the data constraint information in the constraint hint. The set of all data generation scripts corresponding to constraint hints in the constraint hint set is defined as the data generation script set. The number of data generation scripts in the data generation script set is the same as the number of constraint hints in the constraint hint set.
[0060] In some embodiments of this application, to guide the LLM to generate a data generation script based on constraint-enabled prompts, the constraint-enabled prompts can be embedded into a script generation prompt template to obtain corresponding script generation prompts. These prompts are then input into the LLM, thereby guiding the LLM to generate a data generation script based on the data constraint information in the constraint-enabled prompts. The script generation prompt template is an instruction template used to guide the LLM to generate a data generation script based on the data constraint information in the constraint-enabled prompts.
[0061] For example, the script generates a prompt word template as shown below: "List all constraints of {{ prompt}}. And write a complete pythonprogram which is able to generate a batch of test data all with allconstraints. Do not give any extra explanation except the program" In the script generation prompt template, `{{ prompt}}` indicates a constraint-enabled prompt. The script generation prompt generated based on this template instructs the LLM to "list all constraints in the constraint-enabled prompt `{{ prompt}}` and write a complete Python program that can generate a batch of test data satisfying all constraints." This Python program is the data generation script.
[0062] Step 150. Based on the data generation script set, generate test data corresponding to the test data generation request.
[0063] After obtaining the data generation script set, the data generation scripts in the data generation script set are used to generate test data that conforms to the application system and test scenario in the test data generation request.
[0064] In this embodiment, a test data generation request is obtained. Based on LLM and a knowledge base, the initial prompt text in the test data generation request is optimized, and data constraints of the test data to be generated are embedded, resulting in a constraint-enabled prompt. Then, based on the powerful generalization capability of LLM, the data constraint information in the constraint-enabled prompt is understood, thereby generating a data generation script that can batch generate test data. The data generation script is used to batch generate the test data required by the request. According to this embodiment, test data can be automatically generated, achieving higher generation efficiency and test coverage compared to manual construction and data extraction methods. Furthermore, data constraints are defined during data generation, which improves the effectiveness of the generated test data and ensures that the test data conforms to the business logic of the application system under test and the test scenario. In addition, using LLM and a knowledge base for test data generation eliminates the need for specific development for different test scenarios, reducing development and maintenance costs.
[0065] In some embodiments of this application, considering that LLMs exhibit a certain degree of illusion, and software testing scenarios have high requirements for test data, in order to address the uncertainties caused by the illusion of LLMs and ensure that the data generation script can execute normally and adapt to the testing environment, the data generation script can be verified to improve the reliability of the final test data. Specifically, step 150 may include the following steps 1501-1502.
[0066] Step 1501. Perform a verification operation on each data generation script in the data generation script set to obtain the verification result of each data generation script. The verification operation includes at least one of the following: verifying the executability of the data generation script and verifying the candidate test data generated based on the data generation script.
[0067] In some embodiments of this application, a validator and a generator can be used to validate the data generation script. See also Figure 3 This diagram illustrates the workflow of the generator and validator. The leftmost validator persists the data generation script into an executable file. The generator validator verifies whether the executable file can be executed successfully. The validator matrix is used to perform structural optimization and verification on the test data generated by the successfully executed executable file, ultimately yielding the verification result of the data generation script.
[0068] Specifically, step 1501 may include steps 15011-15015.
[0069] Step 15011. Save the data generation script as an executable file.
[0070] In some embodiments of this application, when the data generation script is a Python program, the data generation script can be persisted as a Python script file and the Python script file can be used as an executable file.
[0071] Step 15012: Execute the executable file using a script interpreter and obtain the execution result. The execution result indicates whether the executable file was executed successfully.
[0072] A script interpreter is a program used to execute executable files.
[0073] In some embodiments of this application, when the executable file is a Python script file, the Python interpreter can be invoked to execute the script file to verify its syntax correctness and operational feasibility, ensuring that the generated program has actual execution capability.
[0074] Step 15013. If the executable file executes successfully, load at least one validator corresponding to the test scenario of the data generation script.
[0075] In some embodiments of this application, the verifier is a verification rule designed based on data requirements extracted from daily testing work.
[0076] In some embodiments of this application, upon successful execution of the executable file, to ensure the consistency between the quality of the speed test data generated based on the data generation script and the business logic, a validator group corresponding to the test scenario of the data generation script is loaded. The validator group includes at least one validator to verify the aforementioned candidate test data. The validator group can be denoted as... , ,in, Represents the i-th validator instance in the set. This indicates the number of validators in this test scenario.
[0077] In some embodiments of this application, a pluggable validator matrix can be designed to achieve multi-dimensional automated verification of test data generated by the data generation script. This validator matrix adopts a modular architecture, with each group of validators corresponding to data verification rules in a specific scenario. The verification process can dynamically select the corresponding validator plugin based on a preset rule engine and the intent recognition of the LLM, using the MCP (Model Context Protocol) to achieve refined review of the generated test data.
[0078] Through a plug-in validator mechanism, when new verification requirements arise, only a new, independent validator plugin needs to be developed and inserted into the existing validator matrix, without modifying the core system code, thus improving the scalability of the verification process. Furthermore, once a validator is developed, it can be reused by any test scenario requiring data structure verification, avoiding redundant development and improving the reusability of the verification process. In addition, each validator is essentially a coded and automated representation of specific domain knowledge. For example, the environment adaptability validator embeds environment configuration knowledge from the CMDB. By chaining these validators into the data generation process, it is equivalent to systematically injecting the domain knowledge of human experts into the automated data generation process, which originally only relied on a general LLM, making the generation process business-aware and thus systematically ensuring the correctness of the generated test data at the syntactic, semantic, and business logic levels.
[0079] Step 15014. Verify the candidate test data generated based on the executable file using at least one verifier to obtain the verification result.
[0080] In some embodiments of this application, when the executable file is successfully executed, candidate test data generated based on the executable file can be obtained, and at least one verifier loaded in step 1503 above can be used to collaboratively verify the candidate test data.
[0081] In some embodiments of this application, the at least one verifier may include at least one of the following verifiers: a data structure verifier and an environment adaptability verifier.
[0082] Among them, the data structure validator is suitable for application scenarios that generate test data based on table structure. It is used to verify whether candidate test data has a compliant table structure based on the database in the database management tool.
[0083] In some embodiments of this application, an integrated database management tool is used to connect to the databases of various application systems in the test environment. An LLM (Limited Ledger Management) is employed for intent recognition, and a target database relevant to the test is automatically selected. Then, a data structure validator can be invoked to automatically construct SQL queries based on the table structure of the target database. These SQL queries are then used to query candidate test data, thereby determining the legality and consistency of the candidate test data's data structure based on the query results. Specifically, if the SQL query is successfully accepted and executed by the candidate test data, it proves that the field names, data types, lengths, and NOT NULL constraints of the candidate test data are compatible with the physical structure of the target database, thus confirming that its data structure is a compliant table structure. Conversely, if the SQL query fails to execute, it proves that the data structure of the candidate test data is inconsistent with the target database structure, thus determining that the candidate test data is invalid.
[0084] The environment adaptability verifier is suitable for scenarios where IP addresses and test users frequently change during testing. It verifies whether the environment configuration parameters in candidate test data conform to preset strategies and business expectations, based on the configuration information of the test environment in the Configuration Management Database (CMDB).
[0085] In some embodiments of this application, the configuration information of the test environment is uniformly maintained through a CMDB. The environment adaptability verifier can connect to the CMDB via an MCP Server to automatically verify whether the environment configuration information in the candidate test data conforms to preset strategies and business expectations.
[0086] Based on this, step 15014 above may include at least one of the following: The data structure validator compares the candidate test data with the table structure of the target database corresponding to the test scenario to verify whether the structure of the candidate test data is compliant. The environment adaptability verifier verifies whether the environment configuration parameters in the candidate test data conform to the preset strategy and business expectations, based on the configuration information of the test environment in the configuration management database.
[0087] In some embodiments of this application, the script interpreter can capture exception information in the executable file during execution. Based on this, the verification operation may further include: determining the exception information of the data generation script in the event of executable text execution failure; repairing the data generation script based on the exception information using LLM to obtain a repaired data generation script; and performing a verification operation on the repaired data generation script to obtain a verification result for the repaired data generation script.
[0088] This allows for the automatic repair of problems in the data generation script.
[0089] Step 15015. Based on the execution result and the verification result, generate the verification result of the data generation script.
[0090] In some embodiments of this application, for each data generation script, after obtaining its execution result and verification result, the verification result of the data generation script is determined based on the execution result and verification result. The verification result is used to indicate whether the data generation script can be successfully executed, and which verifiers can verify the candidate test data generated based on the data generation script.
[0091] Step 1502. Based on the candidate test data and verification results generated by each data generation script, determine the test data corresponding to the test data generation request.
[0092] In some embodiments of this application, verification results are generated based on each data generation script in the data generation script set, and the candidate test data generated based on each data generation script is rearranged to select high-quality test data that meets the test data requirements indicated in the test data generation request as the test data corresponding to the test data generation request.
[0093] In some embodiments of this application, step 1502 may include steps 15021-15024.
[0094] Step 15021. Based on the verification results of each data generation script, select candidate data generation scripts that have been successfully executed from at least one data generation script.
[0095] Step 15022. Count the number of successful candidates for each candidate data generation script.
[0096] Here, the number is the number of validators that the candidate test data generated by the candidate data generation script passes.
[0097] The higher the verification pass rate of the candidate data generation script, the more the test data generated based on the candidate data generation script conforms to the business logic of the test scenario, and the better the quality of the test data. Therefore, high-quality candidate test data can be selected based on the verification pass rate of the candidate data generation script as the test data corresponding to the test data generation request.
[0098] In some embodiments of this application, different optimization prompts are all generated based on the same initial prompt text. Therefore, the application system to be tested and the test scenario corresponding to each optimization prompt are consistent with the initial prompt text. Based on this, it can be determined that the validator group used in the verification operation of different data generation scripts is actually consistent, that is, the number of validators used is consistent. Therefore, the verification pass rate can be reflected by the number of passes corresponding to the candidate data generation script; the larger the number of passes, the higher the verification pass rate. By statistically analyzing the number of passes corresponding to each candidate data generation script, the relative magnitude of the verification pass rate of each candidate data generation script can be determined.
[0099] Step 15023. Compare the number of candidates that pass the script with the preset number threshold.
[0100] In some embodiments of this application, for each test scenario, a quantity threshold can be set based on the number of validators used in that test scenario. For example, if the number of validators used in a test scenario is m, then the quantity threshold corresponding to that test scenario is... , can satisfy Any value ∈ (0, m). The number of validators used varies in different test scenarios; therefore, the quantity threshold may differ for different test scenarios. The quantity threshold in step 15023 above is the quantity threshold corresponding to the test scenario indicated in the test data generation request.
[0101] Step 15024. The candidate test data generated by the candidate data generation script with a quantity greater than the quantity threshold is determined as the test data corresponding to the test data generation request.
[0102] If the number of candidate test data generated by the candidate data generation script exceeds the threshold, it indicates that the pass rate of the candidate test data meets the requirements of the test scenario. Therefore, candidate test data generated by the candidate data generation script with a number exceeding the threshold are identified as the test data corresponding to the test data generation request, thus obtaining high-quality test data that meets the data requirements of the test data generation request. The final test data corresponding to the test data generation request can be denoted as... , ,in, This represents candidate test data generated based on the data generation script.
[0103] By introducing verification operations and result rearrangement into the test data, the impact of hallucination phenomena on the final generation of test data is effectively reduced.
[0104] In some embodiments of this application, after obtaining the test data corresponding to the test data generation request, the test data, along with the initial prompt text and / or optimization prompt set, can be solidified through a test data management tool for the preparation of version test data and subsequent in-depth optimization of the existing dataset.
[0105] In some embodiments of this application, in order to improve the accuracy and effectiveness of the data constraint information stored in the knowledge base, at least one of the following maintenance methods can be used to maintain the knowledge base: The system receives editing operations on data constraints in the knowledge base through the application programming interface (API) and updates the knowledge base based on these editing operations. The system receives user-uploaded documents through the user interface and updates the knowledge base based on these documents. The documents contain descriptions of data constraints organized in the form of question-and-answer pairs or rule lists. The valid validation rules used in the validation operation are transformed into new data constraints and added to the knowledge base. Valid validation rules are those that can identify errors in the candidate test data.
[0106] By using the above method, positive feedback constraints can be established during the test data generation process to continuously optimize the test data, eliminating the need for secondary development and reducing the maintenance costs of updating and optimizing the knowledge base later.
[0107] See Figure 4 This is a flowchart illustrating a software test data generation method provided in some embodiments of this application. Wherein, "initialization Prompt" represents the initial prompt text, and "P" represents the optimization prompt set. R represents the set of constraint hints, and R represents the candidate test data generated by the data generation script.
[0108] The software test data generation method provided in this application embodiment can systematically reduce the illusion problem of large models and the uncertainty of the final generation result compared with directly using large models to generate test data.
[0109] Based on the software test data generation method provided in the above embodiments, this application also provides specific implementation methods of the software test data generation apparatus. Please refer to the following embodiments.
[0110] See Figure 5 The software test data generation apparatus 500 provided in this application embodiment includes the following modules: The request acquisition module 501 is used to acquire a test data generation request. The test data generation request includes an initial prompt text, which is used to indicate the application system to be tested and the test scenario. The prompt optimization module 502 is used to perform at least one round of optimization on the initial prompt text through the Large Language Model (LLM) to obtain an optimized prompt set, which includes the optimized prompts obtained in each round of optimization. The constraint embedding module 503 is used to retrieve data constraint information related to each optimization suggestion from a preset knowledge base, and embed the data constraint information into the relevant optimization suggestions to obtain a set of constraint suggestions; The script generation module 504 is used to input the set of constraint hints into the LLM and obtain the set of data generation scripts generated by the LLM. The data determination module 505 is used to generate test data corresponding to the test data generation request based on the data generation script set.
[0111] In this embodiment, a test data generation request is obtained. Based on LLM and a knowledge base, the initial prompt text in the test data generation request is optimized, and data constraints of the test data to be generated are embedded, resulting in a constraint-enabled prompt. Then, based on the powerful generalization capability of LLM, the data constraint information in the constraint-enabled prompt is understood, thereby generating a data generation script that can batch generate test data. The data generation script is used to batch generate the test data required by the request. According to this embodiment, test data can be automatically generated, achieving higher generation efficiency and test coverage compared to manual construction and data extraction methods. Furthermore, data constraints are defined during data generation, which improves the effectiveness of the generated test data and ensures that the test data conforms to the business logic of the application system under test and the test scenario. In addition, using LLM and a knowledge base for test data generation eliminates the need for specific development for different test scenarios, reducing development and maintenance costs.
[0112] In some embodiments of this application, the optimization module 502 is specifically used for: Input the initial prompt text into the LLM, and perform N rounds of re-description on the initial prompt text through the LLM to obtain the optimized prompt generated in each round of re-description, where N≥1; The set of N optimization suggestions generated from N rounds of re-description is taken as the optimization suggestion set.
[0113] In some embodiments of this application, the constraint embedding module 503 is specifically used for: For each optimization suggestion, test intent identification is performed on the optimization suggestion to obtain the corresponding test scenario and application system to be tested; Based on the test scenario and the application system under test corresponding to the optimization prompts, the knowledge base is searched to obtain data constraint information related to the optimization prompts; Embed the data constraint information related to the optimization hints into the optimization hints to obtain the constraint hints corresponding to the optimization hints; The set of constraint hints is defined as the constraint hint set.
[0114] In some embodiments of this application, the data determination module 505 includes: The verification submodule is used to perform verification operations on each data generation script in the data generation script set to obtain the verification result of each data generation script. The verification operations include at least one of the following: verifying the executability of the data generation script and verifying the candidate test data generated based on the data generation script. The filtering submodule is used to determine the test data corresponding to the test data generation request based on the candidate test data and verification results generated by each data generation script.
[0115] In some embodiments of this application, the verification submodule is specifically used for: Save the data generation script as an executable file; The executable file is executed using a script interpreter, and the execution result is used to indicate whether the executable file was executed successfully. If the executable file executes successfully, load at least one validator corresponding to the test scenario of the data generation script; The candidate test data generated based on the executable file is verified using at least one verifier to obtain the verification result. Based on the execution results and verification results, the verification results of the data generation script are generated.
[0116] In some embodiments of this application, the verification submodule is further configured to: In the event that the executable text fails to execute, determine the exception information of the data generation script; Using LLM, a script for generating repaired data is generated based on anomaly information, resulting in a script for generating repaired data. Perform a verification operation on the repaired data generation script to obtain the verification result of the repaired data generation script.
[0117] In some embodiments of this application, candidate test data are validated using at least one validator to obtain a validation result, including at least one of the following: The data structure validator compares the candidate test data with the table structure of the target database corresponding to the test scenario to verify whether the structure of the candidate test data is compliant. The environment adaptability verifier verifies whether the environment configuration parameters in the candidate test data conform to the preset strategy and business expectations, based on the configuration information of the test environment in the configuration management database.
[0118] In some embodiments of this application, the filtering submodule is specifically used for: Based on the verification results of each data generation script, select candidate data generation scripts that have been successfully executed from at least one data generation script; Count the number of passes for each candidate data generation script. The number of passes is the number of validators that the candidate test data generated by the candidate data generation script passes. Compare the number of candidates generated by each candidate data generation script with a preset number threshold. Candidate test data generated by a candidate data generation script with a quantity greater than a threshold are identified as the test data corresponding to the test data generation request.
[0119] In some embodiments of this application, the knowledge base is constructed in the following ways: Receive user-input test data constraint documents; After vectorizing the test data constraint documents, a knowledge base is formed.
[0120] In some embodiments of this application, the knowledge base is maintained in at least one of the following ways: The system receives editing operations on data constraints in the knowledge base through the application programming interface (API) and updates the knowledge base based on these editing operations. The system receives user-uploaded documents through the user interface and updates the knowledge base based on these documents. The documents contain descriptions of data constraints organized in the form of question-and-answer pairs or rule lists. The valid validation rules used in the validation operation are transformed into new data constraints and added to the knowledge base. Valid validation rules are those that can identify errors in the candidate test data.
[0121] The software test data generation device provided in this application embodiment can achieve... Figures 1 to 4 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.
[0122] Figure 6 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.
[0123] Electronic device 600 may include processor 601 and memory 602 storing computer program instructions.
[0124] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0125] Memory 602 may include a large-capacity memory for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to electronic device 600. In a particular embodiment, memory 602 is a non-volatile solid-state memory. Memory 602 may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, generally, memory 602 includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it performs the operations described in any of the software test data generation methods in the above embodiments.
[0126] The processor 601 reads and executes computer program instructions stored in the memory 602 to implement any of the software test data generation methods in the above embodiments.
[0127] In one example, electronic device 600 may further include communication interface 603 and bus 610. For example, Figure 6 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 610 and complete communication with each other.
[0128] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0129] Bus 610 includes hardware, software, or both, that couples components of electronic device 600 together. For example, and not limitingly, the bus may include Accelerated Graphics Port (AGP) or other graphics buses, Enhanced Industry Standard Architecture (EISA) buses, Front Side Bus (FSB), HyperTransport (HT) interconnects, Industry Standard Architecture (ISA) buses, Infinite Bandwidth Interconnects, Low Pin Count (LPC) buses, memory buses, Microchannel Architecture (MCA) buses, Peripheral Component Interconnect (PCI) buses, PCI-Express (PCI-X) buses, Serial Advanced Technology Attachment (SATA) buses, Video Electronics Standards Association Local (VLB) buses, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.
[0130] Furthermore, in conjunction with the software test data generation method in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the software test data generation methods in the above embodiments.
[0131] This application also provides a computer program product, including a computer program, which, when executed, implements any of the methods for generating software test data in the above embodiments.
[0132] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0133] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0134] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0135] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.
[0136] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A method for generating software test data, characterized in that, include: Obtain a test data generation request, the test data generation request containing an initial prompt text, the initial prompt text being used to indicate the application system to be tested and the test scenario; The initial prompt text is optimized at least once using a large language model (LLM) to obtain an optimized prompt set, which includes the optimized prompts obtained in each round of optimization. Data constraint information related to each optimization suggestion is retrieved from a preset knowledge base, and the data constraint information is embedded into the relevant optimization suggestion to obtain a set of constraint-bearing suggestions; Input the set of constraint hints into the LLM to obtain the set of data generation scripts generated by the LLM; Based on the data generation script set, test data corresponding to the test data generation request is generated.
2. The method according to claim 1, characterized in that, The initial prompt text is optimized at least once using a Large Language Model (LLM) to obtain an optimized prompt set, including: The initial prompt text is input into the LLM, and the LLM iteratively performs N rounds of re-description on the initial prompt text to obtain the optimized prompt generated in each round of re-description, where N≥1; The set of N optimization suggestions generated by the N rounds of re-description is taken as the optimization suggestion set.
3. The method according to claim 1, characterized in that, The step involves retrieving data constraint information related to each optimization suggestion from a preset knowledge base and embedding the data constraint information into the relevant optimization suggestion to obtain a set of constraint-encoded suggestions, including: For each optimization suggestion, the test intent is identified to obtain the test scenario and the application system to be tested corresponding to the optimization suggestion; Based on the test scenario and the application system to be tested corresponding to the optimization suggestion, the knowledge base is retrieved to obtain data constraint information related to the optimization suggestion; The data constraint information related to the optimization suggestion is embedded into the optimization suggestion to obtain a constraint suggestion corresponding to the optimization suggestion; The set of constraint hints is defined as the constraint hint set.
4. The method according to any one of claims 1-3, characterized in that, The step of generating test data corresponding to the test data generation request based on the data generation script set includes: A verification operation is performed on each data generation script in the data generation script set to obtain a verification result for each data generation script. The verification operation includes at least one of the following: verifying the executability of the data generation script and validating the candidate test data generated based on the data generation script. Based on the candidate test data generated by each of the data generation scripts and the verification results, the test data corresponding to the test data generation request is determined.
5. The method according to claim 4, characterized in that, The step of performing a verification operation on each of the data generation scripts to obtain a verification result for each of the data generation scripts includes: Save the data generation script as an executable file; The executable file is executed using a script interpreter to obtain an execution result, which indicates whether the executable file was executed successfully. If the executable file executes successfully, at least one verifier corresponding to the test scenario of the data generation script is loaded. The candidate test data generated based on the executable file is verified using the at least one verifier to obtain the verification result. Based on the execution result and the verification result, the verification result of the data generation script is generated.
6. The method according to claim 5, characterized in that, The step of performing a verification operation on each of the data generation scripts to obtain a verification result for each of the data generation scripts further includes: In the event that the executable text fails to execute, determine the exception information of the data generation script; Using the LLM, the data generation script is repaired based on the anomaly information to obtain the repaired data generation script; A verification operation is performed on the repaired data generation script to obtain the verification result of the repaired data generation script.
7. The method according to claim 5, characterized in that, The verification of the candidate test data through the at least one verifier to obtain the verification result includes at least one of the following: The data structure validator compares the candidate test data with the table structure of the target database corresponding to the test scenario to verify whether the structure of the candidate test data is compliant. The environment adaptability verifier verifies whether the environment configuration parameters in the candidate test data conform to the preset strategy and business expectations, based on the configuration information of the test environment in the configuration management database.
8. The method according to any one of claims 5-7, characterized in that, The process of determining the test data corresponding to the test data generation request based on the candidate test data generated by each of the data generation scripts and the verification results includes: Based on the verification results of each data generation script, candidate data generation scripts that have been successfully executed are selected from the at least one data generation script; The pass count for each candidate data generation script is counted, where the pass count is the number of validators that the candidate test data generated by the candidate data generation script passes; The number of candidates generated by each candidate data generation script is compared with a preset number threshold. Candidate test data generated by a candidate data generation script with a quantity greater than the specified threshold are determined as the test data corresponding to the test data generation request.
9. The method according to any one of claims 1-3, characterized in that, The knowledge base is constructed in the following ways: Receive user-input test data constraint documents; The knowledge base is formed by vectorizing the test data constraint documents.
10. The method according to claim 4, characterized in that, The knowledge base is maintained in at least one of the following ways: The system receives editing operations on data constraints in the knowledge base through the application programming interface (API) and updates the knowledge base based on the editing operations. The system receives user-uploaded documents through a user interface and updates the knowledge base based on the documents, wherein the documents contain data constraint descriptions organized in the form of question-and-answer pairs or rule lists. The valid verification rules used in the verification operation are transformed into new data constraints and added to the knowledge base. The valid verification rules are those that can identify errors in the candidate test data.
11. A software test data generation device, characterized in that, include: The request acquisition module is used to acquire a test data generation request. The test data generation request includes an initial prompt text, which is used to indicate the application system to be tested and the test scenario. The suggestion optimization module is used to perform at least one round of optimization on the initial suggestion text using a large language model (LLM) to obtain an optimized suggestion set, wherein the optimized suggestion set includes the optimized suggestions obtained in each round of optimization. The constraint embedding module is used to retrieve data constraint information related to each optimization suggestion from a preset knowledge base, and embed the data constraint information into the relevant optimization suggestions to obtain a set of constraint suggestions; The script generation module is used to input the set of constraint hints into the LLM and obtain the set of data generation scripts generated by the LLM; The data determination module is used to generate test data corresponding to the test data generation request based on the data generation script set.
12. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the software test data generation method as described in any one of claims 1-10.
13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the software test data generation method as described in any one of claims 1-10.
14. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device causes the electronic device to perform the software test data generation method as described in any one of claims 1-10.