A test case generation method, device, equipment and storage medium

Abstract

The application discloses a test case generation method and device, equipment and a storage medium, relates to the technical field of RAG system testing, and comprises the following steps: determining a target RAG system and a target knowledge base of the automobile industry, and determining a target use scenario corresponding to the target RAG system; generating a corresponding target to-be-executed task based on the target use scenario, and generating a corresponding test question according to the target to-be-executed task; retrieving a target information segment corresponding to the test question from the target knowledge base, and generating a corresponding initial test case based on the test question and the target information segment by using a target large model; performing quality auditing on the initial test case, and determining a target test case based on an auditing result, so as to test the target RAG system by using the target test case. The application can efficiently and automatically generate a test case of the RAG system in the automobile industry, which is of high quality, high coverage and high accuracy.

Description

Technical Field

[0001] This application relates to the field of RAG system testing technology, and in particular to a test case generation method, apparatus, device and storage medium. Background Technology

[0002] With the deep penetration of large language models and RAG (Retrieval-augmented Generation) technology into fields such as automotive intelligent cockpit interaction and after-sales maintenance consultation, the performance and reliability of RAG systems are directly related to user experience and even driving safety. Accurate and efficient performance evaluation of these systems has become a key requirement for industry development, and the core of evaluation quality depends on the quality of test cases.

[0003] Currently, the mainstream solutions for building test cases in the industry mainly fall into two categories: one is the manual construction method, which relies on test experts to manually search unstructured data scattered in technical documents, user manuals, fault code tables, and other carriers based on their personal experience, to conceive questions and write standard answers. While this method can ensure the quality of individual test cases to a certain extent, it is limited by human retrieval capabilities and knowledge breadth. Not only is it difficult to comprehensively collect all background data related to the function under test, resulting in incomplete test case coverage, but it also suffers from extremely long information collection and cross-validation times, leading to low generation efficiency. This method simply cannot meet the testing needs of rapidly iterating AI (Artificial Intelligence) systems and is difficult to scale up. The other category is the direct generation method using large language models, which directly submits test requirements to a large language model and instructs it to generate "question-answer" pairs in batches. While this significantly improves generation efficiency, the quality and stability are extremely poor. Because large language models need to complete multiple tasks such as scene understanding, knowledge retrieval, logical reasoning, and text generation simultaneously in a single call, they lack a closed-loop mechanism to constrain them to focus on specific facts and iterative verification. This often results in situations where the questions and answers do not match, the questions contain ambiguous pronouns, and the answers are too general or deviate from specific data. This makes them more prone to generating "illusions" and producing content that does not conform to the facts in the background knowledge base. Consequently, test cases have poor readability and insufficient effectiveness, and they lack a structured and standardized output format.

[0004] In summary, how to efficiently and automatically generate high-quality, comprehensive, and accurate test cases for automotive RAG systems is a pressing technical problem that needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a test case generation method, apparatus, device, and storage medium, capable of efficiently and automatically generating high-quality, high-coverage, and accurate test cases for automotive RAG systems. The specific solution is as follows: Firstly, this application provides a test case generation method, including: Identify the target RAG system and target knowledge base for the automotive industry, and determine the target use cases corresponding to the target RAG system; Based on the target use scenario, generate corresponding target tasks to be executed, and generate corresponding test questions based on the target tasks to be executed; Retrieve target information fragments corresponding to the test question from the target knowledge base, and generate corresponding initial test cases based on the test question and the target information fragments using the target big model; The initial test cases are subject to quality review, and target test cases are determined based on the review results, so as to test the target RAG system using the target test cases.

[0006] Optionally, determining the target use case corresponding to the target RAG system includes: Determine the functional definition of the target RAG system and the role information set in the target RAG system; The function definition and role information are analyzed using a pre-defined planning agent, and the target usage scenario corresponding to the target RAG system is determined based on the analysis results.

[0007] Optionally, generating the corresponding target task to be executed based on the target use scenario includes: Based on the target usage scenario, the product model of the target vehicle and the search targets are determined; the search targets include the device user manual and function description. Based on the target usage scenario, the product model, and the search target, generate the target task to be executed corresponding to the target vehicle; The target task to be executed is the task to be executed by the target RAG system.

[0008] Optionally, generating corresponding test questions based on the target task to be executed includes: Determine the product model of the target vehicle corresponding to the target task to be performed; Based on the product model, retrieve the target product data corresponding to the target vehicle from the preset product database; The test questions corresponding to the target RAG system are generated based on the target product data and the target usage scenario.

[0009] Optionally, retrieving the target information fragment corresponding to the test question from the target knowledge base includes: Based on the product model and the target usage scenario, the corresponding first data block is retrieved from the target knowledge base, and the first data block and the test question are analyzed using the target large model to obtain the corresponding first analysis result; Based on the first analysis result, an initial information fragment corresponding to the test question is determined from the data block, and the target information fragment is determined based on the initial information fragment.

[0010] Optionally, determining the target information fragment based on the initial information fragment includes: The initial information fragment and its corresponding number are deleted from the target knowledge base, and the current knowledge base is used as the updated knowledge base. Based on the initial information fragment and the target usage scenario, the corresponding second data block is retrieved from the updated knowledge base, and the second data block and the test question are analyzed using the target large model to obtain the corresponding second analysis result; Based on the second analysis result, a first information segment corresponding to the test question is determined from the second data block, and the first information segment is used again as the initial information segment; Jump to the step of deleting the initial information fragment and the number corresponding to the initial information fragment from the target knowledge base until the first information fragment corresponding to the test question is empty, and obtain several initial information fragments; The initial information fragments are integrated to obtain the target information fragment.

[0011] Optionally, the step of generating corresponding initial test cases based on the test question and the target information fragment using the target large model includes: Using the target large model, test answers are generated based on the target information fragments, and corresponding first test cases are generated based on the test questions and the test answers; The target fields in the first test case are supplemented to obtain the initial test case; the target fields include test case ID, priority, and expected result; Accordingly, the step of conducting a quality audit of the initial test cases and determining target test cases based on the audit results includes: The initial test cases are reviewed using a pre-defined review agent. If the audit result indicates that the test case passes, then the initial test case will be directly identified as the target test case. If the audit result indicates that the test case is not approved, the initial test case will be adjusted, and the target test case will be determined based on the adjusted test case.

[0012] Secondly, this application provides a test case generation apparatus, comprising: The target usage scenario determination module is used to determine the target RAG system and target knowledge base in the automotive industry, and to determine the target usage scenario corresponding to the target RAG system; The test question generation module is used to generate corresponding target tasks to be executed based on the target use scenario, and to generate corresponding test questions based on the target tasks to be executed. The initial test case generation module is used to retrieve target information fragments corresponding to the test question from the target knowledge base, and generate corresponding initial test cases based on the test question and the target information fragments using the target big model; The target test case determination module is used to perform quality audits on the initial test cases and determine target test cases based on the audit results, so as to use the target test cases to test the target RAG system.

[0013] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned test case generation method.

[0014] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned test case generation method.

[0015] In this application, the target RAG system and target knowledge base in the automotive industry are first determined, and the target use scenario corresponding to the target RAG system is determined. Then, based on the target use scenario, corresponding target tasks to be executed are generated, and corresponding test questions are generated according to the target tasks to be executed. Subsequently, target information fragments corresponding to the test questions are retrieved from the target knowledge base, and initial test cases are generated based on the test questions and the target information fragments using a target big model. Finally, the initial test cases are quality reviewed, and target test cases are determined based on the review results, so as to use the target test cases to test the target RAG system. As can be seen from the above, this application first identifies the target RAG system to be tested within the automotive industry, the target knowledge base, and the target usage scenarios of the target RAG system. This ensures that the generation of test cases has a clear core orientation and scenario constraints, guaranteeing that the generation direction aligns with the actual application needs of the automotive industry RAG system. Next, based on the target usage scenarios, matching target tasks are generated, and corresponding test questions are constructed for each target task. This achieves a layered decomposition from scenario to specific test questions, ensuring that test questions are targeted and scenario-related, avoiding the generation of meaningless generalized questions. Then, based on the test questions, relevant target information fragments are accurately retrieved from the target knowledge base. Using the target big model, initial test cases are generated based on the test questions and the retrieved target information fragments. This ensures that the generated test cases are supported by facts from the knowledge base, reducing the illusion of a big model from the source and guaranteeing the factual accuracy of the initial test cases. Finally, the initial test cases are verified through a quality audit, and the final target test cases are determined based on the audit results, providing qualified test cases for the subsequent target RAG system. In this way, by defining the target use case of the target RAG system, this application ensures that the generated test cases align with the actual usage requirements of the system. Furthermore, by generating test cases based on real information from the target knowledge base, the factual accuracy and content relevance of the test cases are effectively improved. Simultaneously, the target test cases obtained through quality auditing possess higher validity and standardization, enabling the efficient and automatic generation of high-quality, high-coverage, and accurate test cases for automotive industry RAG systems. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0017] Figure 1 A flowchart of a test case generation method provided in this application; Figure 2A specific flowchart for retrieving target information fragments is provided in this application; Figure 3 This application provides a specific flowchart for generating test cases; Figure 4 This application provides a schematic diagram of the structure of a test case generation device; Figure 5 This application provides a structural diagram of an electronic device. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] With the deep penetration of large language models and RAG (Retrieval-augmented Generation) technology into fields such as automotive intelligent cockpit interaction and after-sales maintenance consultation, the performance and reliability of RAG systems are directly related to user experience and even driving safety. Accurate and efficient performance evaluation has become a key requirement for industry development, and the core of evaluation quality depends on the quality of test cases. Currently, the mainstream solutions for building test cases in the industry are mainly divided into two categories: one is the manual construction method. Limited by human retrieval capabilities and knowledge breadth, this method not only struggles to comprehensively collect all background data related to the function under test, resulting in incomplete test case coverage, but also suffers from extremely long information collection and cross-validation times, leading to low generation efficiency. This method simply cannot meet the testing needs of rapidly iterating AI systems and is difficult to achieve large-scale production. Another approach is the direct generation method using large language models. Because large language models must simultaneously perform multiple tasks such as scene understanding, knowledge retrieval, logical reasoning, and text generation in a single call, they lack a closed-loop mechanism to constrain their focus on specific facts and iterative verification. This often results in issues such as mismatched questions and answers, questions containing ambiguous pronouns, and answers that are too general or deviate from specific data. This is more prone to generating "illusions," producing content that does not conform to the background knowledge base, leading to poor readability, insufficient effectiveness, and a lack of structured and standardized output formats. To address this, this application provides a test case generation scheme that can efficiently and automatically generate high-quality, high-coverage, and accurate test cases for automotive RAG systems.

[0020] See Figure 1 As shown in the figure, this application discloses a test case generation method, which may include: Step S11: Determine the target RAG system and target knowledge base for the automotive industry, and determine the target use scenario corresponding to the target RAG system.

[0021] In this embodiment, the target RAG system to be tested in the automotive industry and the target knowledge base related to the automotive industry are first determined; wherein, the target knowledge base stores automotive parts manuals, fault code tables in Excel, user guides in PDF, etc.

[0022] Next, in order to determine the target use case corresponding to the target RAG system, the specific process may include: first, determining the functional definition of the target RAG system and determining the role information set by the target RAG system; then, using a preset planning agent to analyze the functional definition and the role information, and determining the target use case corresponding to the target RAG system based on the analysis results.

[0023] Specifically, in this embodiment, it is necessary to determine the functional definition of the target RAG system and the role it plays, such as a maintenance expert assistant. Then, the planning agent analyzes the functional definition and role of the target RAG system to generate specific target usage scenarios, such as: when a user is driving a certain model of electric vehicle, the central control screen displays an error message "BMS fault," and the user inquires about the meaning and solutions from the maintenance expert assistant.

[0024] Step S12: Generate corresponding target tasks to be executed based on the target usage scenario, and generate corresponding test questions based on the target tasks to be executed.

[0025] In this embodiment, in order to generate corresponding target tasks to be executed based on the target usage scenario, the specific process may include: first, determining the product model of the target vehicle and the search target based on the target usage scenario; the search target includes the device user manual and function description; then, generating the target tasks to be executed corresponding to the target vehicle based on the target usage scenario, the product model and the search target; wherein, the target tasks to be executed are the tasks to be executed by the target RAG system.

[0026] Specifically, in this embodiment, it is necessary to determine the product model of the target vehicle and the search target based on the target usage scenario, such as the device user manual, function description, etc. Then, based on the target usage scenario of the target RAG system, the product model of the target vehicle, and the search target, the target task to be executed corresponding to the target vehicle can be generated. For example, the specific product model involved in the user's question can be determined, such as "X431 pad 9".

[0027] It should be noted that the specific process of generating corresponding test questions based on the target task to be executed may include: first, determining the product model of the target vehicle corresponding to the target task to be executed; then, retrieving the target product data corresponding to the target vehicle from a preset product database based on the product model; and finally, generating the test questions corresponding to the target RAG system based on the target product data and the target usage scenario.

[0028] Specifically, in this embodiment, an Agent invocation tool, such as function calling, can be used to query a product database based on the target vehicle's model number to retrieve target product data corresponding to the target vehicle. The product database can be in the form of an Excel spreadsheet or a database (DB). Next, core descriptive themes can be extracted from the retrieved target product data. Finally, test questions corresponding to the target RAG system can be generated by combining the descriptive themes and the target usage scenario.

[0029] Step S13: Retrieve the target information fragment corresponding to the test question from the target knowledge base, and generate corresponding initial test cases based on the test question and the target information fragment using the target big model.

[0030] In this embodiment, the corresponding target information fragment is retrieved from the target knowledge base according to the test question. The specific process may include: firstly, retrieving the corresponding first data block from the target knowledge base according to the product model and the target usage scenario, and using the target big model to analyze the first data block and the test question to obtain the corresponding first analysis result; then, based on the first analysis result, determining the initial information fragment corresponding to the test question from the data block, and determining the target information fragment based on the initial information fragment. Specifically, the above-mentioned process of determining the target information fragment based on the initial information fragment may include: deleting the initial information fragment and its corresponding number from the target knowledge base, and using the current knowledge base as the updated knowledge base; then, based on the initial information fragment and the target usage scenario, retrieving the corresponding second data block from the updated knowledge base, and using the target large model to analyze the second data block and the test question to obtain the corresponding second analysis result; then, based on the second analysis result, determining the first information fragment corresponding to the test question from the second data block, and using the first information fragment again as the initial information fragment; jumping to the step of deleting the initial information fragment and its corresponding number from the target knowledge base until the first information fragment corresponding to the test question is empty, obtaining several initial information fragments; finally, integrating the initial information fragments to obtain the target information fragment.

[0031] For more details, see Figure 2 As shown, the process for generating the target information fragment is as follows: 1. First-round retrieval: Based on the product model and target usage scenario, relevant first data block D0 is obtained from the target knowledge base using the RAG retrieval tool.

[0032] 2. Information Extraction and Recording: The LLM (Large Language Model) analyzes the first data block D0 and the test question, extracts key information K1 directly related to the test question as the initial information fragment, and records the unique source number N1 of the initial information fragment in the target knowledge base.

[0033] 3. Dynamic knowledge base reduction: Remove the data corresponding to the number set N1 from the subsequent retrieval scope to prevent information duplication and model attention distraction.

[0034] 4. Iterative Loop: Based on the extracted initial information fragment K1 and the target usage scenario, a new round of retrieval is initiated to obtain the second data block D1 after removing the data corresponding to the number N1. LLM analyzes the second data block D1 and the test question, attempts to extract new key information K2 as the first information fragment, and records the number N2 of the first information fragment K2 in the target knowledge base, and then removes the data corresponding to the number N2.

[0035] 5. Convergence Check: Repeat step d. Iterative loop until, in a certain round of data block Dm, LLM can no longer extract new key information, that is, the first information fragment corresponding to the test question in data block Dm is empty. At this point, the loop ends, and all extracted key information {K1,K2,...,Kn} is integrated into the target information fragment and used as the set of factual basis for generating the test answer.

[0036] Next, in this embodiment, the target large model can be used to generate corresponding initial test cases based on the test question and target information fragments. The specific process may include: first, using the target large model, generating test answers based on the target information fragments, and generating corresponding first test cases based on the test question and the test answers; then, supplementing the target fields in the first test cases to obtain the initial test cases; the target fields include test case ID, priority, and expected result.

[0037] Specifically, using LLM, precise and specific test answers are generated based on the integrated set of factual evidence, i.e., the target information fragments. First test cases are then generated based on the test questions and test answers. Next, the first test cases are populated with fields automatically, such as test ID, priority, and expected output results, thus obtaining the initial test cases.

[0038] Step S14: Perform a quality audit on the initial test cases, and determine the target test cases based on the audit results, so as to use the target test cases to test the target RAG system.

[0039] In this embodiment, the initial test cases can be quality reviewed, and the target test cases can be determined based on the review results. The specific process may include: first, using a preset review agent to review the initial test cases; if the review result indicates that the initial test cases have passed, then the initial test cases are directly determined as the target test cases; if the review result indicates that the initial test cases have failed, then the initial test cases are adjusted, and the target test cases are determined based on the adjusted test cases.

[0040] Specifically, in this embodiment, an auditing agent, or the aforementioned planning agent or execution agent, can be used to execute different instructions to audit the quality of the initial test cases. The audit focuses on: whether the test questions and answers strictly match; whether the test questions contain undefined pronouns; and whether the test answers are derived from extracted facts rather than general summaries. If the audit passes, the initial test cases are directly identified as target test cases; if the audit fails, the corresponding steps are returned to adjust the initial test cases to obtain the target test cases. Finally, the final qualified target test cases can be saved to the test case library in a structured format, such as JSON or CSV, so that the target RAG system can be tested using the target test cases in the library.

[0041] In one specific implementation, see Figure 3As shown, the specific process for generating test cases can be as follows: First, determine the functional definition of the RAG system under test in the automotive industry, the automotive industry background knowledge base, and the role settings of the RAG system; then, plan and schedule the Agent to first analyze the received functional definition of the RAG system under test, and then generate usage scenarios by combining the automotive industry background knowledge base and role settings, and then decompose the generated scenarios into executable tasks; then, perform task decomposition, first clarify the core query product, such as a certain car model, and at the same time determine the search dimensions, such as device usage, function description, etc.; then start the Function. The calling process first specifies the target product, then retrieves the full data corresponding to the product from product databases such as Excel or DB. Next, it extracts the core descriptive theme from the retrieved product data. Then, it generates test questions based on the usage scenario and theme. Following this, it retrieves relevant data based on the test questions, extracts key information, records the extracted information numbers, and removes the extracted information from the knowledge base. It then determines whether new information has been extracted: if new information has been extracted, it continues to extract key information; if not, it integrates all retrieved key information to generate an answer, and then generates test cases based on the test answer to the test questions. Next, it refines the standard fields in the test cases, such as expected results and priorities. Then, a quality audit agent checks the test cases, including whether the question and answer match, whether the question contains undefined pronouns, and whether the answer is a summary. Finally, it determines whether the audit is passed: if the audit fails, it returns to the test question generation stage for adjustments; if the audit passes, it enters the output layer, ultimately outputting a structured test case file containing the question, answer, test results, priority, etc. As can be seen, this embodiment eliminates the need for tedious manual information gathering and cross-validation, and automates the connection of all sub-tasks through an agent. Therefore, compared to purely manual methods, it significantly improves the speed of test case generation, enabling large-scale production. Furthermore, because the information relied upon for answer generation in this embodiment comes entirely from a traceable set of factual evidence refined through multiple rounds of "retrieval-extraction-verification" cycles, and the large model, acting as an "information integrator," fundamentally eliminates the "illusion" inherent in large models, ensuring a high degree of consistency between the answer and the knowledge base, and guaranteeing factual accuracy. Additionally, since specific entities, such as product models, have been identified through task decomposition before the final test question generation, and pronoun checks and fit judgments are explicitly set in the quality review stage, issues with unclear referencing are effectively filtered out, and questions and answers are forced to be constructed based on the same set of specific facts, ensuring the inherent consistency of the test cases. Moreover, given that the cyclical information refinement mechanism continuously mines knowledge from different angles until information is exhausted, the generated individual test cases can cover more dimensions of knowledge points in the usage scenario, resulting in a more in-depth and comprehensive test case set overall.

[0042] As can be seen from the above, this embodiment first determines the target RAG system and target knowledge base in the automotive industry, and then determines the target use scenario corresponding to the target RAG system. Next, based on the target use scenario, corresponding target tasks to be executed are generated, and corresponding test questions are generated according to the target tasks to be executed. Subsequently, target information fragments corresponding to the test questions are retrieved from the target knowledge base, and using the target big model, corresponding initial test cases are generated based on the test questions and the target information fragments. Finally, the initial test cases undergo quality review, and target test cases are determined based on the review results, so that the target RAG system can be tested using the target test cases. As can be seen from the above, this embodiment first determines the target RAG system to be tested within the automotive industry, the target knowledge base, and the target usage scenarios of the target RAG system. This ensures that the generation of test cases has a clear core orientation and scenario constraints, guaranteeing that the generation direction aligns with the actual application needs of the automotive industry RAG system. Next, based on the target usage scenarios, matching target tasks are generated, and corresponding test questions are constructed for each target task. This achieves a layered decomposition from scenario to specific test questions, ensuring that test questions are targeted and scenario-related, avoiding the generation of meaningless generalized questions. Then, based on the test questions, relevant target information fragments are accurately retrieved from the target knowledge base. Using the target big model, initial test cases are generated based on the test questions and the retrieved target information fragments. This ensures that the generated test cases are supported by facts from the knowledge base, reducing the illusion of a big model from the source and guaranteeing the factual accuracy of the initial test cases. Finally, the initial test cases are verified through a quality audit, and the final target test cases are determined based on the audit results, providing qualified test cases for the subsequent target RAG system. In this way, by defining the target use case of the target RAG system, the generated test cases are made to fit the actual usage requirements of the system. Furthermore, by generating test cases based on real information from the target knowledge base, the factual accuracy and content relevance of the test cases are effectively improved. At the same time, the target test cases obtained through quality auditing have higher validity and standardization, enabling efficient and automatic generation of high-quality, high-coverage, and accurate test cases for automotive industry RAG systems.

[0043] Accordingly, see Figure 4 As shown in the figure, this application embodiment also provides a test case generation device, which may include: The target usage scenario determination module 11 is used to determine the target RAG system and target knowledge base in the automotive industry, and to determine the target usage scenario corresponding to the target RAG system. The test question generation module 12 is used to generate corresponding target tasks to be executed based on the target use scenario, and to generate corresponding test questions based on the target tasks to be executed. The initial test case generation module 13 is used to retrieve target information fragments corresponding to the test question from the target knowledge base, and generate corresponding initial test cases based on the test question and the target information fragments using the target big model; The target test case determination module 14 is used to perform quality audits on the initial test cases and determine target test cases based on the audit results, so as to use the target test cases to test the target RAG system.

[0044] In some specific embodiments, the target usage scenario determination module 11 may include: A function definition determination unit is used to determine the function definition of the target RAG system and to determine the role information set by the target RAG system. The target usage scenario determination unit is used to analyze the function definition and role information using a preset planning agent, and determine the target usage scenario corresponding to the target RAG system based on the analysis results.

[0045] In some specific embodiments, the test question generation module 12 may include: The retrieval target determination unit is used to determine the product model of the target vehicle and the retrieval target based on the target usage scenario; the retrieval target includes the device user manual and function description. The target task determination unit is used to generate the target task to be executed corresponding to the target vehicle based on the target usage scenario, the product model and the search target; wherein the target task to be executed is the task to be executed by the target RAG system.

[0046] In some specific embodiments, the test question generation module 12 may include: The product model determination unit is used to determine the product model of the target vehicle corresponding to the target task to be performed; The target product data retrieval unit is used to retrieve target product data corresponding to the target car from a preset product database based on the product model. The test question generation unit is used to generate the test questions corresponding to the target RAG system based on the target product data and the target usage scenario.

[0047] In some specific embodiments, the initial test case generation module 13 may include: The first data block retrieval submodule is used to retrieve the corresponding first data block from the target knowledge base according to the product model and the target usage scenario, and to analyze the first data block and the test question using the target big model to obtain the corresponding first analysis result. The target information fragment generation submodule is used to determine the initial information fragment corresponding to the test question from the data block based on the first analysis result, and to determine the target information fragment based on the initial information fragment.

[0048] In some specific implementations, the target information fragment generation submodule may include: The number deletion unit is used to delete the initial information fragment and the number corresponding to the initial information fragment from the target knowledge base, and use the current knowledge base as the updated knowledge base. The second analysis result determination unit is used to retrieve the corresponding second data block from the updated knowledge base based on the initial information fragment and the target usage scenario, and to analyze the second data block and the test question using the target large model to obtain the corresponding second analysis result. The first information fragment determination unit is used to determine, based on the second analysis result, a first information fragment corresponding to the test question from the second data block, and to use the first information fragment as the initial information fragment again; An initial information fragment acquisition unit is used to jump to the step of deleting the initial information fragment and the number corresponding to the initial information fragment from the target knowledge base until the first information fragment corresponding to the test question is empty, and to acquire a number of the initial information fragments. A target information fragment determination unit is used to integrate the initial information fragments to obtain the target information fragment.

[0049] In some specific embodiments, the initial test case generation module 13 may include: The first test case generation unit is used to generate test answers based on the target information fragment using the target large model, and to generate corresponding first test cases based on the test questions and the test answers; An initial test case generation unit is used to supplement the target fields in the first test case to obtain the initial test case; the target fields include test case ID, priority, and expected result; Accordingly, the target test case determination module 14 may include: A quality audit unit is used to audit the quality of the initial test cases using a preset audit agent. The target test case determination unit is used to directly determine the initial test case as the target test case if the audit result indicates that the audit is successful. The initial test case adjustment unit is used to adjust the initial test cases if the audit result indicates that the audit is not successful, and to determine the target test cases based on the adjusted test cases.

[0050] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the test case generation method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0051] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0052] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0053] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the test case generation method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0054] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned test case generation method. The specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0055] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0056] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0057] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0058] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0059] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A test case generation method, characterized in that, include: Identify the target RAG system and target knowledge base for the automotive industry, and determine the target use cases corresponding to the target RAG system; Based on the target use scenario, generate corresponding target tasks to be executed, and generate corresponding test questions based on the target tasks to be executed; Retrieve target information fragments corresponding to the test question from the target knowledge base, and generate corresponding initial test cases based on the test question and the target information fragments using the target big model; The initial test cases are subject to quality review, and target test cases are determined based on the review results, so as to test the target RAG system using the target test cases.

2. The test case generation method according to claim 1, characterized in that, Determining the target use case corresponding to the target RAG system includes: Determine the functional definition of the target RAG system and the role information set in the target RAG system; The function definition and role information are analyzed using a pre-defined planning agent, and the target usage scenario corresponding to the target RAG system is determined based on the analysis results.

3. The test case generation method according to claim 1, characterized in that, The generation of corresponding target tasks to be executed based on the target use scenario includes: Based on the target usage scenario, the product model of the target vehicle and the search targets are determined; the search targets include the device user manual and function description. Based on the target usage scenario, the product model, and the search target, generate the target task to be executed corresponding to the target vehicle; The target task to be executed is the task to be executed by the target RAG system.

4. The test case generation method according to claim 3, characterized in that, The step of generating corresponding test questions based on the target task to be executed includes: Determine the product model of the target vehicle corresponding to the target task to be performed; Based on the product model, retrieve the target product data corresponding to the target vehicle from the preset product database; The test questions corresponding to the target RAG system are generated based on the target product data and the target usage scenario.

5. The test case generation method according to claim 4, characterized in that, The step of retrieving the target information fragment corresponding to the test question from the target knowledge base includes: Based on the product model and the target usage scenario, the corresponding first data block is retrieved from the target knowledge base, and the first data block and the test question are analyzed using the target large model to obtain the corresponding first analysis result; Based on the first analysis result, an initial information fragment corresponding to the test question is determined from the data block, and the target information fragment is determined based on the initial information fragment.

6. The test case generation method according to claim 5, characterized in that, Determining the target information fragment based on the initial information fragment includes: The initial information fragment and its corresponding number are deleted from the target knowledge base, and the current knowledge base is used as the updated knowledge base. Based on the initial information fragment and the target usage scenario, the corresponding second data block is retrieved from the updated knowledge base, and the second data block and the test question are analyzed using the target large model to obtain the corresponding second analysis result; Based on the second analysis result, a first information segment corresponding to the test question is determined from the second data block, and the first information segment is used again as the initial information segment; Jump to the step of deleting the initial information fragment and the number corresponding to the initial information fragment from the target knowledge base until the first information fragment corresponding to the test question is empty, and obtain several initial information fragments; The initial information fragments are integrated to obtain the target information fragment.

7. The test case generation method according to any one of claims 1 to 6, characterized in that, The process of generating corresponding initial test cases based on the test question and the target information fragment using the target large model includes: Using the target large model, test answers are generated based on the target information fragments, and corresponding first test cases are generated based on the test questions and the test answers; The target fields in the first test case are supplemented to obtain the initial test case; the target fields include test case ID, priority, and expected result; Accordingly, the step of conducting a quality audit of the initial test cases and determining target test cases based on the audit results includes: The initial test cases are reviewed using a pre-defined review agent. If the audit result indicates that the test case passes, then the initial test case will be directly identified as the target test case. If the audit result indicates that the test case is not approved, the initial test case will be adjusted, and the target test case will be determined based on the adjusted test case.

8. A test case generation device, characterized in that, include: The target usage scenario determination module is used to determine the target RAG system and target knowledge base in the automotive industry, and to determine the target usage scenario corresponding to the target RAG system; The test question generation module is used to generate corresponding target tasks to be executed based on the target use scenario, and to generate corresponding test questions based on the target tasks to be executed. The initial test case generation module is used to retrieve target information fragments corresponding to the test question from the target knowledge base, and generate corresponding initial test cases based on the test question and the target information fragments using the target big model; The target test case determination module is used to perform quality audits on the initial test cases and determine target test cases based on the audit results, so as to use the target test cases to test the target RAG system.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the test case generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the test case generation method as described in any one of claims 1 to 7.

Images