Test case quality evaluation method and system based on reflection distillation

Abstract

The invention relates to a test case quality evaluation method and system based on reflection distillation. The method comprises the following steps: inputting an initial test case data set into a teacher model, and judging logic to generate a thinking chain; inputting the thinking chain into the evaluation model, verifying reasoning validity, calculating a consistency score, and screening out high-quality thinking chain data; taking the high-quality thinking chain data as a fine tuning data set, and performing fine tuning training on the student model to obtain a test case quality evaluation model; receiving a to-be-evaluated test case, and performing element disassembly on the to-be-evaluated test case to obtain a precondition, an execution action and an expected result; similarity matching is carried out on the basis of the execution action and actions in an action vector library, and historical test cases corresponding to the candidate actions are obtained to serve as candidate evidences; condition evaluation and sorting screening are carried out on the candidate evidences, and support evidences are obtained; and inputting the to-be-evaluated test case and the support evidence into the student model, and outputting a test case quality evaluation conclusion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and machine learning, and particularly relates to a test case quality evaluation method and system based on reflection distillation. BACKGROUND

[0002] In the research and development and verification process of vehicle related functions, the quality of vehicle test cases directly affects the detection effect of potential defects of vehicle control systems, so the quality evaluation of vehicle test cases is one of the links to ensure the reliability of vehicle systems. The current mainstream vehicle test case quality evaluation methods mainly include manual evaluation and traditional model evaluation. Manual evaluation requires professional vehicle testers to determine the rationality and effectiveness of test cases combined with vehicle domain knowledge, which depends on the experience accumulation of evaluators, and usually includes vehicle test case review, logical consistency verification, function coverage analysis and other processes, which is applied to small-scale and high-precision vehicle test scenarios. Traditional model evaluation mainly uses a single language model to directly determine the quality of vehicle test cases, generates evaluation results by inputting vehicle test cases, and is suitable for medium and large-scale vehicle test case batch evaluation scenarios. The basic components include an input module, a model reasoning module and a result output module, and the automatic determination of test cases is completed through the vehicle test evaluation logic built in the model. In the field of knowledge distillation, the existing technology uses a teacher model to generate reasoning paths and answers, filters correct answers to build a training set to train a student model, so as to migrate the reasoning ability of a large model to a small model. In the field of automobile test case generation and evaluation, the existing technology provides domain knowledge support through a test knowledge base, combines a fine-tuned large language model to perform semantic analysis on a requirement document, and evaluates the quality of test cases through multi-dimensional indicators.

[0003] The existing vehicle test case quality evaluation technology has the following deficiencies: firstly, the manual evaluation efficiency is low. With the increase of vehicle intelligence and networked functions and the improvement of system logic complexity, the number of vehicle test cases is increasing, and the evaluation speed of professional vehicle testers is difficult to match the batch processing demand. The evaluation results are affected by subjective experience, and the judgment standards of different evaluators are different, resulting in insufficient evaluation consistency. Secondly, the traditional model evaluation scheme has a balance problem between performance and accuracy. Although a large language model can ensure a certain evaluation accuracy, the model is large in size, time-consuming in reasoning, and high in deployment cost, which is difficult to adapt to vehicle test site edge devices or lightweight test scenarios. Although a small parameter model has the advantage of lightweight, it lacks deep understanding of the logic of vehicle test cases due to limited learning ability, and there is room for improvement in evaluation accuracy and misjudgment rate. Thirdly, the existing evaluation scheme needs to be improved in the evidence support mechanism. The evaluation process is not supported by relevant vehicle test case evidence, but relies only on model reasoning or manual experience, and the reliability and persuasiveness of the evaluation results need to be improved. Fourthly, the quality control mechanism in the traditional model training process needs to be strengthened, and the effectiveness of the training data cannot be guaranteed, which affects the stability of the model evaluation performance. SUMMARY

[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a test case quality evaluation method and system based on reflection distillation, which migrates the reasoning ability of a large model to a lightweight student model through a teacher-judge-student three-stage distillation pipeline, combines evidence retrieval mechanism and quality control strategy, and realizes the balance between accuracy and efficiency of vehicle test case evaluation, reduces the deployment cost and improves the reliability and consistency of the evaluation results.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions.

[0006] In a first aspect, the present application provides a test case quality evaluation method based on reflection distillation, which adopts the following technical solutions: A test case quality evaluation method based on reflection distillation, comprising: inputting an initial test case data set to a teacher model, and generating a thought chain containing a judgment conclusion and a reasoning process based on a test case rationality judgment logic by the teacher model; inputting the thought chain generated by the teacher model to a judge model, verifying the reasoning effectiveness of each thought chain by the judge model and calculating a consistency score, and screening high-quality thought chain data based on the consistency score; using the screened high-quality thought chain data as a fine-tuning data set to fine-tune the student model, and obtaining a lightweight test case quality evaluation model; receiving a test case to be evaluated, performing element decomposition on the test case to be evaluated to obtain a precondition, an execution action, and an expected result; performing similarity matching between the execution action and actions in an action vector library to obtain a historical test case corresponding to a candidate action as a candidate evidence; performing conditional evaluation and sorting screening on the candidate evidence to obtain supporting evidence; and inputting the test case to be evaluated and the supporting evidence into the student model to output a test case quality evaluation conclusion.

[0007] Further, in the above method, the consistency score calculated by the evaluation model includes: extracting a positive label probability representing valid reasoning and a negative label probability representing invalid reasoning in the evaluation model response; when the evaluation model tends to make a positive determination, the consistency score is equal to the positive label probability; and when the evaluation model tends to make a negative determination, the consistency score is equal to 1 minus the negative label probability.

[0008] Further, in the above method, filtering low-score samples from four dimensions of logical integrity, sufficiency of basis, consistency of conclusion, and no logical contradiction based on the consistency score, and retaining high-consistency-score thought chain data.

[0009] Further, in the above method, the precondition includes a function item, a parameter value, and an environment requirement.

[0010] Further, in the above method, the function item includes at least one of a keyless entry function and a power mode.

[0011] Further, in the above method, the similarity matching between the execution action and the actions in the action vector library includes calculating the cosine similarity between the execution action and the actions in the action vector library, and matching candidate actions according to a similarity threshold.

[0012] Further, in the above method, the conditional evaluation of the candidate evidence includes evaluation based on a function item coverage rate, a value consistency, and a quantity penalty, wherein the calculation formula of the quantity penalty is , wherein is the absolute value of the quantity difference.

[0013] Further, in the above method, the sorting screening of the candidate evidence includes calculating an evaluation score of each candidate evidence by comprehensively considering the function item coverage rate, the value consistency, and the quantity penalty, arranging the test cases in descending order of the score, and taking the test cases with high rankings as the supporting evidence.

[0014] Further, in the above method, the support evidence includes top five historical test cases.

[0015] Further, in the above method, the parameter quantity of the teacher model is greater than that of the student model, and the evaluation model is a high-performance language model.

[0016] Further, in the above method, the test case is a vehicle test case, and the test case rationality determination logic includes test working condition conformity and core function coverage.

[0017] Further, in the above method, outputting the test case quality evaluation conclusion includes outputting the determination result, reasoning process of the student model and the support evidence.

[0018] In a second aspect, the present application provides a test case quality evaluation system based on reflective distillation, which adopts the following technical solution: A test case quality evaluation system based on reflective distillation, comprising: A reflective distillation training module, comprising: A teacher unit configured to receive an initial test case data set, generate a thought chain containing a determination conclusion and a reasoning process based on test case rationality determination logic; An evaluation unit connected to the teacher unit through a data interface, configured to receive the thought chain generated by the teacher unit, verify the reasoning validity of each thought chain and calculate a consistency score, and filter out high-quality thought chain data based on the consistency score; and A student unit connected to the evaluation unit through a data transmission link, configured to receive the high-quality thought chain data filtered by the evaluation unit for fine-tuning training, and obtain a lightweight test case quality evaluation model; and A quality evaluation and evidence retrieval module, comprising: An input unit communicatively connected to the student unit and the evidence retrieval unit, configured to receive a test case to be evaluated; An evidence retrieval unit integrated with an action vector library, configured to decompose the test case to be evaluated to obtain preconditions, execution actions and expected results, perform similarity matching between the execution actions and the actions in the action vector library to obtain candidate evidence, and perform condition evaluation and sorting filtering on the candidate evidence to obtain support evidence; and An evaluation output unit connected to the student unit and the evidence retrieval unit, configured to input the test case to be evaluated and the support evidence into the student unit, and output a test case quality evaluation conclusion.

[0019] In a third aspect, the present application provides a readable storage medium, which adopts the technical scheme as follows: A readable storage medium, which stores computer instructions, the computer instructions are executed by a processor to implement the method according to any one of the first aspect.

[0020] Compared with the prior art, the present application has at least one of the following beneficial technical effects: The test case quality evaluation method based on reflection distillation of the present application effectively migrates the reasoning ability of the large parameter teacher model to the lightweight student model through the teacher-judge-student three-stage distillation pipeline, significantly reduces the model deployment cost and reasoning time consumption while maintaining the evaluation accuracy; the consistency score verification and high-quality data filtering of the thinking chain are performed by the judge model, which improves the effectiveness of the training data and the stability of the model evaluation performance; the supporting evidence is obtained by element disassembly of the test case to be evaluated and similarity matching based on the action vector library, which provides historical test case evidence for the evaluation conclusion and enhances the reliability and persuasiveness of the evaluation result; the overall scheme realizes the comprehensive improvement of the vehicle test case quality evaluation in terms of accuracy, efficiency and interpretability, and is suitable for vehicle test site edge device deployment and large-scale batch evaluation scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 A flowchart of an embodiment of the test case quality evaluation method based on reflection distillation of the present application is shown.

[0023] Figure 2 A flowchart of an embodiment of the consistency score calculation method of the present application is shown.

[0024] Figure 3 A flowchart of an embodiment of the evidence retrieval method of the present application is shown.

[0025] Figure 4 A flowchart of an embodiment of the element disassembly and output step in the test case quality evaluation of the present application is shown.

[0026] Figure 5 A block diagram of an embodiment of the test case quality evaluation system based on reflection distillation of the present application is shown. DETAILED DESCRIPTION

[0027] With reference to the drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application. In addition, it should be understood that the specific implementation described herein is only used to illustrate and explain the present application, and is not used to limit the present application.

[0028] It should be noted that the sequence of the following embodiments is not limited as the preferred sequence of the embodiments of the present application. In the following embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0029] The execution sequence of the method steps described in the embodiments of the present application can be executed according to the sequence described in the specific implementation, or the execution sequence of each step can be adjusted on the premise of solving the technical problems according to actual needs, which is not listed one by one here.

[0030] The present application will be further described in detail below with reference to the accompanying drawings.

[0031] Referring to Figure 1 , Figure 1 A flowchart of a test case quality evaluation method 100 based on reflective distillation is shown. The method 100 includes a reflective distillation training phase and a quality evaluation phase, and a lightweight model is trained through a teacher-evaluator-student distillation pipeline, and efficient evaluation of test case quality is realized in combination with evidence retrieval.

[0032] In step 102, the initial test case data set is input to the teacher model, and the teacher model generates a thinking chain containing a judgment conclusion and a reasoning process based on test case rationality judgment logic. In some embodiments, the teacher model adopts Qwen3-8B large language model as a specific implementation. Before inputting the initial test case data set, the data set is standardized to ensure uniform data format and complete elements. When the test case is a vehicle test case, the test case rationality judgment logic includes test working condition conformity and core function coverage.

[0033] Continuing to refer to Figure 1In step 104, the thought chain generated by the teacher model is input to the evaluator model, the reasoning validity of each thought chain is verified by the evaluator model, and a consistency score is calculated, and high-quality thought chain data is selected based on the consistency score. The evaluator model is a high-performance language model. In some embodiments, the evaluator model uses the Qwen-Plus high-performance language model as a specific implementation. The evaluator model calculates the reasoning consistency score by verifying whether the reasoning is effective to support the judgment, and filters low-score samples from the dimensions of logical integrity, sufficiency of basis, consistency of conclusion, and absence of logical contradiction, and retains high-consistency-score high-quality thought chain data.

[0034] In step 106, the filtered high-quality thought chain data is used as a fine-tuning data set to fine-tune the student model, and a lightweight test case quality evaluation model is obtained. The parameter quantity of the teacher model is greater than that of the student model. In some embodiments, the student model uses the Qwen3-1.7B minimal language model as a specific implementation. Through knowledge distillation, the evaluation logic and reasoning ability of the teacher model are migrated to the student model, realizing lightweight deployment of the model while maintaining evaluation accuracy.

[0035] As shown in Figure 1 Further shown in step 108, the test case to be evaluated is received, and the elements are disassembled to obtain the premise condition, the execution action, and the expected result. The premise condition covers information such as function items, parameter values, and environmental requirements.

[0036] In step 110, similarity matching is performed based on the execution action and the actions in the action vector library to obtain historical test cases corresponding to the candidate action as candidate evidence. Similarity matching is achieved by calculating the cosine similarity between the execution action and the actions in the action vector library, and candidate actions are matched according to a similarity threshold, and historical test cases corresponding to the candidate actions are associated and obtained.

[0037] In step 112, the candidate evidence is conditionally evaluated and sorted and screened to obtain supporting evidence. The condition evaluation is based on function item coverage, value consistency, and quantity penalty indicators, and the evaluation score of each candidate evidence is calculated, and the test cases ranked in descending order are selected as supporting evidence.

[0038] In step 114, the test case to be evaluated and the supporting evidence are input to the student model, and the test case quality evaluation conclusion is output. The student model combines the evaluation logic learned by fine-tuning to determine the rationality of the test case, and outputs the determination result, the reasoning process, and the supporting evidence.

[0039] Referring to Figure 2 , Figure 2A flowchart showing the consistency score calculation method 200. The method 200 is used to calculate the reasoning consistency score when the evaluator model performs reflection verification on the thought chain, so as to screen high-quality thought chain data.

[0040] In step 202, the affirmative label probability indicating that the reasoning is valid and the negative label probability indicating that the reasoning is invalid in the evaluator model response are extracted. The evaluator model receives the case to be verified generated by the teacher model, the reasoning process and the judgment conclusion, and outputs a response containing the affirmative label (Yes) and the negative label (No). The affirmative label probability and the negative label probability are extracted from the response for subsequent consistency score calculation.

[0041] With reference to Figure 2 , in step 204, it is determined whether the evaluator model tends to affirmative judgment or negative judgment. The determination is based on the relative size of the affirmative label probability and the negative label probability output by the evaluator model. When the evaluator model considers that the reasoning is valid to support the judgment, the evaluator model tends to affirmative judgment; when the evaluator model considers that the reasoning is insufficient to support the judgment, the evaluator model tends to negative judgment.

[0042] When the evaluator model tends to affirmative judgment, the method 200 enters step 206, and the consistency score is equal to the affirmative label probability, that is, . This calculation method reflects the degree of affirmation of the evaluator model on the validity of the reasoning. The higher the affirmative label probability, the more sufficient the support of the reasoning process to the judgment conclusion.

[0043] As Figure 2 further shown, when the evaluator model tends to negative judgment, the method 200 enters step 208, and the consistency score is equal to 1 minus the negative label probability, that is, . This calculation method converts negative judgment into consistency score. The lower the negative label probability, the higher the consistency score, indicating that the reasoning process still has a certain validity.

[0044] In steps 210 and 212, high-quality thinking chain data is screened based on the consistency score. The screening process filters low-score samples from four dimensions of logical integrity, basis sufficiency, conclusion consistency and logical contradiction, and retains thinking chain data with high consistency scores. The logical integrity dimension evaluates whether the reasoning process contains complete logical derivation steps; the basis sufficiency dimension evaluates whether the reasoning process provides sufficient basis for judgment; the conclusion consistency dimension evaluates whether the reasoning process is consistent with the judgment conclusion; and the logical contradiction dimension evaluates whether there is logical conflict in the reasoning process. Step 210 corresponds to the screening path after step 206, and step 212 corresponds to the screening path after step 208. Both paths perform the same four-dimensional filtering logic, remove low-quality noise data, and retain high-quality thinking chain data with high consistency scores for fine-tuning training of the student model.

[0045] Referring to Figure 3 , Figure 3 A flowchart of an evidence retrieval method 300 is shown. The method 300 is used to retrieve supporting evidence related to the test case to be evaluated from historical test cases, to provide supporting information for test case quality evaluation.

[0046] In step 302, the cosine similarity of the execution action and the actions in the action vector library is calculated. The action vector library pre-imports 20,000 pieces of action data of historical vehicle test cases, and each piece of action data is stored in the action vector library after vectorization processing. When the execution action of the test case to be evaluated is received, the execution action is converted into a vector representation, and the cosine similarity with each action vector in the action vector library is calculated to quantify the semantic correlation between the execution action and the historical action.

[0047] Continuing to refer to Figure 3 In step 304, candidate actions are matched according to the similarity threshold, and the historical test cases corresponding to the candidate actions are obtained. When the cosine similarity of the execution action and a certain action in the action vector library exceeds the preset similarity threshold, the action is identified as a candidate action. The method 300 obtains the historical test cases corresponding to the candidate actions as the initial candidate evidence for subsequent conditional evaluation and sorting screening.

[0048] In step 306, the candidate evidence is conditionally evaluated based on the function item coverage, value consistency and quantity penalty. The function item coverage evaluates the matching degree of the function items in the candidate evidence and the function items in the premise conditions of the test case to be evaluated; the value consistency evaluates the consistency degree of the parameter values in the candidate evidence and the parameter values in the premise conditions of the test case to be evaluated. The quantity penalty is used to control the redundancy of the retrieval results, and the calculation formula of the quantity penalty is wherein This represents the absolute value of the quantity difference. The quantity difference reflects the quantitative deviation between candidate evidence and the test cases to be evaluated in a specific dimension. The larger the quantity difference, the lower the quantity penalty score, thereby reducing the evaluation score of redundant candidate evidence.

[0049] like Figure 3 As further shown, in step 308, the evaluation score for each candidate piece of evidence is calculated by combining functional coverage, value consistency, and quantity penalty. The evaluation score comprehensively reflects the degree of matching between the candidate evidence and the test cases to be evaluated. The higher the functional coverage, the better the value consistency, and the higher the quantity penalty score, the higher the evaluation score of the candidate evidence.

[0050] In step 310, the top five historical test cases are selected as supporting evidence and sorted in descending order of score. Method 300 sorts all candidate evidence by evaluation score from highest to lowest and selects the top five historical test cases as the final supporting evidence. The supporting evidence includes the top five historical test cases, which have a high correlation with the test cases to be evaluated, providing a reference for judging the quality of the student model.

[0051] Reference Figure 4 , Figure 4 The flowchart illustrates a method 400 for element decomposition and output in test case quality assessment. Method 400 is used to structurally decompose the vehicle test cases to be evaluated and output assessment conclusions based on supporting evidence.

[0052] In step 402, the vehicle test cases to be evaluated are decomposed into elements. The vehicle test cases cover core functional test scenarios such as tailgate control, Bluetooth key, and thermal management. Element decomposition breaks down the vehicle test cases into structured components to facilitate subsequent evidence matching and quality assessment.

[0053] Continue to refer to Figure 4 In step 404, preconditions are obtained, including functional items, parameter values, and environmental requirements. Functional items include at least one of keyless entry and power modes. In some implementations, functional items also include vehicle functional item definitions such as autonomous driving perception functions and braking control functions, which are linked through a knowledge graph. Parameter values ​​include onboard parameter specifications such as vehicle speed range and braking pressure parameters, which are also linked through a knowledge graph. Environmental requirements cover the external constraints necessary for test execution.

[0054] like Figure 4As further shown, in step 406, the execution action and expected result are obtained. The execution action describes the operational steps that need to be performed in the test case, and the expected result describes the state or output that should be achieved after the execution action. The execution action serves as the core index for subsequent evidence retrieval and is used for similarity matching with historical actions in the action vector library.

[0055] In step 408, test cases and supporting evidence are input into the lightweight student model. As mentioned earlier, supporting evidence is obtained through evidence retrieval method 300, including the top five historical test cases. The student model receives the preconditions, actions, expected results, and supporting evidence of the test cases to be evaluated, and determines the reasonableness of the test cases by combining the fine-tuned evaluation logic.

[0056] In step 410, the judgment result, reasoning process, and supporting evidence are output. The output test case quality assessment conclusion includes the output of the student model's judgment result, reasoning process, and supporting evidence. The judgment result indicates whether the test case is reasonable; the reasoning process demonstrates the logical derivation steps of the student model to arrive at the judgment conclusion; the supporting evidence provides historical test cases related to the test case to be evaluated as corroborating information, enhancing the reliability and interpretability of the assessment conclusion.

[0057] In summary, the test case quality assessment method based on reflective distillation described in this invention achieves knowledge transfer through a three-stage distillation pipeline of teacher-judge-student. The teacher model generates thought chains containing judgment conclusions and reasoning processes; the judge model filters high-quality thought chain data through consistency score verification; and the student model fine-tunes its training based on the filtered high-quality data to achieve lightweight assessment capabilities. In the quality assessment stage, supporting evidence is obtained by decomposing the test cases to be assessed into elements and performing similarity matching and conditional evaluation based on an action vector library. Finally, the student model combines the test cases and supporting evidence to output the assessment conclusion. This method significantly reduces model deployment costs and inference time while maintaining assessment accuracy. The reflective verification mechanism of the judge model improves the effectiveness of training data and the stability of model performance. The evidence retrieval mechanism provides historical test case evidence for the assessment conclusion, enhancing the reliability, persuasiveness, and interpretability of the assessment results. It is suitable for edge device deployment and large-scale batch assessment scenarios in vehicle testing environments.

[0058] This invention also discloses a test case quality assessment system based on reflective distillation.

[0059] Reference Figure 5 , Figure 5This diagram illustrates the architecture of a test case quality assessment system based on reflective distillation. The system comprises two main components: a reflective distillation training module and a quality assessment and evidence retrieval module. It employs a cloud-edge distributed deployment architecture to achieve efficient test case quality assessment.

[0060] The reflective distillation training module includes a teacher unit, a judge unit, and a student unit. The teacher unit is configured to receive an initial test case dataset and generate thought chains containing judgment conclusions and reasoning processes based on the test case rationality judgment logic. In some implementations, the teacher unit is implemented using a large language model and deployed in the cloud to utilize cloud computing resources to complete the thought chain generation task. The judge unit connects to the teacher unit via a data interface and is configured to receive the thought chains generated by the teacher unit, verify the reasoning validity of each thought chain, calculate a consistency score, and filter high-quality thought chain data based on the consistency score. In some implementations, the judge unit is implemented using a high-performance language model and completes the reflective verification and filtering of thought chains by calling a cloud API. The student unit connects to the judge unit via a data transmission link and is configured to receive the high-quality thought chain data filtered by the judge unit for fine-tuning training to obtain a lightweight test case quality assessment model. The student unit is deployed on edge devices to meet the real-time evaluation needs of vehicle testing sites.

[0061] Continue to refer to Figure 5 The quality assessment and evidence retrieval module includes an input unit, an evidence retrieval unit, and an assessment output unit. The input unit is communicatively connected to both the student unit and the evidence retrieval unit, and is configured to receive test cases to be evaluated. When the input unit receives a test case to be evaluated, it synchronously transmits the test case to both the student unit and the evidence retrieval unit for parallel processing.

[0062] like Figure 5 As further illustrated, the evidence retrieval unit integrates an action vector library, configured to decompose the test cases to be evaluated into elements to obtain preconditions, actions to be performed, and expected results. Based on the similarity matching between the actions to be performed and the actions in the action vector library, candidate evidence is obtained. This candidate evidence is then evaluated and ranked to obtain supporting evidence. The evidence retrieval unit also integrates a vehicle domain knowledge graph interface for retrieving logically corroborating information. This knowledge graph interface is linked to basic data such as vehicle function definition and onboard parameter specifications, providing domain knowledge support for evidence retrieval.

[0063] The evaluation output unit connects to both the student unit and the evidence retrieval unit. It is configured to input the test cases to be evaluated and supporting evidence into the student unit and output a test case quality evaluation conclusion. The evaluation output unit integrates the student unit's judgment results with the supporting evidence output by the evidence retrieval unit to form a complete evaluation report containing the judgment conclusion, reasoning process, and corroborating information.

[0064] In terms of system deployment, teacher units are deployed in the cloud, while student units are deployed on edge devices, forming a cloud-edge distributed deployment architecture. This distributed architecture places computationally intensive tasks such as thought chain generation and model training in the cloud, while lightweight quality assessment inference tasks are performed on edge devices, achieving a reasonable allocation of computing resources. The system is deployed on an edge server at the vehicle testing site, with a memory requirement of less than or equal to 16GB, meeting the resource constraints of the edge devices.

[0065] In terms of system integration, the system connects with the vehicle test management platform via an API interface, enabling batch import of test cases and automatic feedback of evaluation results. Through the API interface, the system retrieves test cases to be evaluated in batches from the vehicle test management platform and automatically feeds back the evaluation results, seamlessly integrating with the existing vehicle testing process.

[0066] For system debugging, test cases not used in model training were selected as debugging data to verify the system's core metrics. These core metrics included element decomposition accuracy, action matching accuracy, evaluation result accuracy, and consistency. Targeted optimizations were performed for any metrics that did not meet the standards to ensure system performance met testing requirements.

[0067] Regarding system operation and maintenance, the vehicle action vector library and vehicle domain knowledge graph data are updated regularly to ensure the timeliness and accuracy of evidence retrieval. Updates to the action vector library include supplementing with newly added test action data, while updates to the knowledge graph include updating vehicle parameter specifications and related testing standards. The student model is incrementally fine-tuned regularly using newly added high-quality test case data to continuously optimize model evaluation performance. A system operation status monitoring mechanism has been established to regularly check the connection status of each module and data transmission links, promptly handle operational anomalies, and ensure the stable operation of on-site vehicle testing and evaluation.

[0068] In summary, the test case quality assessment system based on reflective distillation described in this embodiment of the invention achieves efficient and accurate assessment of test cases through the collaborative cooperation of the reflective distillation training module and the quality assessment and evidence retrieval module. The reflective distillation training module employs a three-stage pipeline to complete knowledge distillation: teacher units generate thought chains, judge units verify and screen high-quality data, and student units fine-tune training. The quality assessment and evidence retrieval module completes quality assessment by receiving test cases to be assessed through an input unit, obtaining supporting evidence based on an action vector library and knowledge graph through an evidence retrieval unit, and integrating and outputting assessment conclusions through an assessment output unit. This system adopts a cloud-edge distributed deployment architecture, deploying computationally intensive tasks in the cloud and lightweight inference tasks on edge devices. This maintains assessment accuracy while meeting the resource constraints of edge devices. Through an API interface with the vehicle test management platform, it enables batch import of test cases and automatic feedback of assessment results, making it suitable for large-scale batch assessment scenarios in vehicle testing environments.

[0069] This invention also discloses a readable storage medium.

[0070] A computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any of the above embodiments. The computer-readable storage medium may include any entity or device capable of carrying a computer program, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc. The computer program includes computer program code. The computer program code may be in the form of source code, object code, an executable file, or some intermediate form, etc. The computer-readable storage medium may include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), and a software distribution medium, etc.

[0071] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0072] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a system including a processing module or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0073] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A test case quality assessment method based on reflective distillation, characterized in that, include: The initial test case dataset is input into the teacher model, which then generates a thought chain containing judgment conclusions and reasoning processes based on the test case rationality judgment logic. The thought chain generated by the teacher model is input into the evaluation model, which verifies the reasoning validity of each thought chain and calculates a consistency score. Based on the consistency score, high-quality thought chain data is selected. The selected high-quality thought chain data is used as a fine-tuning dataset to fine-tune the student model, resulting in a lightweight test case quality assessment model. Receive test cases to be evaluated, break down the test cases to be evaluated into elements, and obtain the preconditions, execution actions and expected results; Based on the similarity matching between the executed action and the actions in the action vector library, historical test cases corresponding to the candidate actions are obtained as candidate evidence. The candidate evidence is evaluated and ranked to obtain supporting evidence; as well as The test cases to be evaluated and the supporting evidence are input into the student model, and the test case quality evaluation conclusion is output.

2. The method according to claim 1, characterized in that, The evaluation model calculates the consistency score by including: Extract the positive label probability representing valid reasoning and the negative label probability representing invalid reasoning from the response of the evaluation model; When the evaluation model tends towards a positive decision, the consistency score equals the positive label probability; and When the evaluation model tends to make a negative judgment, the consistency score is equal to 1 minus the negative label probability.

3. The method according to claim 2, characterized in that, The selection of high-quality thought chain data based on the consistency score involves filtering low-scoring samples from four dimensions: logical completeness, sufficiency of evidence, consistency of conclusions, and absence of logical contradictions, while retaining thought chain data with high consistency scores.

4. The method according to claim 1, characterized in that, The prerequisites include functional items, parameter values, and environmental requirements.

5. The method according to claim 4, characterized in that, The features include at least one of keyless entry and power mode.

6. The method according to claim 1, characterized in that, Similarity matching based on the executed action and actions in the action vector library includes calculating the cosine similarity between the executed action and actions in the action vector library, and matching candidate actions according to a similarity threshold.

7. The method according to claim 1, characterized in that, The conditional evaluation of the candidate evidence includes evaluation based on functional item coverage, value consistency, and quantity penalty, wherein the formula for calculating the quantity penalty is as follows: ,in It is the absolute value of the quantity difference.

8. The method according to claim 7, characterized in that, The process of sorting and filtering the candidate evidence includes calculating an evaluation score for each candidate piece of evidence by comprehensively considering the coverage of the functional items, the consistency of the values, and the quantity penalty, sorting them in descending order of score, and selecting the top-ranked test cases as the supporting evidence.

9. The method according to claim 8, characterized in that, The supporting evidence includes the top five historical test cases.

10. The method according to claim 1, characterized in that, The teacher model has more parameters than the student model, and the evaluation model is a high-performance language model.

11. The method according to claim 1, characterized in that, The test cases are vehicle test cases, and the logic for determining the reasonableness of the test cases includes the degree of compliance with test conditions and the coverage of core functions.

12. The method according to claim 1, characterized in that, The output test case quality assessment conclusion includes the judgment result of the student model, the reasoning process, and the supporting evidence.

13. A test case quality assessment system based on reflective distillation, characterized in that, include: The reflective distillation training module includes: The teacher unit is configured to receive an initial test case dataset and generate a thought chain containing judgment conclusions and reasoning processes based on the test case rationality judgment logic. The evaluation unit, connected to the teacher unit via a data interface, is configured to receive thought chains generated by the teacher unit, verify the reasoning validity of each thought chain and calculate a consistency score, and filter out high-quality thought chain data based on the consistency score; and The student unit, connected to the evaluation unit via a data transmission link, is configured to receive high-quality thought process data filtered by the evaluation unit for fine-tuning and training, thereby obtaining a lightweight test case quality assessment model; and The quality assessment and evidence retrieval module includes: The input unit is communicatively connected to the student unit and the evidence retrieval unit, and is configured to receive test cases to be evaluated. The evidence retrieval unit, integrated with an action vector library, is configured to decompose the test cases to be evaluated into elements to obtain preconditions, actions to be performed, and expected results; perform similarity matching between the actions to be performed and the actions in the action vector library to obtain candidate evidence; and perform condition evaluation and ranking filtering on the candidate evidence to obtain supporting evidence; and The evaluation output unit is connected to the student unit and the evidence retrieval unit, respectively, and is configured to input the test case to be evaluated and the supporting evidence into the student unit, and output the test case quality evaluation conclusion.

14. A readable storage medium, characterized in that, The readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-12.

Images