Quality evaluation method of test data set of vehicle function points and server

Abstract

The invention discloses a quality evaluation method for a test data set of a vehicle function point, a server and a computer readable storage medium. The method comprises the steps of performing evaluation processing on a test data set based on a plurality of preset evaluation dimensions, and determining an evaluation result of the test data set in each preset evaluation dimension. And then, based on a preset quality evaluation rule, according to an evaluation result, evaluating the quality of the test data set. Thus, through multi-dimensional quantitative evaluation of the test data set, the quality of the test data set can be comprehensively reflected, and the quality level of the test data set can be visually known, so that the performance of the vehicle-mounted dialogue system can be verified by using the high-quality test data set, the performance level of the vehicle-mounted dialogue system is guaranteed, and the test efficiency is improved. And the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of knowledge quality assessment, and particularly to a method, a server, and a computer-readable storage medium for quality assessment of a test data set of vehicle function points. Background Art

[0002] To enhance the driving experience, an in-vehicle dialogue system is provided for users to control the vehicle through voice interaction. Before the in-vehicle dialogue system is officially released, it is often necessary to verify whether the in-vehicle dialogue system can execute vehicle function points according to voice requests through a pre-configured test data set. However, in related technologies, the quality of the test data set is not evaluated. Thus, the quality of the test data set cannot be ensured, which may affect the performance verification of the in-vehicle dialogue system, and further lead to a poor experience for users using the in-vehicle dialogue system. Summary of the Invention

[0003] This application provides a method, a server, and a computer-readable storage medium for quality assessment of a test data set of vehicle function points.

[0004] An embodiment of this application provides a method for quality assessment of a test data set of vehicle function points, the method including:

[0005] Based on multiple preset evaluation dimensions, perform an evaluation process on the test data set to determine the evaluation result of the test data set in each of the preset evaluation dimensions;

[0006] Based on a preset quality assessment rule, evaluate the quality of the test data set according to the evaluation result.

[0007] In this way, based on multiple preset evaluation dimensions, the server performs an evaluation process on the test data set to determine the evaluation result of the test data set in each preset evaluation dimension. Then, based on the preset quality assessment rule, the server evaluates the quality of the test data set according to the evaluation result. In this way, through the multi-dimensional quantitative evaluation of the test data set, the quality of the test data set can be comprehensively reflected, and the quality level of the test data set can be intuitively understood, so as to be able to use a high-quality test data set to verify the performance of the in-vehicle dialogue system, ensure the performance level of the in-vehicle dialogue system, and thus improve the user experience.

[0008] In some embodiments, the preset evaluation dimension includes a first preset evaluation dimension, and the first preset evaluation dimension is used to evaluate the sentence pattern diversity of the test data set. The performing an evaluation process on the test data set based on multiple preset evaluation dimensions to determine the evaluation result of the test data set in each of the preset evaluation dimensions includes:

[0009] Based on the standard expression of the vehicle function points, determine the first data set according to the test data set;

[0010] Determine the first evaluation result of the test data set in the first preset evaluation dimension according to the first ratio of the data quantity of the first data set and the data quantity of the test data set, wherein the larger the first ratio, the richer the sentence patterns of the test data set.

[0011] In this way, based on the standard expression of the vehicle function points, the server determines the first data set according to the test data set. Then, the server determines the first evaluation result of the test data set in the first preset evaluation dimension according to the first ratio of the data quantity of the first data set and the data quantity of the test data set, wherein the larger the first ratio, the richer the sentence patterns of the test data set. In this way, by converting the non-standard expressions in the test data set into the standard expressions of the vehicle function points, the sentence pattern diversity related to this function point in the test data set can be accurately evaluated. And the obtained first ratio can intuitively reflect the sentence pattern diversity level of the test data set, which is convenient for improvement.

[0012] In some embodiments, the determining the first data set according to the test data set based on the standard expression of the vehicle function points includes:

[0013] Based on the standard expression of the vehicle function points, perform entity normalization processing on the test data set to determine the second data set;

[0014] Perform duplicate removal processing on the second data set to determine the first data set, and the first data set includes the remaining data in the second data set after the duplicate removal processing.

[0015] In this way, based on the standard expression of the vehicle function points, the server performs entity normalization processing on the test data set to determine the second data set. Then, the server performs duplicate removal processing on the second data set to determine the first data set, and the first data set includes the remaining data in the second data set after the duplicate removal processing. In this way, through the duplicate removal processing, the duplicate sentences in the test data set can be eliminated, the level of sentence pattern diversity in the test data set can be accurately evaluated, and the influence of duplicate sentences on the evaluation result can be avoided.

[0016] In some embodiments, the preset evaluation dimension includes a second preset evaluation dimension for evaluating the sentence smoothness of the test data set. The evaluating and processing the test data set based on multiple preset evaluation dimensions to determine the evaluation result of the test data set in each preset evaluation dimension includes:

[0017] Based on a preset model, determine a third data set according to the test data set;

[0018] Determine the second evaluation result of the test data set in the second preset evaluation dimension according to the second ratio of the data quantity of the third data set to the data quantity of the test data set, where the greater the second ratio, the smoother the sentence pattern of the test data set.

[0019] In this way, based on the preset model, the server determines the third data set according to the test data set. Then, the server determines the second evaluation result of the test data set in the second preset evaluation dimension according to the second ratio of the data quantity of the third data set to the data quantity of the test data set, where the greater the second ratio, the smoother the sentence pattern of the test data set. In this way, through the preset language model, the smoothness of the sentences in the test data set can be automatically evaluated, improving the evaluation efficiency and accuracy. Moreover, through the preset language model, the subjectivity of manual evaluation can be avoided, improving the objectivity of the evaluation result.

[0020] In some embodiments, the determining the third data set according to the test data set based on the preset model includes:

[0021] Based on the preset model, calculate the perplexity score of each test data in the test data set;

[0022] Based on the preset rules, determine the deviation score of each test data according to the perplexity score and the preset perplexity score threshold;

[0023] Determine the third data set according to the test data whose deviation score is greater than the preset deviation score threshold.

[0024] In this way, based on the preset model, the server calculates the perplexity score of each test data in the test data set. Then, based on the preset rules, the server determines the deviation score of each test data according to the perplexity score and the preset perplexity score threshold. Finally, the server determines the third data set according to the test data whose deviation score is greater than the preset deviation score threshold. In this way, by calculating the perplexity score and the deviation score, the unsmooth sentences in the test data set can be identified, thereby constructing the third data set.

[0025] In some embodiments, the preset evaluation dimension includes a third preset evaluation dimension for evaluating the coverage of the test data set. The evaluating and processing the test data set based on multiple preset evaluation dimensions to determine the evaluation result of the test data set in each preset evaluation dimension includes:

[0026] Based on the standard expression of vehicle function points, perform entity normalization processing on the test data set to determine the fourth data set;

[0027] Perform entity normalization on the obtained online dataset to determine the fifth dataset;

[0028] Determine a sixth dataset based on the fourth dataset and the fifth dataset, where the sixth dataset is the intersection of the fourth dataset and the fifth dataset;

[0029] Determine the third evaluation result of the test dataset in the third preset evaluation dimension according to the third ratio of the data quantity of the sixth dataset and the data quantity of the fifth dataset, where the larger the third ratio, the wider the coverage of the test dataset.

[0030] In this way, based on the standard expression of vehicle function points, the server performs entity normalization on the test dataset to determine the fourth dataset. Then, the server performs entity normalization on the obtained online dataset to determine the fifth dataset. Next, the server determines the sixth dataset based on the fourth dataset and the fifth dataset, and the sixth dataset is the intersection of the fourth dataset and the fifth dataset. Finally, the server determines the third evaluation result of the test dataset in the third preset evaluation dimension according to the third ratio of the data quantity of the sixth dataset and the data quantity of the fifth dataset, where the larger the third ratio, the wider the coverage of the test dataset. In this way, by calculating the third ratio, the coverage of the test dataset for the online dataset can be quantitatively evaluated, thereby comprehensively verifying the performance of the in-vehicle dialogue system and ensuring its stability and reliability in various scenarios.

[0031] In some embodiments, the preset evaluation dimension includes a fourth preset evaluation dimension for evaluating the correctness of the test dataset. Based on multiple preset evaluation dimensions, evaluating the test dataset to determine the evaluation result of the test dataset in each preset evaluation dimension includes:

[0032] Based on a large language model, determine a seventh dataset according to the test dataset, where the large language model can output the seventh dataset in the test dataset that can execute the vehicle function points based on pre-configured prompt words;

[0033] Determine the fourth evaluation result of the test dataset in the fourth preset evaluation dimension according to the fourth ratio of the data quantity of the seventh dataset and the data quantity of the test dataset, where the larger the fourth ratio, the higher the correctness of the test dataset.

[0034] Thus, based on the large language model, the server determines the seventh dataset according to the test dataset, where the large language model can output the seventh dataset in the test dataset that can execute vehicle function points based on pre-configured prompt words. Then, the server determines the fourth evaluation result of the test dataset in the fourth preset evaluation dimension according to the fourth ratio of the data quantity of the seventh dataset to the data quantity of the test dataset, where the greater the fourth ratio, the higher the correctness of the test dataset. In this way, by calculating the fourth ratio, the correctness of the test data in the test dataset and its correlation with vehicle function points can be quantitatively evaluated, and the function of the in-vehicle dialogue system can be accurately evaluated to ensure that it can correctly understand and execute user instructions.

[0035] In some embodiments, the evaluating the quality of the test dataset according to the evaluation result based on a preset quality evaluation rule includes:

[0036] Based on a preset evaluation algorithm, determining a quality score of the test dataset according to the determined multiple evaluation results;

[0037] Based on a preset quality evaluation rule, evaluating the quality of the test dataset according to the quality score.

[0038] Thus, based on a preset evaluation algorithm, the server determines the quality score of the test dataset according to the determined multiple evaluation results. Then, based on a preset quality evaluation rule, the server evaluates the quality of the test dataset according to the quality score. In this way, through the preset quality evaluation rule and algorithm, multiple evaluation results can be integrated to evaluate the overall quality of the test dataset, intuitively understand the quality of the test dataset, and perform targeted optimization.

[0039] An embodiment of the present application provides a server, which includes a processor and a memory. A computer program is stored on the memory, and when the computer program is executed by the processor, the above method is implemented.

[0040] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0041] Additional aspects and advantages of the embodiments of the present application will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description of the embodiments in conjunction with the accompanying drawings, where:

[0043] Figure 1It is one of the flow diagrams of the quality assessment method for some embodiments of the present application;

[0044] Figure 2 It is the second of the flow diagrams of the quality assessment method for some embodiments of the present application;

[0045] Figure 3 It is the third of the flow diagrams of the quality assessment method for some embodiments of the present application;

[0046] Figure 4 It is the fourth of the flow diagrams of the quality assessment method for some embodiments of the present application;

[0047] Figure 5 It is the fifth of the flow diagrams of the quality assessment method for some embodiments of the present application;

[0048] Figure 6 It is the sixth of the flow diagrams of the quality assessment method for some embodiments of the present application;

[0049] Figure 7 It is the seventh of the flow diagrams of the quality assessment method for some embodiments of the present application;

[0050] Figure 8 It is the eighth of the flow diagrams of the quality assessment method for some embodiments of the present application. Detailed Embodiments

[0051] The following details the embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary only for explaining the embodiments of the present application and should not be construed as limiting the embodiments of the present application.

[0052] Currently, in-vehicle dialogue systems can provide a natural and fluent voice interaction experience. Users can control vehicle function points through voice interaction, such as adjusting the air conditioner, navigation, and playing music, etc., thus freeing their hands and being more focused on driving. To ensure the functional reliability and user experience of the in-vehicle dialogue system, before the in-vehicle dialogue system is officially released, it must be strictly tested and verified using a pre-configured test dataset.

[0053] As an important tool for verifying the performance of the in-vehicle dialogue system, the quality of the test dataset is crucial for the performance evaluation of the in-vehicle dialogue system. However, in the related art, the focus is mainly on the functional testing of the in-vehicle dialogue system, while neglecting the quality assessment of the test dataset.

[0054] In this way, the quality of the test data set cannot be ensured, which may affect the performance verification of the in-vehicle dialogue system, and further lead to a poor experience for users when using the in-vehicle dialogue system. If the test data set cannot comprehensively cover the voice requests that users may make in various scenarios, it may result in some vehicle function points not being fully tested, thus leaving potential hidden dangers. If there are errors, ambiguities, or ungrammatical sentences in the test data set, it will affect the accuracy and reliability of the test results.

[0055] Based on the above problems, please refer to Figure 1 , the embodiments of the present application provide a method for evaluating the quality of a test data set of vehicle function points, and the method includes:

[0056] 01: Based on multiple preset evaluation dimensions, perform an evaluation process on the test data set to determine the evaluation results of the test data set in each preset evaluation dimension;

[0057] 02: Based on a preset quality evaluation rule, evaluate the quality of the test data set according to the evaluation results.

[0058] The embodiments of the present application also provide a server, including a memory and a processor. The method for evaluating the quality of a test data set of vehicle function points in the embodiments of the present application can be implemented by the server in the embodiments of the present application. Specifically, a computer program is stored in the memory, and the processor is used to perform an evaluation process on the test data set based on multiple preset evaluation dimensions to determine the evaluation results of the test data set in each preset evaluation dimension. And based on a preset quality evaluation rule, evaluate the quality of the test data set according to the evaluation results.

[0059] The embodiments of the present application also provide a quality evaluation device. The method for evaluating the quality of a test data set of vehicle function points in the embodiments of the present application can be implemented by the quality evaluation device in the embodiments of the present application. Specifically, the quality evaluation device includes a determination module and a quality evaluation module. The determination module is used to perform an evaluation process on the test data set based on multiple preset evaluation dimensions to determine the evaluation results of the test data set in each preset evaluation dimension. The quality evaluation module is used to evaluate the quality of the test data set based on a preset quality evaluation rule according to the evaluation results.

[0060] Specifically, vehicle function points refer to the functions on the vehicle that can be triggered by users through voice requests, such as turning on the air conditioner, turning on the AC, setting the navigation, and playing music. The in-vehicle dialogue system uses speech recognition technology and natural language processing technology to understand the voice requests of users and convert these voice requests into operations on vehicle function points, thereby realizing the control of the vehicle.

[0061] The test data set refers to a set of data used to verify and evaluate the performance of an in-vehicle dialogue system, that is, a set of data used to test the understanding and execution capabilities of the in-vehicle dialogue system for specific functional points, including a series of simulated or real user voice requests, and the corresponding expected execution results of vehicle functional points. The main purpose of the test data set is to ensure that the in-vehicle dialogue system can accurately understand and execute various instructions proposed by users through voice interaction. In some embodiments, the test data set includes test data for testing all vehicle functional points, and all test data for a certain vehicle functional point are placed in one place, that is, equivalently, there are multiple test data subsets for different vehicle functional points in the test data set.

[0062] The multiple preset evaluation dimensions refer to multiple aspects used to measure the quality of the test data set, including sentence pattern diversity, sentence pattern smoothness, coverage, and correctness, etc.

[0063] Sentence pattern diversity refers to the richness of different sentence patterns in the test data set, reflecting whether the test cases can cover various different expression ways that users may use, so as to comprehensively test the functions and performance of the in-vehicle dialogue system.

[0064] Sentence pattern smoothness refers to the degree to which the grammar of the sentences in the test cases conforms to the rules, the semantic logic is clear, and it is easy to understand.

[0065] Coverage refers to the coverage degree of the test cases for the real user data online, reflecting whether the test cases can effectively simulate the usage scenarios of real users and discover potential problems. Test cases with high coverage can more effectively discover potential problems of the in-vehicle dialogue system and improve the performance of the in-vehicle dialogue system.

[0066] Correctness refers to whether the test data set can truly and effectively simulate the operations of users so that the vehicle controls the vehicle functional points according to the corresponding operation instructions. Test cases with high correctness of the data set can more effectively test the functions and performance of the in-vehicle dialogue system and improve the reliability and user experience of the in-vehicle dialogue system.

[0067] The preset quality evaluation rules refer to a set of predefined rules for evaluating the quality of test cases, and are the criteria for determining the overall evaluation result of the vehicle knowledge information based on the evaluation results of the test data set in each preset dimension. The preset quality evaluation rules are formulated based on test experience and domain knowledge, and combined with specific test objectives and requirements.

[0068] Select multiple preset evaluation dimensions, such as sentence pattern diversity, sentence smoothness, coverage, correctness, etc. Then, according to each selected evaluation dimension, evaluate and process the test data set to determine the evaluation results in each dimension. Finally, use the preset quality evaluation rules to fuse the evaluation results of each dimension to obtain the final quality score of the test data set.

[0069] In summary, in the quality evaluation method and server for the test data set of vehicle function points provided by the embodiments of the present application, based on multiple preset evaluation dimensions, the server evaluates and processes the test data set to determine the evaluation results of the test data set in each preset evaluation dimension. Then, based on the preset quality evaluation rules, the server evaluates the quality of the test data set according to the evaluation results. In this way, through the multi-dimensional quantitative evaluation of the test data set, the quality of the test data set can be comprehensively reflected, and the quality level of the test data set can be intuitively understood, so as to be able to use the high-quality test data set to verify the performance of the in-vehicle dialogue system, ensure the performance level of the in-vehicle dialogue system, and thus improve the user experience.

[0070] Please refer to Figure 2 , in some embodiments, the preset evaluation dimension includes a first preset evaluation dimension, and the first preset evaluation dimension is used to evaluate the sentence pattern diversity of the test data set. Step 01 (evaluating and processing the test data set based on multiple preset evaluation dimensions to determine the evaluation results of the test data set in each preset evaluation dimension) includes:

[0071] 011: Based on the standard expression of the vehicle function point, determine the first data set according to the test data set;

[0072] 012: Determine the first evaluation result of the test data set in the first preset evaluation dimension according to the first ratio of the data quantity of the first data set and the data quantity of the test data set.

[0073] In some embodiments, the determination module is further configured to determine the first data set based on the standard expression of the vehicle function point according to the test data set, and determine the first evaluation result of the test data set in the first preset evaluation dimension according to the first ratio of the data quantity of the first data set and the data quantity of the test data set.

[0074] In some embodiments, the processor is further configured to determine the first data set based on the standard expression of the vehicle function point according to the test data set, and determine the first evaluation result of the test data set in the first preset evaluation dimension according to the first ratio of the data quantity of the first data set and the data quantity of the test data set.

[0075] Specifically, the sentence pattern diversity of the test dataset refers to whether the voice requests for a certain vehicle function point in the test dataset have multiple different sentence patterns and expressions. The more sentence patterns and expressions the voice requests for a certain vehicle function point in the test dataset have, the richer the sentence pattern diversity of the test dataset. If the sentence pattern diversity of the test dataset is insufficient, then even if the in-vehicle dialogue system cannot accurately understand certain voice requests from users, it cannot be verified. For example, due to incorrect training methods, the in-vehicle dialogue system cannot recognize complex sentence patterns such as "I want to go to XX place. Can you help me with navigation?" or "I want to listen to some rock music. Can you play it?" However, during testing, only simple instructions such as "Navigate to XX place" or "Play music" are used to test the in-vehicle dialogue system. In this way, after the in-vehicle dialogue system is officially released, when users give complex sentence patterns such as "I want to go to XX place. Can you help me with navigation?" or "I want to listen to some rock music. Can you play it?", the in-vehicle dialogue system will misunderstand the user's intention or be completely unable to respond.

[0076] The standard expression of a vehicle function point refers to the pre-defined standardized expression for the in-vehicle function point. For example, the standard expression of the vehicle function point "air conditioner" is "air conditioner". In some cases, users may not use the standard expression and may call "air conditioner" "AC", etc. It is necessary to uniformly map "air conditioner" and "AC" to [air conditioner].

[0077] The first dataset refers to the dataset extracted from the test dataset after certain processing, including multiple voice requests with different sentence patterns for each vehicle function point. For example, if the test dataset includes the following test data: "Turn on the AC, turn on the air conditioner, turn on the air conditioner for a while, adjust the air conditioner temperature, turn off the air conditioner cooling mode, open the window, and turn on the music player", then the obtained first dataset is "Turn on the [air conditioner] for a while, turn on the [air conditioner], adjust the [air conditioner] temperature, set the [air conditioner] mode, open the [window], and turn on the [music player]".

[0078] The server replaces the entity words (i.e., vehicle function points) in each test data of the test dataset with the standard expressions of the vehicle function points to obtain the first dataset. Then, the server determines the first evaluation result of the test dataset in the first preset evaluation dimension according to the first ratio of the data quantity of the first dataset to the data quantity of the test dataset. Among them, the larger the first ratio, the richer the sentence patterns of the test dataset.

[0079] For example, if the test data set includes the following test data: "Turn on the AC, turn on the air conditioner, turn on the air conditioner, adjust the air conditioner temperature, turn off the air conditioner cooling mode, open the window, turn on the navigation, open the car door, close the car door, and turn on the music player", then the first data set obtained is "Turn on [air conditioner], turn on [air conditioner], adjust [air conditioner] temperature, set [air conditioner] mode, open [window], turn on [navigation], open [car door], close [car door], and turn on [music player]". In this way, according to the first ratio of 0.9 between the number of data in the first data set and the number of data in the test data set, the first evaluation result of the test data set in the first preset evaluation dimension is determined to be 0.9.

[0080] In this way, the server determines the sub-evaluation results of the vehicle knowledge information in each preset sub-dimension according to the vehicle knowledge information and the preset sub-evaluation rules. Then, based on the preset quality evaluation rules, the server performs a quality evaluation of the vehicle knowledge information according to multiple sub-evaluation results. In this way, through different sub-dimensions and sub-evaluation rules, the quality of all aspects of the vehicle knowledge information can be evaluated in detail, improving the accuracy and comprehensiveness of the evaluation. In addition, the detailed evaluation results can provide specific guidance for the optimization and update of the vehicle knowledge information, and targeted optimization helps to continuously improve the quality of the vehicle knowledge information.

[0081] Please refer to Figure 3 , in some embodiments, step 011 (determining the first data set according to the test data set based on the standard expression of the vehicle function point) includes:

[0082] 0111: Based on the standard expression of the vehicle function point, perform entity normalization processing on the test data set to determine the second data set;

[0083] 0112: Perform deduplication processing on the second data set to determine the first data set.

[0084] In some embodiments, the quality evaluation device further includes a determination module. The determination module is further configured to perform entity normalization processing on the test data set based on the standard expression of the vehicle function point to determine the second data set, and perform deduplication processing on the second data set to determine the first data set.

[0085] In some embodiments, the processor is further configured to perform entity normalization processing on the test data set based on the standard expression of the vehicle function point to determine the second data set, and perform deduplication processing on the second data set to determine the first data set.

[0086] Specifically, entity normalization refers to a method of uniformly representing entities in text data that have the same meaning but different expressions. In an in-vehicle dialogue system, entities usually refer to specific objects or vehicle function points, such as "air conditioner" and "window". Through entity normalization, the test data in the test dataset can be converted into structured data, facilitating subsequent processing and analysis. For example, the test dataset includes the following test data: "Turn on the AC, turn on the air conditioner, turn on the air conditioner once, adjust the temperature of the air conditioner, turn off the cooling mode of the air conditioner, open the window, and turn on the music player". Then, the second dataset obtained after entity normalization processing is: "Turn on the [air conditioner], turn on the [air conditioner], turn on the [air conditioner] once, adjust the temperature of the [air conditioner], set the mode of the [air conditioner], open the [window], and turn on the [music player]".

[0087] Duplicate removal processing refers to a processing method of deleting duplicate test data in the normalized test dataset and only retaining one test data. In an in-vehicle dialogue system, duplicate removal processing can be used to eliminate redundant information in test cases, improving test efficiency and accuracy. Continuing with the above example, after performing duplicate removal processing on the second dataset obtained after entity normalization processing: "Turn on the [air conditioner], turn on the [air conditioner], turn on the [air conditioner] once, adjust the temperature of the [air conditioner], set the mode of the [air conditioner], open the [window], and turn on the [music player]", the first dataset obtained is: "Turn on the [air conditioner], turn on the [air conditioner], adjust the temperature of the [air conditioner], set the mode of the [air conditioner], open the [window], turn on the [navigation], open the [door], close the [door], and turn on the [music player]", that is, only one copy of the repeated "Turn on the [air conditioner]" is retained.

[0088] According to the standard expressions of vehicle function points, entity normalization processing is performed on the test data in the test dataset, that is, converting the non-standard expressions of vehicle function points in the test data into standard expressions. For example, converting the entity word "AC" in "Turn on the AC" that can refer to the air conditioner into "air conditioner", that is, changing it to "Turn on the air conditioner".

[0089] Next, duplicate removal processing is performed on the test dataset after entity normalization processing, deleting duplicate function point expressions. For example, after normalizing both "Turn on the AC" and "Turn on the air conditioner" to "Turn on the air conditioner", only one copy is retained. The dataset after duplicate removal processing is the first dataset, including all the vehicle function point expressions after duplicate removal.

[0090] Thus, based on the standard representation of vehicle function points, the server performs entity normalization on the test data set to determine the second data set. Then, the server performs deduplication on the second data set to determine the first data set, which includes the remaining data after deduplication in the second data set. In this way, through deduplication, duplicate sentences in the test data set can be eliminated, the level of sentence pattern diversity in the test data set can be accurately evaluated, and the influence of duplicate sentences on the evaluation result can be avoided.

[0091] Please refer to Figure 4 , in some embodiments, the preset evaluation dimension includes a second preset evaluation dimension, and the second preset evaluation dimension is used to evaluate the sentence smoothness of the test data set. Step 01 (evaluating and processing the test data set based on multiple preset evaluation dimensions to determine the evaluation result of the test data set in each preset evaluation dimension) includes:

[0092] 013: Based on a preset model, determine a third data set according to the test data set;

[0093] 014: Determine a second evaluation result of the test data set in the second preset evaluation dimension according to the second ratio of the data quantity of the third data set to the data quantity of the test data set.

[0094] In some embodiments, the determination module is further configured to determine a third data set according to the test data set based on a preset model. The determination module is further configured to determine a second evaluation result of the test data set in the second preset evaluation dimension according to the second ratio of the data quantity of the third data set to the data quantity of the test data set.

[0095] In some embodiments, the processor is further configured to determine a third data set according to the test data set based on a preset model. And determine a second evaluation result of the test data set in the second preset evaluation dimension according to the second ratio of the data quantity of the third data set to the data quantity of the test data set.

[0096] Specifically, the sentence smoothness of the test dataset refers to whether the expressions of the test data in the test dataset conform to the grammar rules and expression habits of human language, whether they are fluent and natural, and easy to understand. If the sentence smoothness of some test data in the test dataset is insufficient, the in-vehicle dialogue system cannot understand the intention of these test data, and thus cannot control the vehicle. These test data are meaningless and only consume computing resources. For example, the test dataset includes two test data: "Open the window, open, open, open" and "Open the window and turn on the air conditioner". For these two test data, the in-vehicle dialogue system cannot correctly understand them. The situation shown is that for the vehicle function point of "window", the in-vehicle dialogue system cannot respond, resulting in incorrect test results and unable to truly reflect the performance of the in-vehicle dialogue system. If the problem with the test dataset is not discovered, it may be necessary to re-correct the entire in-vehicle dialogue system, greatly increasing the cost.

[0097] The preset model refers to a model such as the Kenlm language model that learns the probability distribution between words by analyzing a large amount of text data, so as to be able to predict the probability of the occurrence of a given word sequence. Taking the Kenlm language model as an example, the Kenlm language model will evaluate each sentence in the test case and calculate a sentence difficulty score, called the perplexity (ppl) score. The lower the ppl score, the smoother the sentence and the more in line with human language habits. That is, the Kenlm language model can help identify problems such as unsmooth sentences, faulty sentences, and non-human languages that may exist in the test case.

[0098] The third dataset refers to the dataset obtained by extracting from the test dataset after certain processing, including all voice requests that conform to the grammar rules and expression habits of human language. For example, if the test dataset includes the following test data: "Turn on the air conditioner, adjust the air conditioner temperature, open the window and turn on the air conditioner, open, open, open the window a little bit, and turn on the music player", then the obtained third dataset is "Turn on the air conditioner, adjust the air conditioner temperature, and turn on the music player".

[0099] Based on the preset model, the server determines the third dataset according to the test dataset. Then, the server determines the second evaluation result of the test dataset in the second preset evaluation dimension according to the second ratio of the data quantity of the third dataset and the data quantity of the test dataset. Among them, the larger the second ratio, the smoother the sentence pattern of the test dataset.

[0100] For example, the test data set is "Turn on the air conditioner, adjust the temperature of the air conditioner, open the window and then turn off the air conditioner, open the window a few times, and turn on the music player". After processing the above test data set based on the Kenlm language model, a third data set "Turn on the air conditioner, adjust the temperature of the air conditioner, and turn on the music player" is obtained. Then, the second ratio is 0.6, and the second evaluation result of the test data set in the second preset evaluation dimension is 0.6.

[0101] In this way, based on the preset model, the server determines the third data set according to the test data set. Then, the server determines the second evaluation result of the test data set in the second preset evaluation dimension according to the second ratio of the number of data in the third data set and the number of data in the test data set. Among them, the larger the second ratio, the smoother the sentence pattern of the test data set. In this way, through the preset language model, the smoothness of the sentences in the test data set can be automatically evaluated, improving the evaluation efficiency and accuracy. Moreover, through the preset language model, the subjectivity of manual evaluation can be avoided, and the objectivity of the evaluation result can be improved.

[0102] Please refer to Figure 5 , in some embodiments, step 013 (based on the preset model, determining the third data set according to the test data set) includes:

[0103] 0131: Based on the preset model, calculate the perplexity score of each test data in the test data set;

[0104] 0132: Based on the preset rules, determine the deviation score of each test data according to the perplexity score and the preset perplexity score threshold;

[0105] 0133: Determine the third data set according to the test data whose deviation score is greater than the preset deviation score threshold.

[0106] In some embodiments, the determination module is further configured to calculate the perplexity score of each test data in the test data set based on the preset model. And based on the preset rules, determine the deviation score of each test data according to the perplexity score and the preset perplexity score threshold. And determine the third data set according to the test data whose deviation score is greater than the preset deviation score threshold.

[0107] In some embodiments, the processor is further configured to calculate the perplexity score of each test data in the test data set based on the preset model. And based on the preset rules, determine the deviation score of each test data according to the perplexity score and the preset perplexity score threshold. And determine the third data set according to the test data whose deviation score is greater than the preset deviation score threshold.

[0108] Specifically, the perplexity score refers to a sentence difficulty score, called the perplexity (ppl) score, which is evaluated and calculated for each sentence in the test case based on a preset model such as the Kenlm language model.

[0109] The preset perplexity score threshold refers to the judgment criterion for distinguishing whether the test data is smooth or not based on the perplexity score. Using different preset models, the preset perplexity score threshold will also be different and needs to be set according to the actual situation. In the implementation manner of this application, with the Kenlm language model as the preset model, the preset perplexity score threshold is 1200. In some implementation manners, it can be simply considered that if the ppl score obtained by processing the test data according to the Kenlm language model is higher than the preset perplexity score threshold, the sentence is considered not smooth; if the ppl score obtained by processing the test data according to the Kenlm language model is lower than the preset perplexity score threshold, the sentence is considered smooth.

[0110] The deviation score refers to an index used to measure the smoothness of the test data, reflecting the degree of difference between the smoothness of the test data and the preset perplexity score threshold.

[0111] The preset rule refers to the formula for calculating the deviation of the sentence smoothness, which can more accurately help to judge whether the sentence is smooth or not. In the implementation manner of this application, the preset rule is that the deviation score = (ppl score - preset perplexity score threshold) / preset perplexity score threshold. In other implementation manners, other formulas can be used, which are not limited here.

[0112] The preset deviation score threshold refers to the judgment criterion for distinguishing whether the test data is smooth or not based on the deviation score. Using different preset models, the preset deviation score threshold will also be different and needs to be set according to the actual situation. In the implementation manner of this application, with the Kenlm language model as the preset model, the preset deviation score threshold is -0.15. If the deviation score of the test data is higher than the preset deviation score threshold, the sentence is considered smooth; if the deviation score of the test data is lower than the preset deviation score threshold, the sentence is considered not smooth.

[0113] Use a pre-trained language model (such as the Kenlm language model) to evaluate each test data in the test dataset and calculate the perplexity score ppl score for each sentence. The higher the perplexity score, the less smooth the sentence is.

[0114] Next, compare the perplexity score of each sentence with a preset perplexity score threshold, and calculate the deviation score according to a preset rule. The preset rule can be: deviation score = (pplscore - preset perplexity score threshold) / preset perplexity score threshold. The deviation score ranges from -1 to 1, and the higher the value, the smoother the sentence.

[0115] Finally, according to the preset deviation score threshold, filter out the test data with a deviation score greater than the threshold, and use this test data as the third data set. The third data set contains all the smooth test data in the test data set.

[0116] For example, the test data set is "[Turn on the air conditioner], [Turn on the air conditioner], [Turn on the air conditioner, set the temperature to 25 degrees, turn on the air conditioner, set the temperature to 25 degrees, and turn the fan to maximum], [Turn on the air conditioner, set the temperature to 25 degrees, turn the fan to maximum, and then turn it off], [Turn on the air conditioner, set the temperature to 25 degrees, turn the fan to maximum, then turn it off, and then turn it on again], [Turn on the air conditioner, set the temperature to 25 degrees, turn the fan to maximum, then turn it off, then turn it on again, and then turn it off again]". Based on the calculation and processing of the above preset model and preset rule, the third data set obtained is "[Turn on the air conditioner], [Turn on the air conditioner], [Turn on the air conditioner, set the temperature to 25 degrees, turn on the air conditioner, set the temperature to 25 degrees, and turn the fan to maximum]".

[0117] In this way, based on the preset model, the server calculates the perplexity score of each test data in the test data set. Next, based on the preset rule, the server determines the deviation score of each test data according to the perplexity score and the preset perplexity score threshold. Finally, the server determines the third data set based on the test data with a deviation score greater than the preset deviation score threshold. In this way, by calculating the perplexity score and the deviation score, the unsmooth sentences in the test data set can be identified, thereby constructing the third data set.

[0118] Please refer to Figure 6 , in some embodiments, the preset evaluation dimension includes a third preset evaluation dimension, and the third preset evaluation dimension is used to evaluate the coverage of the test data set. Step 01 (Based on multiple preset evaluation dimensions, evaluate and process the test data set to determine the evaluation results of the test data set in each preset evaluation dimension) includes:

[0119] 015: Based on the standard expressions of vehicle function points, perform entity normalization processing on the test data set to determine the fourth data set;

[0120] 016: Perform entity normalization processing on the obtained online data set to determine the fifth data set;

[0121] 017: Determine the sixth data set according to the fourth data set and the fifth data set;

[0122] 018: Determine a third evaluation result of the test data set in a third preset evaluation dimension according to a third ratio of the data quantity of the sixth data set to the data quantity of the fifth data set.

[0123] In some embodiments, the determination module is further configured to perform entity normalization processing on the test data set based on the standard expression of vehicle function points to determine a fourth data set, and perform entity normalization processing on the obtained online data set to determine a fifth data set. The determination module is further configured to determine a sixth data set according to the fourth data set and the fifth data set, and determine a third evaluation result of the test data set in a third preset evaluation dimension according to a third ratio of the data quantity of the sixth data set to the data quantity of the fifth data set.

[0124] In some embodiments, the processor is further configured to perform entity normalization processing on the test data set based on the standard expression of vehicle function points to determine a fourth data set, and perform entity normalization processing on the obtained online data set to determine a fifth data set. The processor is further configured to determine a sixth data set according to the fourth data set and the fifth data set, and determine a third evaluation result of the test data set in a third preset evaluation dimension according to a third ratio of the data quantity of the sixth data set to the data quantity of the fifth data set.

[0125] Specifically, the coverage of the test data set refers to the degree to which the test data set covers the vehicle function points to be tested, and is used to measure whether the test data set can comprehensively test all vehicle function points and scenarios. A high coverage means that the test data set can more comprehensively test the vehicle function points to be tested, thereby improving the reliability and effectiveness of the test. If the coverage is insufficient, it may cause the test data set to fail to cover all function points and scenarios, thus missing potential defects. These defects may appear after the product is released, affecting the user experience and product quality.

[0126] The obtained online data set refers to the real user interaction data obtained from the actual usage environment. These data can be sourced from the following aspects: First, previous version data, which is the interaction data of real users with the in-vehicle dialogue system collected from the previously released versions. These data can reflect the habits and preferences of users in actual use, such as commonly used functions, commonly used sentence expressions, etc. Second, internal test data, which is the interaction data of internal test users with the in-vehicle dialogue system collected before the official release. The internal test data can help testers evaluate the coverage of the test data set and discover possible missing scenarios in the test data set.

[0127] According to the standard expression of vehicle function points, entity normalization is performed on the test data in the test data set, that is, the non-standard expression of vehicle function points in the test data is converted into a standard expression to determine the fourth data set. For example, entity words such as "AC" in "turn on the AC" that can refer to the air conditioner are converted into "air conditioner", that is, it becomes "turn on the air conditioner".

[0128] Next, the server also performs entity normalization on the obtained online data set to determine the fifth data set. Then, the intersection of the fourth data set and the fifth data set is calculated to obtain the sixth data set.

[0129] Finally, the number of data in the sixth data set is divided by the number of data in the online data set to obtain the third ratio. According to the size of the third ratio, the evaluation result of the test data set in the coverage dimension is determined. The larger the third ratio, the more usage scenarios of online real users are covered by the test data set, and the higher the coverage.

[0130] For example, assume that the test data set contains 100 sentences and the online data set contains 200 sentences. After entity normalization and intersection calculation, the intersection of the test data set and the online data set contains 80 sentences. Calculate the third ratio: 80 / 200 = 0.4, that is, the third evaluation result of the test data set is 0.4, indicating that the test data set covers 40% of the online real user usage scenarios.

[0131] In this way, based on the standard expression of vehicle function points, the server performs entity normalization on the test data set to determine the fourth data set. Next, the server performs entity normalization on the obtained online data set to determine the fifth data set. Then, the server determines the sixth data set according to the fourth data set and the fifth data set, and the sixth data set is the intersection of the fourth data set and the fifth data set. Finally, the server determines the third evaluation result of the test data set in the third preset evaluation dimension according to the third ratio of the number of data in the sixth data set and the number of data in the fifth data set, where the larger the third ratio, the wider the coverage of the test data set. In this way, by calculating the third ratio, the coverage of the test data set for the online data set can be quantitatively evaluated, so as to comprehensively verify the performance of the in-vehicle dialogue system and ensure its stability and reliability in various scenarios.

[0132] Please refer to Figure 7 , in some embodiments, the preset evaluation dimension includes a fourth preset evaluation dimension, and the fourth preset evaluation dimension is used to evaluate the correctness of the test data set. Step 01 (evaluating and processing the test data set based on multiple preset evaluation dimensions to determine the evaluation result of the test data set in each preset evaluation dimension) includes:

[0133] 019: Based on the large language model, determine the seventh data set according to the test data set;

[0134] 030: Determine the fourth evaluation result of the test data set in the fourth preset evaluation dimension according to the fourth ratio of the data quantity of the seventh data set to the data quantity of the test data set.

[0135] In some embodiments, the determination module is further configured to determine the seventh data set based on a large language model according to the test data set, and determine the fourth evaluation result of the test data set in the fourth preset evaluation dimension according to the fourth ratio of the data quantity of the seventh data set to the data quantity of the test data set.

[0136] In some embodiments, the processor is further configured to determine the seventh data set based on a large language model according to the test data set, and determine the fourth evaluation result of the test data set in the fourth preset evaluation dimension according to the fourth ratio of the data quantity of the seventh data set to the data quantity of the test data set.

[0137] Specifically, usually there are dirty data and other use cases that do not conform to the function in a large test data set. Using these dirty data as test data will reduce the overall test metrics.

[0138] Dirty data refers to invalid data, error data, and abnormal data that do not conform to the test objectives or requirements. These data usually have a negative impact on the test results, reduce the test metrics, and affect the accuracy and reliability of the test. Invalid data refers to data that has no practical meaning or value, such as null values, duplicate values, and outliers. Error data refers to data that contains errors, such as spelling mistakes, grammar mistakes, and format mistakes. Abnormal data refers to data that deviates from the normal range, such as numerical values outside the normal range and text that does not conform to the normal logic.

[0139] The correctness of the test data set refers to whether the test cases can truly reflect the user's intentions and actual usage scenarios. Only a test data set with high correctness can ensure the accuracy of the test results, thus effectively discovering defects and errors in the in-vehicle dialogue system. Moreover, a test data set with high correctness can avoid invalid tests and improve the test efficiency. In addition, by evaluating the correctness of the test data set, potential problems can be discovered and solved, thereby improving the product quality and user experience of the in-vehicle dialogue system. In some cases, if the correctness of the test data set is insufficient, it may lead to some vehicle function points or scenarios not being tested, resulting in missed tests. If the correctness of the test data set is insufficient, it may also lead to false alarms in the test results, wasting test resources.

[0140] For example, the test dataset contains an incorrect test data "turn on and off the air conditioner", and the command that the in-vehicle dialogue system should actually execute is "turn off the air conditioner". If the tester uses this incorrect test data for testing, the test result may show that the in-vehicle dialogue system functions normally, but there are actually defects. This will lead to missed testing, ultimately affecting product quality and user experience.

[0141] User intent refers to whether the test case can accurately express the function that the user wants to achieve through the in-vehicle dialogue system.

[0142] Actual usage scenarios refer to whether the test case can cover various scenarios that users may encounter in actual use, such as different accents, different voice commands, different contexts, etc.

[0143] The seventh dataset refers to the test dataset that has been evaluated by the large language model and is determined to be able to represent the vehicle functions that users can perform through voice.

[0144] The large language model refers to a model that can understand and evaluate whether the statements in the test dataset can represent the vehicle function points that users can perform through voice. The large language model is configured with prompt words. Based on the pre-configured prompt words, this model can judge whether the test data in the test dataset can be recognized by the in-vehicle dialogue system and execute the corresponding vehicle function points. The input of the large language model is the test data in the test dataset, and the output of the large language model is the judgment result, that is, "yes" or "no". In some embodiments, the pre-configured prompt words may be "Please help me judge whether this statement can represent the vehicle functions that users can perform through voice: [function point], just return yes or no, and do not need to output any additional information. Statement: [query in the test set under this function point]".

[0145] It should be noted that the large language model may be based on pre-trained language models, such as the GPT-3 model or the BERT model, and is fine-tuned to better understand and evaluate statements related to vehicle functions.

[0146] Use the pre-trained large language model, based on the pre-configured prompt words, to judge whether the statements in the test dataset can be recognized by the in-vehicle dialogue system and execute the corresponding vehicle function points. And according to the output results of the large language model, screen out the sentences that can execute the vehicle function points to form the seventh dataset.

[0147] Next, calculate the ratio of the number of data in the seventh dataset to the number of data in the test dataset to obtain the fourth ratio. The larger the fourth ratio, the more sentences in the test dataset that can execute the vehicle function points, and the higher the correctness of the test dataset.

[0148] Thus, based on the large language model, the server determines the seventh dataset according to the test dataset, where the large language model can output the seventh dataset in the test dataset that can execute vehicle function points based on pre-configured prompt words. Then, the server determines the fourth evaluation result of the test dataset in the fourth preset evaluation dimension according to the fourth ratio of the data quantity of the seventh dataset to the data quantity of the test dataset. The larger the fourth ratio is, the higher the correctness of the test dataset is. In this way, by calculating the fourth ratio, the correctness of the test data in the test dataset and its correlation with vehicle function points can be quantitatively evaluated, and the function of the in-vehicle dialogue system can be accurately evaluated to ensure that it can correctly understand and execute user instructions.

[0149] Please refer to Figure 8 , in some embodiments, step 02 (evaluating the quality of the test dataset according to the evaluation result based on the preset quality evaluation rule) includes:

[0150] 021: Based on the preset evaluation algorithm, determine the quality score of the test dataset according to the determined multiple evaluation results;

[0151] 022: Based on the preset quality evaluation rule, evaluate the quality of the test dataset according to the quality score.

[0152] In some embodiments, the determination module is further configured to determine the quality score of the test dataset based on the preset evaluation algorithm according to the determined multiple evaluation results. The evaluation module is further configured to evaluate the quality of the test dataset based on the preset quality evaluation rule according to the quality score.

[0153] In some embodiments, the processor is further configured to determine the quality score of the test dataset based on the preset evaluation algorithm according to the determined multiple evaluation results. And evaluate the quality of the test dataset based on the preset quality evaluation rule according to the quality score.

[0154] Specifically, the preset evaluation algorithm refers to a pre-defined algorithm for quantifying the quality of the test dataset. In some embodiments, the weight of each dimension can be set, and an overall quality score threshold can be set to determine whether the test dataset meets the acceptable quality standard.

[0155] The preset quality evaluation rule refers to pre-defined evaluation criteria and guidelines for judging the quality of the test dataset. The preset quality evaluation rule is based on different evaluation dimensions, such as diversity, fluency, coverage, and correctness, etc. In some embodiments, a passing quality score can be set, and only the test dataset that reaches the passing quality score is considered to have acceptable quality.

[0156] Thus, when the quality assessment result of the vehicle knowledge information is to be optimized, based on the second preset large language model, the server optimizes the vehicle knowledge information to determine the optimized vehicle knowledge information. Then, based on the preset vehicle knowledge assessment dimensions, the server assesses the quality of the optimized vehicle knowledge information according to the optimized vehicle knowledge information and the preset assessment rules. In this way, by optimizing the vehicle knowledge information to be optimized and re-performing the quality assessment process, the number of vehicle knowledge information that meets the quality requirements can be increased.

[0157] According to the evaluation results of the test data set on multiple preset evaluation dimensions, such as diversity, coverage, fluency, and correctness, use the preset evaluation algorithm to calculate the quality score of the test data set.

[0158] Then, based on the preset quality assessment rules, for example, divide different quality levels according to the range of the quality score, and evaluate the quality of the test data set.

[0159] For example, the weights of diversity, fluency, coverage, and correctness are 0.2, 0.2, 0.3, and 0.3 respectively. The diversity score of the test data set is 0.8, the fluency score is 0.9, the coverage score is 0.7, and the correctness score is 0.85. Then the quality score of the test data set is 0.2 * 0.8 + 0.2 * 0.9 + 0.3 * 0.7 + 0.3 * 0.85 = 0.805.

[0160] The preset quality assessment rule is to divide the quality score into four levels: 0.9 - 1.0 is excellent; 0.8 - 0.89 is good; 0.7 - 0.79 is average; 0 - 0.69 is poor.

[0161] The quality score of this test data set is 0.805, belonging to the "good" level.

[0162] It should be noted that different preset evaluation algorithms and quality assessment rules can be selected according to actual needs.

[0163] Thus, based on the preset evaluation algorithm, the server determines the quality score of the test data set according to the determined multiple evaluation results. Then, based on the preset quality assessment rules, the server evaluates the quality of the test data set according to the quality score. In this way, through the preset quality assessment rules and algorithms, multiple evaluation results can be integrated to evaluate the overall quality of the test data set, intuitively understand the quality of the test data set, and perform targeted optimization.

[0164] This application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the quality assessment method for the test data set of vehicle function points as described above are implemented.

[0165] It can be understood that a computer program includes computer program code. The computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), and software distribution medium, etc.

[0166] In the description of this specification, the descriptions referring to terms such as "specifically", "furthermore", "specially", "understandably", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0167] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of executable request code including one or more steps for implementing a specific logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, where the functions can be executed in a manner that may not be in the order shown or discussed, including in a substantially simultaneous manner according to the functions involved or in the reverse order, which should be understood by those skilled in the art of the embodiments of the present application.

[0168] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for evaluating the quality of a test data set of vehicle function points, characterized in that, The method includes: Based on multiple preset evaluation dimensions, evaluate the test data set to determine the evaluation results of the test data set in each of the preset evaluation dimensions; Based on a preset quality evaluation rule, evaluate the quality of the test data set according to the evaluation results.

2. The method according to claim 1, wherein The preset evaluation dimensions include a first preset evaluation dimension for evaluating the sentence pattern diversity of the test data set. The step of based on multiple preset evaluation dimensions, evaluating the test data set to determine the evaluation results of the test data set in each of the preset evaluation dimensions includes: Based on the standard expression of the vehicle function point, determine a first data set according to the test data set; According to the first ratio of the data quantity of the first data set to the data quantity of the test data set, determine the first evaluation result of the test data set in the first preset evaluation dimension, where the larger the first ratio, the richer the sentence patterns of the test data set.

3. The method according to claim 2, wherein The step of based on the standard expression of the vehicle function point, determining a first data set according to the test data set includes: Based on the standard expression of the vehicle function point, perform entity normalization processing on the test data set to determine a second data set; Perform deduplication processing on the second data set to determine the first data set, and the first data set includes the remaining data in the second data set after the deduplication processing.

4. The method according to claim 1, wherein The preset evaluation dimensions include a second preset evaluation dimension for evaluating the sentence smoothness of the test data set. The step of based on multiple preset evaluation dimensions, evaluating the test data set to determine the evaluation results of the test data set in each of the preset evaluation dimensions includes: Based on a preset model, determine a third data set according to the test data set; According to the second ratio of the data quantity of the third data set to the data quantity of the test data set, determine the second evaluation result of the test data set in the second preset evaluation dimension, where the larger the second ratio, the smoother the sentence patterns of the test data set.

5. The method according to claim 4, characterized in that, The step of based on a preset model, determining a third data set according to the test data set includes: Based on the preset model, calculate the perplexity score of each test data in the test data set; Based on a preset rule, determine the deviation score of each test data according to the perplexity score and a preset perplexity score threshold; Determine the third data set according to the test data with the deviation score greater than the preset deviation score threshold.

6. The method according to claim 1, characterized in that, The preset evaluation dimensions include a third preset evaluation dimension for evaluating the coverage of the test data set. The step of based on multiple preset evaluation dimensions, evaluating the test data set to determine the evaluation results of the test data set in each of the preset evaluation dimensions includes: Based on the standard expression of the vehicle function point, perform entity normalization processing on the test data set to determine a fourth data set; Perform entity normalization processing on the obtained online data set to determine a fifth data set; Determine a sixth data set according to the fourth data set and the fifth data set, where the sixth data set is the intersection of the fourth data set and the fifth data set; Determine a third evaluation result of the test data set in the third preset evaluation dimension according to a third ratio of the data quantity of the sixth data set and the data quantity of the fifth data set, where the larger the third ratio, the wider the coverage of the test data set.

7. The method according to claim 1, characterized in that, The preset evaluation dimension includes a fourth preset evaluation dimension for evaluating the correctness of the test data set. Based on multiple preset evaluation dimensions, evaluating the test data set to determine an evaluation result of the test data set in each preset evaluation dimension includes: Based on a large language model, determine a seventh data set according to the test data set, where the large language model can output the seventh data set in the test data set that can execute the vehicle function points based on pre-configured prompt words; Determine a fourth evaluation result of the test data set in the fourth preset evaluation dimension according to a fourth ratio of the data quantity of the seventh data set and the data quantity of the test data set, where the larger the fourth ratio, the higher the correctness of the test data set.

8. The method according to claim 1, wherein Evaluating the quality of the test data set according to the evaluation result based on a preset quality evaluation rule includes: Determine a quality score of the test data set based on a preset evaluation algorithm according to the determined multiple evaluation results; Evaluate the quality of the test data set according to the quality score based on a preset quality evaluation rule.

9. A server, characterized in that, The server includes a processor and a memory, and a computer program is stored on the memory. When the computer program is executed by the processor, the method described in any one of claims 1-8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, the steps of the method described in any one of claims 1-8 are implemented.

Images