Vehicle-mounted voice interaction system test method and system, and storage medium

Abstract

The invention provides a vehicle-mounted voice interaction system test method and system, and a storage medium, and relates to the technical field of new energy vehicle evaluation, and the method comprises the steps: constructing a test system comprising subjective indexes, basic indexes and corrected objective indexes, and combining subjective scores, objective data (wake-up rate, recognition rate and the like) and corrected data (sight line deviation and steering wheel hand separation); and generating a standardized score through historical database order score conversion, and finally calculating a comprehensive test result by using a fuzzy matrix for weighted fusion. The test deviation is comprehensively corrected, and an accurate optimization basis is provided for automobile manufacturers.

Description

Technical Field

[0001] The present invention relates to the technical field of new energy vehicle evaluation, and particularly to a test method, system and storage medium for an in-vehicle voice interaction system. Background Art

[0002] With the rapid development of the intelligence of the vehicle cockpit, the interaction between people and vehicles has entered a new stage. In particular, the in-vehicle voice interaction system has developed rapidly and has gradually become an important carrier of the cockpit user experience and the core position of product competition. Compared with the traditional interaction methods such as buttons, levers or screen touches, voice interaction can minimize the visual attention of the driver; and it has the ability of active interaction, which is a necessary means to create a high-level human-machine interaction method. Subsequently, the test methods and tools for the in-vehicle voice interaction system have gradually emerged and play an increasingly important role in the R & D process of the voice system.

[0003] Currently, the main test methods for in-vehicle voice interaction are divided into subjective tests and objective tests. The subjective test includes the most basic internationally common MOS scoring method, which evaluates the product through the most intuitive user experience of the terminal usage effect, bypassing the problem of the conversion from consumer language to engineering language. However, the evaluation process and results are affected by the test environment and testers, with poor repeatability and consistency, and it is also difficult to reverse-dig problems from the conclusions, weakening its support for R & D improvement work; the objective test avoids the influence of environmental and human factors on the experimental results through standardized and automated tests, but it is difficult to directly convert the relevant test results into the terminal user experience effect, and blindly improving performance may miss the best balance point of the marginal effect; and the premise of the objective test is for engineering indicators whose test results can be quantified, but it is helpless for many test contents that cannot be quantified.

[0004] Currently, many research methods also attempt to combine subjective tests and objective tests to obtain a more comprehensive test system, but they ignore a problem existing in both subjective and objective tests, that is, when actually using the in-vehicle voice function, it is not a single voice channel interaction, but will inevitably integrate visual or tactile interactions. For example, the driver and passengers will inevitably, through the visual channel, quickly search for, confirm or correct the execution results of voice commands on the central control screen or instrument panel. This is not only due to user habits, but also an indication of the imperfect voice interaction system. Therefore, only by including this usage scenario in the test system can the product performance and user experience of the in-vehicle voice system be more comprehensively and realistically reflected. Summary of the Invention

[0005] To overcome the deficiencies of the prior art, the purpose of the present invention is to provide a method, system and storage medium for testing an in-vehicle voice interaction system, which can comprehensively and comprehensively correct the test results of voice interaction, and can provide a reference basis for the optimization and upgrading of vehicle models by automobile manufacturers.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] A method for testing an in-vehicle voice interaction system, comprising:

[0008] Constructing a test index system; the test index system includes subjective test indexes, basic objective test indexes and corrected objective test indexes;

[0009] Subjectively evaluating the subjective test indexes through a preset subjective scale to obtain the scores of each subjective test index, and respectively obtaining the quantitative data of the basic objective test index and the corrected objective test index through objective measurement methods;

[0010] Respectively performing weighted averaging on the quantitative data of the basic objective test index and the corrected objective test index to obtain weighted values, and respectively converting the weighted values of the basic objective test index and the corrected objective test index into standardized scores through the rank score conversion method in combination with the historical database;

[0011] Determining the weight vector V1 of the subjective test index and the basic objective test index and the weight vector V2 of the corrected objective test index based on the fuzzy matrix;

[0012] Defining the scores of the subjective test indexes and the standardized scores of the basic objective test indexes as the score vector B1, and defining the standardized scores of the corrected objective test indexes as the score vector B2;

[0013] Determining the final test score according to the weight vector V1, the score vector B1, the weight vector V2 and the score vector B2.

[0014] Preferably, the subjective test indexes include: voice naturalness, appropriateness of reply complexity and function richness; the basic objective test indexes include: wake-up success rate W n , false wake-up success rate NW n , recognition success rate I n and response time T n ; the corrected objective test indexes include: number of times of line-of-sight deviation, maximum single-line-of-sight deviation time, total line-of-sight deviation duration, number of times of steering wheel off-hand, maximum single off-hand time and total off-hand duration.

[0015] Preferably, the calculation formula for the quantitative data of the basic objective test index includes:

[0016] W n＝ m1’ / m

[0017] NW n＝ m2’ / m

[0018] I n＝ m3’ / m

[0019]

[0020] Among them, m1’ is the number of successful awakenings recorded by the system, m2’ is the number of false awakenings recorded by the system, m3’ is the number of identifications recorded by the system, m is the total number of repeated tests, T nm is the time interval of the m-th test, T nm = T nm2 - T nm1 T nm1 is the moment when the m-th interaction instruction is sent, T nm2 is the moment when the m-th interactive system completes the response, W n is the wake-up success rate, NW n is the false wake-up success rate, I n is the identification success rate, T n is the response time.

[0021] Preferably, the calculation formula for the weighted value of the basic objective test index includes:

[0022]

[0023] Among them, W, NW, I, and T are the weighted values of the wake-up success rate W n respectively, the weighted value of the false wake-up success rate NW n respectively, the weighted value of the identification success rate I n respectively, and the weighted value of the response time T n respectively, and n is the number of test cases.

[0024] Preferably, the calculation formula for the standardized score of the basic objective test index includes:

[0025]

[0026]

[0027] Among them, A W is the total number of records of the weighted value of the wake-up success rate W in the database, P n is the ranking position of the test result of the wake-up success rate in the database this time, A W is the total number of records of the weighted value of the false wake-up success rate NW in the database, A NW is the false wake-up success rate NW in the database nThe total number of records of the weighted value, P NW Is the ranking position of the false wake-up success rate test result in the database, A I Is the recognition success rate I in the database n The total number of records of the weighted value, P I Is the ranking position of the recognition success rate test result in the database, A T Is the response time T in the database n The total number of records of the weighted value, P T Is the ranking position of the response time test result in the database.

[0028] Preferably, the calculation process of the weighted value of the corrected objective test index includes:

[0029] For the corrected objective test index, the m test results of any test case are averaged and weighted to obtain the initial weight value of the corrected objective test index; the calculation formula for the initial weight value of the corrected objective test index is:

[0030] Where m is the total number of repeated tests, n AOI-n 、T AOI-max-n 、T AOI-total-n 、n HOD-n 、T HOD-max-n And T HOD-total-n Are respectively the initial weight value of the number of sight deviation times n AOI-nm Of the mth test of the nth test case, the initial weight value of the single maximum sight deviation time T AOI-max-nm Of the mth test of the nth test case, the initial weight value of the total sight deviation duration T AOI-total-nm Of the mth test of the nth test case, the initial weight value of the number of times the steering wheel is released from the hand n HOD-nm Of the mth test of the nth test case, the initial weight value of the single maximum release time T HOD-max-nm Of the mth test of the nth test case, the initial weight value of the total release duration T HOD-total-nm Of the mth test of the nth test case;

[0031] The initial weight value of the obtained corrected objective test index is averaged and weighted to obtain the weighted value of the corrected objective test index; the calculation formula for the weighted value of the corrected objective test index is:

[0032]

[0033] Where n AOI 、T AOI-max 、T AOI-total, n HOD , T HOD-max and T HOD-total are the weighted values of the number of line-of-sight deviations, the weighted value of the maximum single-line-of-sight deviation time, the weighted value of the total line-of-sight deviation duration, the weighted value of the number of times the steering wheel is released, the weighted value of the maximum single release time, and the weighted value of the total release duration, respectively.

[0034] Preferably, the calculation formula for the standardized score of the corrected objective test index includes:

[0035]

[0036] where is the total number of records of the number of line-of-sight deviation data in the database, is the ranking position of the current test result in the database, is the total number of records of the maximum single-line-of-sight deviation time value in the database, is the ranking position of the current test result in the database, is the total number of records of the total line-of-sight deviation duration value in the database, is the ranking position of the current test result in the database, is the total number of records of the number of times the steering wheel is released data in the database, is the ranking position of the current test result in the database, is the total number of records of the maximum single release time value in the database, is the ranking position of the current test result in the database, is the total number of records of the total release duration value in the database, is the ranking position of the current test result in the database, and are the standardized scores of the number of line-of-sight deviations, the maximum single-line-of-sight deviation time, the total line-of-sight deviation duration, the number of times the steering wheel is released, the maximum single release time, and the total release duration, respectively.

[0037] Preferably, the calculation formula for the final test score G is:

[0038] G = V1 × B1 · V2 × B2.

[0039] A vehicle-mounted voice interaction system test system includes:

[0040] A system construction module for constructing a test index system; the test index system includes subjective test indexes, basic objective test indexes, and corrected objective test indexes;

[0041] The subjective and objective evaluation module subjectively evaluates the subjective test indicators through a preset subjective scale to obtain the scores of each subjective test indicator, and respectively obtains the quantification data of the basic objective test indicator and the corrected objective test indicator through an objective measurement method;

[0042] The score determination module is used to respectively perform weighted averaging on the quantification data of the basic objective test indicator and the corrected objective test indicator to obtain weighted values, and respectively convert the weighted values of the basic objective test indicator and the corrected objective test indicator into standardized scores through the rank score conversion method in combination with the historical database;

[0043] The weight vector acquisition module is used to determine the weight vector V1 of the subjective test indicator and the basic objective test indicator and the weight vector V2 of the corrected objective test indicator based on the fuzzy matrix;

[0044] The score vector acquisition module is used to define the scores of the subjective test indicators and the standardized scores of the basic objective test indicators as the score vector B1, and define the standardized scores of the corrected objective test indicators as the score vector B2;

[0045] The final score determination module is used to determine the final test score according to the weight vector V1, the score vector B1, the weight vector V2 and the score vector B2.

[0046] A storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned vehicle-mounted voice interaction system test method is implemented.

[0047] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0048] The present invention provides a method, a system and a storage medium for testing an in-vehicle voice interaction system. The method includes: constructing a test index system; the test index system includes subjective test indexes, basic objective test indexes and corrected objective test indexes; subjectively evaluating the subjective test indexes through a preset subjective scale to obtain the scores of each subjective test index, and respectively obtaining the quantitative data of the basic objective test indexes and the corrected objective test indexes through objective measurement methods; respectively performing weighted averaging on the quantitative data of the basic objective test indexes and the corrected objective test indexes to obtain weighted values, and respectively converting the weighted values of the basic objective test indexes and the corrected objective test indexes into standardized scores through a rank score conversion method in combination with a historical database; determining a weight vector V1 of the subjective test indexes and the basic objective test indexes and a weight vector V2 of the corrected objective test indexes based on a fuzzy matrix; defining the scores of the subjective test indexes and the standardized scores of the basic objective test indexes as a score vector B1, and defining the standardized scores of the corrected objective test indexes as a score vector B2; determining a final test score according to the weight vector V1, the score vector B1, the weight vector V2 and the score vector B2. The present invention can comprehensively and comprehensively correct the voice interaction test results, and can provide a reference basis for the vehicle model optimization and upgrade of automobile manufacturers. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0050] Figure 1 It is a flowchart of the method provided by an embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of the technical route provided by an embodiment of the present invention;

[0052] Figure 3 It is a schematic diagram of the system structure provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0054] The object of the present invention is to provide a method, a system and a storage medium for testing an in-vehicle voice interaction system, which can comprehensively and comprehensively correct the voice interaction test results and provide a reference basis for the model optimization and upgrade of automobile manufacturers.

[0055] In order to make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] Figure 1 As shown in the flowchart of the method provided by the embodiment of the present invention, Figure 1 the present invention provides a method for testing an in-vehicle voice interaction system, including:

[0057] Step 100: Construct a test index system; the test index system includes subjective test indexes, basic objective test indexes and corrected objective test indexes;

[0058] Step 200: Subjectively evaluate the subjective test indexes through a preset subjective scale to obtain the scores of each subjective test index, and respectively obtain the quantitative data of the basic objective test indexes and the corrected objective test indexes through objective measurement methods;

[0059] Step 300: Respectively perform weighted averaging on the quantitative data of the basic objective test indexes and the corrected objective test indexes to obtain weighted values, and respectively convert the weighted values of the basic objective test indexes and the corrected objective test indexes into standardized scores through the rank score conversion method in combination with the historical database;

[0060] Step 400: Determine the weight vector V1 of the subjective test indexes and the basic objective test indexes and the weight vector V2 of the corrected objective test indexes based on the fuzzy matrix;

[0061] Step 500: Define the scores of the subjective test indexes and the standardized scores of the basic objective test indexes as the score vector B1, and define the standardized scores of the corrected objective test indexes as the score vector B2;

[0062] Step 600: Determine the final test score according to the weight vector V1, the score vector B1, the weight vector V2 and the score vector B2.

[0063] Optionally, as Figure 2 shown, the technical route of this embodiment is as follows:

[0064] Step 1: Use the subjective scale to evaluate the three subjective test indexes of voice naturalness S1, appropriateness of reply complexity S2, and function richness S3. After in-depth experience, the test subjects give specific evaluation scores according to the descriptions of each scoring criterion given in the subjective scale, using a 10-point system.

[0065] Step 2: Use objective measurement methods to test four objective indicators: wake-up success rate, false wake-up success rate, recognition success rate, and response time, and six objective indicators: number of times of line-of-sight deviation, maximum single line-of-sight deviation time, total line-of-sight deviation duration, number of times of steering wheel leaving hand, maximum single leaving hand time, and total leaving hand duration.

[0066] Step 3: For specific objective test indicators, specific test cases C = {C1, C2, C3... C n} can be formulated in combination with the test purpose, application scenario, etc., where C n = (noise type, noise decibel, maintaining vehicle speed, command statement...). The test cases corresponding to different indicators can be the same or different.

[0067] Step 4: During the test process of a specific test case C n , the artificial mouth automatically plays the statements corresponding to the test case. Starting from the moment the command is sent, the high-definition industrial camera records the changes in the central control and instrument screens, collects the changes in the interface, completes the recognition of key icons or texts in the picture frames based on the image recognition algorithm, and compares them with the true icons or texts, accurately judges and records the response results and process time of the voice system. All the above work is completed by the automated test tool. The same command is repeated for m tests to obtain the following basic test indicators.

[0068] Wake-up success rate W n It refers to the probability that the voice system can be successfully awakened when playing a command containing a specified wake-up word. Through m repeated tests of a certain wake-up statement command, the system automatically records the number of times m' that the voice interaction system is successfully awakened. By comparison, the wake-up success rate W for this command is obtained n＝ m' / m, where m' is the number of times the system is recorded as being successfully awakened, and m is the total number of repeated tests.

[0069] False wake-up rate NW n It refers to the probability that the voice system is falsely awakened when playing a command that does not contain a specified wake-up word. Through m repeated tests of a certain wake-up statement command, the system automatically records the number of times m' that the voice interaction system is falsely awakened. By comparison, the false wake-up rate NW for this command is obtained n＝ m' / m, where m' is the number of times the system is recorded as being falsely awakened, and m is the total number of repeated tests.

[0070] Recognition rate I n It refers to the probability that the voice system correctly understands and executes the feedback when playing a command containing specified interaction content. Through m repeated tests of a certain statement command, the system automatically records the number of times m' that the voice interaction system successfully recognizes and gives an accurate feedback. By comparison, the recognition rate I for this command is obtainedn＝ m’ / m, where m’ is the number of recognized times recorded by the system, and m is the total number of repeated tests.

[0071] Response time T n refers to the time interval from the sending of the interaction instruction to the completion of recognition by the voice system and the completion of the response action. T nm = T nm2 - T nm1 , where T nm is the time interval of the m-th test, and T nm1 is the moment when the m-th interaction instruction is sent, and T nm2 is the moment when the m-th interaction system completes the response. When performing m repeated tests, the response time is weighted and averaged.

[0072] Step 5: Average and weight the quantization results of the n test cases of the obtained objective test indicators to obtain the quantization test results of each test indicator.

[0073] Wake-up rate

[0074] False wake-up rate

[0075] Recognition rate

[0076] Response time

[0077] Step 6: For the final test results of the 4 objective test indicators, convert the objective values into scores on a 10-point scale through the rank score conversion method and the historical database. Specifically, where S is the final score, A is the number of test result records of this indicator in the database, and P is the ranking position of the test result of this indicator in the total records. Thus, the specific scores of the 4 objective indicators can be obtained.

[0078] Wake-up rate score In the formula, A W is the total number of records of the wake-up rate measurement values in the database, and P W is the ranking position of the current test result in the database.

[0079] False wake-up rate In the formula, A NW is the total number of records of the false wake-up rate measurement values in the database, and P NW is the ranking position of the current test result in the database.

[0080] Recognition rate In the formula, A I is the total number of records of the recognition rate measurement values in the database, and P IIs the ranking position of the test results in the database.

[0081] Response time Where A T Is the total number of records of the response time measurement values in the database, and P T Is the ranking position of the test results in the database.

[0082] Step 7: Determine the index weights based on the fuzzy matrix, and perform weighted fusion for 3 subjective test indexes and 4 objective test indexes. Specifically, invite multiple experts to give judgments on the relative importance between every two of the 7 indexes, and transform it into a fuzzy complementary matrix R1 = (r ij ) 7×7 , and further obtain a fuzzy consistency matrix Y1 = (y ij ) 7×7 .

[0083]

[0084] Among them, R1 is the fuzzy complementary matrix for the above indexes, and r ij Is the element in the i-th row and j-th column of the matrix; Y1 is the fuzzy consistency matrix, and y ij Is the element in the i-th row and j-th column of the matrix; the specific transformation process is that y i Is equal to the sum of all elements in the i-th row of the R1 matrix. Further, the specific element value y ij In Y1 is obtained through the final formula calculation.

[0085] Optionally, the process of transforming the complementary 1-5 scale method into R1: form a 7×7 matrix with rows and columns of the 7 indexes, where r ij Represents the importance degree of the i-th index compared to the j-th index (that is, compare the 7 indexes with all 7 indexes including itself in turn), and assign values to the matrix elements according to the relative importance degree under subjective judgment. The specific rules are as follows: i and j are equally important, assign a value of 0.5; i is slightly more important than j, then r ij = 0.6, r ji = 0.4; i is significantly more important than j, then r ij = 0.7, r ji = 0.3; i is strongly more important than j, then r ij = 0.8, r ji = 0.2; i is extremely more important than j, then r ij = 0.9, r ji = 0.1.

[0086] Finally, the weight vector V1 is obtained by row normalization, and the test scores of the seven indicators are defined as the score vector B1. The preliminary test results of the in-vehicle voice interaction system are further obtained through G1 = V1 × B1.

[0087] Step 8: On the basis of testing the voice interaction process and results based on machine vision, supplementary evaluation of the interaction effect is carried out through an eye tracker and steering wheel off-hand monitoring. The specific steps are to establish test cases C n , and repeat the experiment m times. The spatio-temporal data set of the gaze points of the subjects during the entire interaction process is recorded by the eye tracker where In the formula, x and y are the planar landing points of the line of sight at time t, and t is the absolute time. Further, mathematical statistics are performed on the eye movement data to obtain the following indicators.

[0088] The number of times of line-of-sight deviation refers to the defined AOI area (in this solution, AOI refers to the entire front windshield area and the outer rearview mirror area). When the line of sight stays within this area, it is considered that the driver is focused on the road ahead and driving safely; otherwise, it is considered that the driver's attention is distracted due to in-cabin interaction, which is not conducive to driving safety. The gaze point data of the subjects can be obtained through the eye tracker and further converted into fixation points; when the first fixation point appears outside the AOI and returns to the AOI until the last fixation point, we call it a deviation. Record the number of times of line-of-sight deviation n in the mth test of the nth test case AOI-nm , the maximum single-line-of-sight deviation time T AOI-max-nm and the total duration T of line-of-sight deviation AOI-total-nm .

[0089] Step 9: Perform average weighting on the m test results of the Nth test case to obtain

[0090]

[0091] Step 10: Perform average weighting on the quantization results of the n test cases of the obtained objective test indicators to obtain the quantization test results of each test indicator.

[0092] The number of times of line-of-sight deviation

[0093] The maximum single-line-of-sight deviation time

[0094] The total duration of line-of-sight deviation

[0095] Step 11: Further, in addition to possible distraction of visual attention, the test subjects may complete the entire interaction process with the assistance of large-screen touch due to reasons such as unsmooth interaction process, slow response, or lack of function. The steering wheel off-hand monitoring device can monitor the driver's one-hand off-hand through the signal changes of multiple capacitive systems. Through the steering wheel off-hand monitoring device, the number of times n of the steering wheel off-hand in the m-th test of the n-th use case can be obtained HOD-nm , the maximum single off-hand time T HOD-max-nm and the total off-hand duration T HOD-total-nm .

[0096] Step 12: Perform average weighting on the m test results of the N-th test case to obtain

[0097]

[0098] Step 13: Perform average weighting on the quantization results of the n test cases of the obtained objective test indicators to obtain the quantization test results of each test indicator.

[0099] The number of times the steering wheel is off-hand

[0100] The maximum single off-hand time

[0101] The total off-hand duration

[0102] Step 14: For the final test results of the 6 objective test indicators, convert the objective values into scores from 0 to 1 through the rank score conversion method and the historical database. Specifically, where S is the final score, A is the number of test result records of this indicator in the database, and P is the ranking position of the test result of this indicator in the total records. Thus, the specific scores of the 6 objective indicators can be obtained.

[0103] Score of the number of times of line-of-sight deviation In the formula is the total number of records of the number of times of line-of-sight deviation data in the database,[[]] is the ranking position of the test result in the database.

[0104] Score of the maximum single line-of-sight deviation time In the formula is the total number of records of the maximum single line-of-sight deviation time value in the database,[[]] is the ranking position of the test result in the database.

[0105] Score of the total duration of line-of-sight deviation In the formula is the total number of records of the total duration of line-of-sight deviation value in the database,[[]] The ranking position of the test results in the database.

[0106] Score for the number of times the steering wheel is released Where is the total number of records of the number of times the steering wheel is released in the database, is the ranking position of the test results in the database.

[0107] Score for the maximum single release time Where is the total number of records of the maximum single release time value in the database, is the ranking position of the test results in the database.

[0108] Score for the total release duration Where is the total number of records of the total release duration value in the database, is the ranking position of the test results in the database.

[0109] Step 15: Similarly, for the 6 indicators used for result correction, the expert constructs a fuzzy complementary matrix R2 = (r ij ) 6×6 and a fuzzy consistency matrix Y2 = (y ij ) 6×6 , where R2 is the fuzzy complementary matrix for the above 6 indicators used for result correction; furthermore, the corresponding weight vector V2 is obtained, the scores of the 6 indicators form a score vector B2, and the correction coefficient is obtained through G2 = V2 × B2.

[0110] Step 16: Finally, the final test score of the in-vehicle voice interaction system is obtained through G = G1·G2.

[0111] Corresponding to the above method, as Figure 3 shown, this embodiment also provides an in-vehicle voice interaction system test system, including:

[0112] A system construction module for constructing a test index system; the test index system includes subjective test indicators, basic objective test indicators, and corrected objective test indicators;

[0113] A subjective and objective evaluation module that subjectively evaluates the subjective test indicators through a preset subjective scale to obtain the scores of each subjective test indicator, and respectively obtains the quantitative data of the basic objective test indicators and the corrected objective test indicators through objective measurement methods;

[0114] A score determination module, configured to perform weighted averaging on the quantization data of the basic objective test index and the corrected objective test index respectively to obtain weighted values, and respectively convert the weighted values of the basic objective test index and the corrected objective test index into standardized scores through the rank score conversion method in combination with the historical database;

[0115] A weight vector acquisition module, configured to determine the weight vectors of the subjective test index, the basic objective test index, and the corrected objective test index based on the fuzzy matrix;

[0116] A score vector acquisition module, configured to define the score of the subjective test index and the standardized score of the basic objective test index as a score vector, and define the standardized score of the corrected objective test index as a score vector;

[0117] A final score determination module, configured to determine the final test score according to the weight vector, the score vector, the weight vector, and the score vector.

[0118] This embodiment also provides a storage medium storing a computer program, and when the computer program is executed by a processor, it implements the above vehicle-mounted voice interaction system test method.

[0119] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0120] In this article, specific examples are used to elaborate on the principles and implementation manners of the present invention. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A test method for an in-vehicle voice interaction system, characterized in that, Including: Construct a test index system; the test index system includes subjective test indexes, basic objective test indexes, and modified objective test indexes; Subjectively evaluate the subjective test indexes through a preset subjective scale to obtain the scores of each subjective test index, and respectively obtain the quantitative data of the basic objective test indexes and the modified objective test indexes through objective measurement methods; Respectively perform weighted averaging on the quantitative data of the basic objective test indexes and the modified objective test indexes to obtain weighted values, and respectively convert the weighted values of the basic objective test indexes and the modified objective test indexes into standardized scores through the rank score conversion method in combination with the historical database; Determine the weight vector V1 of the subjective test indexes and the basic objective test indexes and the weight vector V2 of the modified objective test indexes based on the fuzzy matrix; Define the scores of the subjective test indexes and the standardized scores of the basic objective test indexes as the score vector B1, and define the standardized scores of the modified objective test indexes as the score vector B2; Determine the final test score according to the weight vector V1, the score vector B1, the weight vector V2, and the score vector B2.

2. The method for testing an in-vehicle voice interaction system according to claim 1, wherein The subjective test indicators include: voice naturalness, appropriateness of response complexity, and richness of functions; the basic objective test indicators include: wake-up success rate W n , false wake-up success rate NW n , recognition success rate I n and response time T n ; the corrected objective test indicators include: number of sight line deviations, maximum single sight line deviation time, total sight line deviation duration, number of times the steering wheel is released, maximum single release time, and total release duration.

3. The vehicle-mounted voice interaction system testing method according to claim 2, wherein The calculation formula for the quantitative data of the basic objective test indexes includes: W n＝ m1’ / m NW n＝ m2’ / m I n＝ m3’ / m Among them, m1’ is the number of successful wake-up times recorded by the system, m2’ is the number of false wake-up times recorded by the system, m3’ is the number of recognition times recorded by the system, m is the total number of repeated tests, T nm is the time interval of the m-th test, T nm = T nm2 - T nm1 , T nm1 is the moment when the m-th interaction instruction is sent, T nm2 is the moment when the m-th interactive system completes the response, W n is the wake-up success rate, NW n is the false wake-up success rate, I n is the recognition success rate, T n is the response time.

4. The method for testing an in-vehicle voice interaction system according to claim 2, wherein The calculation formula for the weighted value of the basic objective test indexes includes: Among them, W, NW, I, and T are the weighted values of the wake-up success rate W n , the weighted value of the false wake-up success rate NW n , the weighted value of the recognition success rate I n , and the weighted value of the response time T n , respectively, where n is the number of test cases.

5. The test method for the in-vehicle voice interaction system according to claim 2, wherein The calculation formula for the standardized score of the basic objective test indexes includes: Among them, A W is the total number of records of the weighted value of the wake-up success rate W n in the database, and P W is the ranking position of the test result of the current wake-up success rate in the database, and A NW is the total number of records of the weighted value of the false wake-up success rate NW n in the database, and P NW is the ranking position of the test result of the current false wake-up success rate in the database, and A I is the total number of records of the weighted value of the recognition success rate I n in the database, and P I is the ranking position of the test result of the current recognition success rate in the database, and A T is the total number of records of the weighted value of the response time T n in the database, and P T is the ranking position of the test result of the current response time in the database.

6. The method for testing an in-vehicle voice interaction system according to claim 2, wherein The calculation process for the weighted value of the modified objective test indexes includes: For the corrected objective test metric, the m test results of any test case are averaged and weighted to obtain the initial weight value of the corrected objective test metric; the calculation formula for the initial weight value of the corrected objective test metric is as follows: where m is the total number of repeated tests, and n AOI-n , T AOI-max-n , T AOI-total-n , n HOD-n , T HOD-max-n and T HOD-total-n are respectively the initial weight of the number of line-of-sight deviations n AOI-nm in the m-th test of the n-th test case, the initial weight of the maximum single-line-of-sight deviation time T AOI-max-nm in the m-th test of the n-th test case, the initial weight of the total line-of-sight deviation duration T AOI-total-nm in the m-th test of the n-th test case, the initial weight of the number of times the steering wheel is released n HOD-nm in the m-th test of the n-th test case, the initial weight of the maximum single release time T HOD-max-nm in the m-th test of the n-th test case, and the initial weight of the total release duration T HOD-total-nm in the m-th test of the n-th test case; Average and weight the initial weights of the obtained corrected objective test metrics to obtain the weighted values of the corrected objective test metrics; the calculation formula for the weighted values of the corrected objective test metrics is: where n AOI 、T AOI-max 、T AOI-total 、n HOD 、T HOD-max and T HOD-total are the weighted values of the number of times of line-of-sight deviation, the weighted value of the maximum single-line-of-sight deviation time, the weighted value of the total line-of-sight deviation duration, the weighted value of the number of times the steering wheel is released, the weighted value of the maximum single release time, and the weighted value of the total release duration, respectively.

7. The method for testing an in-vehicle voice interaction system according to claim 2, wherein The calculation formula for the standardized score of the modified objective test indexes includes: Among them, is the total number of records of the number of times of line-of-sight deviation in the database, is the ranking position of the current test result in the database, is the total number of records of the value of the maximum single line-of-sight deviation time in the database, is the ranking position of the current test result in the database, is the total number of records of the total line-of-sight deviation duration value in the database, is the ranking position of the current test result in the database, is the total number of records of the number of times of steering wheel hand-off in the database, is the ranking position of the current test result in the database, is the total number of records of the value of the maximum single hand-off time in the database, is the ranking position of the current test result in the database, is the total number of records of the total hand-off duration value in the database, is the ranking position of the current test result in the database, and are the standardized scores of the number of times of line-of-sight deviation, the maximum single line-of-sight deviation time, the total line-of-sight deviation duration, the number of times of steering wheel hand-off, the maximum single hand-off time, and the total hand-off duration, respectively.

8. The test method for an in-vehicle voice interaction system according to claim 1, wherein The calculation formula for the final test score G is: G = V1 × B1 · V2 × B2.

9. A test system for an in-vehicle voice interaction system, characterized in that, Including: A system construction module for constructing a test index system; the test index system includes subjective test indexes, basic objective test indexes, and modified objective test indexes; A subjective and objective evaluation module that subjectively evaluates the subjective test indexes through a preset subjective scale to obtain the scores of each subjective test index, and respectively obtains the quantitative data of the basic objective test indexes and the modified objective test indexes through objective measurement methods; A score determination module for respectively performing weighted averaging on the quantitative data of the basic objective test indexes and the modified objective test indexes to obtain weighted values, and respectively converting the weighted values of the basic objective test indexes and the modified objective test indexes into standardized scores through the rank score conversion method in combination with the historical database; A weight vector acquisition module for determining the weight vector V1 of the subjective test indexes and the basic objective test indexes and the weight vector V2 of the modified objective test indexes based on the fuzzy matrix; A score vector acquisition module for defining the scores of the subjective test indexes and the standardized scores of the basic objective test indexes as the score vector B1, and defining the standardized scores of the modified objective test indexes as the score vector B2; A final score determination module for determining the final test score according to the weight vector V1, the score vector B1, the weight vector V2, and the score vector B2.

10. A storage medium, characterized in that, A computer program is stored, and when the computer program is executed by a processor, it implements the vehicle-mounted voice interaction system testing method according to any one of claims 1 to 8.

Images