Voice interaction test method and device, vehicle, electronic equipment and storage medium

Abstract

The invention provides a voice interaction test method and device, a vehicle, electronic equipment and a storage medium. The method comprises the steps of collecting feedback voice played by a target vehicle and a bus signal of the target vehicle; performing voice feature comparison on the feedback voice and the standard voice, and performing signal comparison on the bus signal and the standard signal; and judging whether the target vehicle correctly executes the voice instruction based on the voice feature comparison result and the signal comparison result. Compared with the prior art, the voice feature comparison is performed on the feedback voice and the preset standard voice, the feedback voice is the most direct expression whether the voice interaction function is successfully executed or not, and whether the target vehicle accurately recognizes and responds to the voice instruction of the user or not can be accurately judged; the bus signal is compared with a preset standard signal, and whether the target vehicle executes the corresponding operation according to the expectation or not can be determined; therefore, comprehensive and accurate testing of the vehicle voice interaction system is realized.

Description

Technical Field

[0001] The present disclosure relates to the field of testing technology, and in particular to a method and device for voice interaction testing, a vehicle, an electronic device, and a storage medium. Background Art

[0002] With the rapid development of artificial intelligence and the Internet of Things (IoT), voice interaction, as a natural and convenient method of human-computer interaction, is being applied in a variety of fields, including smart homes, in-vehicle systems, and intelligent customer service. Voice interaction testing is a critical step in ensuring that voice systems accurately and reliably understand and execute user commands.

[0003] In vehicle systems, voice interaction can be tested by extracting internal logs, but this invasive testing method cannot accurately simulate the user's usage environment. Voice interaction testing mostly uses image recognition (capturing screen responses) or capturing motion responses (actuator action signals) for testing. However, image recognition can be interfered with by other elements on the screen, and capturing motion responses can also be affected by errors in the natural language processing process, which seriously affects the accuracy of voice interaction testing. Therefore, how to conduct comprehensive and accurate testing of voice interaction systems and improve the accuracy of voice interaction testing has become an urgent problem to be solved. Summary of the Invention

[0004] The present disclosure provides a method and apparatus for voice interaction testing, a vehicle, an electronic device, and a storage medium. Its primary purpose is to achieve comprehensive and accurate testing of a voice interaction system by simulating a user's usage environment, thereby improving the accuracy of voice interaction testing.

[0005] According to a first aspect of the present disclosure, a method for voice interaction testing is provided, comprising:

[0006] Collecting feedback voice played by a target vehicle and a bus signal of the target vehicle, wherein the feedback voice is instruction execution content generated after the target vehicle executes a voice instruction;

[0007] Comparing the speech characteristics of the feedback speech with the standard speech to generate a speech characteristic comparison result;

[0008] Performing a signal comparison between the bus signal and a standard signal to generate a signal comparison result;

[0009] Based on the voice feature comparison result and the signal comparison result, it is determined whether the target vehicle correctly executes the voice command and a determination result is generated.

[0010] In some embodiments, comparing the speech features of the feedback speech with the standard speech to generate a speech feature comparison result includes:

[0011] Preprocessing the feedback speech and extracting Mel-frequency cepstral coefficient features of the preprocessed feedback speech;

[0012] Calculating the Euclidean distance between the Mel-cespectral coefficient feature of the feedback speech and the Mel-cespectral coefficient feature of the standard speech;

[0013] The similarity between the feedback speech and the standard speech is determined according to the Euclidean distance, and the speech feature comparison result is generated based on the similarity.

[0014] In some embodiments, before collecting the feedback voice played by the target vehicle and the bus signal of the target vehicle, the method further includes:

[0015] Determine the standard voice corresponding to each test command voice, extract the Mel-frequency cepstral coefficient features of each standard voice and generate a standard voice feature database, wherein the test command voice is processed by the target vehicle voice recognition to obtain the voice command.

[0016] In some embodiments, determining whether the target vehicle correctly executes the voice command based on the voice feature comparison result and the signal comparison result, and generating a determination result, includes:

[0017] Obtaining the similarity between the feedback speech and the standard speech in the speech feature comparison result, and generating a first result indicating that the speech feature comparison passes if the similarity exceeds a similarity threshold;

[0018] determining, based on the signal comparison result, whether the first identification information of the bus signal is consistent with the second identification information of the standard signal, and generating a second result indicating that the signal comparison is passed if the first identification information is consistent with the second identification information;

[0019] A determination result is generated based on the first result and the second result that the target vehicle correctly executes the voice command.

[0020] In some embodiments, the method further comprises:

[0021] Obtaining performance evaluation indicators of the target vehicle during the test, wherein the performance evaluation indicators include wake-up rate, wake-up delay, command response delay, command execution delay, the number of times the target vehicle correctly executes voice commands, and the number of times the voice commands are executed;

[0022] Based on the performance evaluation index, the success rate of the voice command execution is calculated and it is determined whether the target vehicle passes the test.

[0023] According to a second aspect of the present disclosure, a device for voice interaction testing is provided, comprising:

[0024] a collection unit, configured to collect feedback voice played by a target vehicle and a bus signal of the target vehicle, wherein the feedback voice is instruction execution content generated after the target vehicle executes a voice instruction;

[0025] A first comparison unit is configured to compare voice features of the feedback voice with a standard voice to generate a voice feature comparison result;

[0026] A second comparison unit is used to perform signal comparison on the bus signal standard signal to generate a signal comparison result;

[0027] A judgment unit is used to judge whether the target vehicle correctly executes the voice command based on the voice feature comparison result and the signal comparison result, and generate a judgment result.

[0028] In some embodiments, the first comparison unit comprises:

[0029] An extraction module, configured to pre-process the feedback speech and extract Mel-cepstral coefficient features of the pre-processed feedback speech;

[0030] A calculation module, configured to calculate the Euclidean distance between the Mel-cespectral coefficient feature of the feedback speech and the Mel-cespectral coefficient feature of the standard speech;

[0031] The first generating module is configured to determine the similarity between the feedback speech and the standard speech according to the Euclidean distance, and generate the speech feature comparison result.

[0032] In some embodiments, the apparatus further comprises:

[0033] The extraction unit is used to determine the standard voice corresponding to each test command voice before collecting the feedback voice played by the target vehicle and the bus signal of the target vehicle, extract the Mel-frequency cepstral coefficient features of each standard voice and generate a standard voice feature database, wherein the test command voice is obtained after the target vehicle voice recognition processing.

[0034] In some embodiments, the determining unit includes:

[0035] A second generating module is configured to obtain a similarity between the feedback speech and the standard speech in the speech feature comparison result, and generate a first result indicating that the speech feature comparison has passed if the similarity exceeds a similarity threshold;

[0036] a third generating module, configured to determine, based on the signal comparison result, whether the first identification information of the bus signal is consistent with the second identification information of the standard signal, and generate a second result indicating that the signal comparison is passed if the first identification information is consistent with the second identification information;

[0037] The fourth generating module is used to generate a judgment result of whether the target vehicle correctly executes the voice command based on the first result and the second result.

[0038] In some embodiments, the apparatus further comprises:

[0039] an acquisition unit, configured to acquire performance evaluation indicators of the target vehicle during the test, wherein the performance evaluation indicators include wake-up rate, wake-up delay, command response delay, command execution delay, the number of times the target vehicle correctly executes voice commands, and the number of times the voice commands are executed;

[0040] A determination unit is used to calculate the success rate of the voice command execution based on the performance evaluation index and determine whether the target vehicle passes the test.

[0041] According to a third aspect of the present disclosure, a vehicle is provided, comprising the apparatus for voice interaction testing as described in the second aspect.

[0042] According to a fourth aspect of the present disclosure, there is provided an electronic device, including:

[0043] at least one processor; and

[0044] a memory communicatively connected to the at least one processor; wherein,

[0045] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.

[0046] According to a fifth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0047] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described in the first aspect above.

[0048] The present disclosure provides a method and apparatus for voice interaction testing, a vehicle, an electronic device, and a storage medium. The method collects feedback voice played by a target vehicle and a bus signal of the target vehicle, wherein the feedback voice is the command execution content generated after the target vehicle executes a voice command; compares the voice characteristics of the feedback voice with a standard voice to generate a voice characteristic comparison result; compares the bus signal with the standard signal to generate a signal comparison result; and determines whether the target vehicle correctly executes the voice command based on the voice characteristic comparison result and the signal comparison result, and generates a judgment result. Compared with the related art, the present disclosure compares the voice characteristics of the feedback voice with a preset standard voice. The feedback voice is the most direct manifestation of the success of the voice interaction function, and can accurately determine whether the target vehicle accurately recognizes and responds to the user's voice command; compares the bus signal with the preset standard signal to confirm whether the target vehicle executes the corresponding operation as expected; and by combining and analyzing the voice characteristic comparison result and the signal comparison result, it can ensure that the test not only covers the accuracy of voice recognition, but also takes into account the reliability of the vehicle execution system, thereby achieving a comprehensive and accurate test of the vehicle voice interaction system.

[0049] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0051] Figure 1 A flowchart of a method for voice interaction testing provided in an embodiment of the present disclosure;

[0052] Figure 2 A flowchart of another method for voice interaction testing provided by an embodiment of the present disclosure;

[0053] Figure 3 This is a schematic diagram of the framework of a voice interaction test system;

[0054] Figure 4 A schematic diagram of the structure of a device for voice interaction testing provided in an embodiment of the present disclosure;

[0055] Figure 5 A schematic structural diagram of another apparatus for voice interaction testing provided in an embodiment of the present disclosure;

[0056] Figure 6 A schematic block diagram of an exemplary electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0058] Voice interaction testing verifies whether the voice interaction system can correctly perform its intended functions. This includes testing the system's recognition accuracy, response time, and result correctness for different voice commands, thereby ensuring that the user's voice commands are accurately executed.

[0059] By testing and optimizing the voice interaction system, we can make it more in line with user habits and needs. This includes improving voice recognition accuracy, optimizing system response time, and refining the interactive interface, thereby enhancing user satisfaction and experience when using the voice interaction system.

[0060] In actual applications, voice interaction systems may face various complex environments and conditions, such as noise interference and accent differences. Through testing, potential problems in the system under these conditions can be discovered and resolved, thereby ensuring system stability.

[0061] The following describes the method and device for voice interaction testing, vehicle, electronic device, and storage medium of the embodiments of the present disclosure with reference to the accompanying drawings.

[0062] Figure 1 A flowchart of a method for voice interaction testing provided in an embodiment of the present disclosure.

[0063] like Figure 1 As shown, the method comprises the following steps:

[0064] Step 101 : collecting feedback voice played by a target vehicle and a bus signal of the target vehicle, wherein the feedback voice is instruction execution content generated after the target vehicle executes a voice instruction.

[0065] In an embodiment of the present disclosure, in order to ensure that the target vehicle can accurately execute voice commands and provide clear and accurate feedback to the user, the tester needs to collect the feedback voice played by the target vehicle after executing the voice command. This feedback voice is usually a broadcast voice generated by speech synthesis technology after the vehicle executes the user's voice command, which is used to inform the user of the execution result or content of the command. For example, if the user's voice command is "navigate to the city center", then after the target vehicle executes this command, it may play a feedback voice, such as "A route to the city center has been planned for you, and the estimated driving time is 20 minutes". This feedback voice contains the content of the command execution, that is, the navigation destination and the estimated driving time.

[0066] Bus signals communicate between various systems within the vehicle, containing various status information and control instructions when the vehicle executes commands. By collecting bus signals, testers can understand the vehicle's internal operating status when executing voice commands and the coordination between various systems. Voice feedback reflects the voice interaction system's understanding and response to user commands, while bus signals reflect the status of the vehicle's internal systems during actual execution. Combining these two methods ensures that testing not only covers the accuracy of voice recognition but also considers the reliability of the vehicle's execution system, thereby achieving a more comprehensive assessment of the vehicle's voice interaction system.

[0067] Step 102: Compare the voice features of the feedback voice with the standard voice to generate a voice feature comparison result.

[0068] In the embodiments of the present disclosure, in order to ensure that the feedback voice played by the target vehicle after executing the voice command is consistent with the expected standard voice, and that the bus signal is consistent with the preset standard signal, it is necessary to perform a detailed comparison and analysis of the captured feedback voice and the bus signal. When comparing the feedback voice, the feedback voice is feature extracted. The features of the feedback voice can be extracted using Mel-Frequency Cepstral Coefficients (MFCC) or Perceptual Linear Prediction (PLP). It should be noted that the embodiments of the present disclosure do not limit the use of features of any sound category for comparison processing.

[0069] Step 103: perform signal comparison between the bus signal and a standard signal, and generate a signal comparison result.

[0070] In an embodiment of the present disclosure, during bus signal comparison, testers compare collected bus signals against preset standard signals one by one. This comparison can determine whether the vehicle's internal systems function as expected when executing voice commands. In other embodiments, this comparative analysis can also be used to determine whether any potential faults or issues exist.

[0071] Step 104 : Based on the voice feature comparison result and the signal comparison result, determine whether the target vehicle correctly executes the voice command, and generate a determination result.

[0072] In an embodiment of the present disclosure, after comparing the voice features of the feedback voice with the standard voice, a quantitative index of the similarity between the feedback voice and the preset standard voice will be obtained. This index is usually calculated based on the difference in a series of acoustic parameter features. If the difference between the feedback voice and the standard voice in these acoustic parameter features is small, that is, the similarity is high, then it can be preliminarily considered that the target vehicle has correctly executed the instruction in terms of voice output. In this embodiment, the similarity can be calculated by calculating the Euclidean distance of the features extracted between the feedback voice and the preset standard voice, or by using a dynamic time warping (DTW) algorithm. The embodiment of the present disclosure does not limit the method used to calculate the similarity.

[0073] The bus signal comparison results can be verified by comparing key parameters such as, but not limited to, timing, amplitude, and waveform to determine if the target vehicle's intended signal matches the standard signal generated when executing the command. If the actual bus signal and the standard signal match or differ within an acceptable range in these parameters, the target vehicle's internal systems are operating as expected when executing the voice command.

[0074] By comprehensively considering the results of voice feature comparison and signal comparison, if both indicate that the target vehicle correctly executed the voice command, it can be determined that the target vehicle correctly executed the voice command. It is important to note that the results of voice feature comparison and signal comparison can be affected by various factors, such as ambient noise and vehicle hardware differences. Therefore, when making the final judgment, testers need to comprehensively evaluate the target vehicle's performance based on actual conditions and other test data.

[0075] The present disclosure provides a method for voice interaction testing, which includes collecting feedback voice played by a target vehicle and a bus signal of the target vehicle, wherein the feedback voice is the command execution content generated after the target vehicle executes a voice command; performing voice feature comparison on the feedback voice with a standard voice to generate a voice feature comparison result; performing signal comparison on the bus signal with the standard signal to generate a signal comparison result; and judging whether the target vehicle correctly executes the voice command based on the voice feature comparison result and the signal comparison result, and generating a judgment result. Compared with the related art, the present disclosure compares the voice features of the feedback voice with a preset standard voice. The feedback voice is the most direct manifestation of the success of the voice interaction function, and can accurately judge whether the target vehicle accurately recognizes and responds to the user's voice command; and comparing the bus signal with the preset standard signal to confirm whether the target vehicle executes the corresponding operation as expected; by combining and analyzing the voice feature comparison result and the signal comparison result, it can be ensured that the test not only covers the accuracy of voice recognition, but also considers the reliability of the vehicle execution system, thereby achieving a comprehensive and accurate test of the vehicle voice interaction system.

[0076] In order to clearly illustrate the embodiments of the present disclosure, this embodiment provides a flowchart of another method for voice interaction testing.

[0077] like Figure 2 As shown, the method comprises the following steps:

[0078] Step 201 , determining the standard speech corresponding to each test instruction speech, extracting the Mel-frequency cepstral coefficient features of each of the standard speech and generating a standard speech feature database, wherein the test instruction speech is processed by the target vehicle speech recognition to obtain the speech instruction.

[0079] Specifically, in step 201, before testing the voice interaction system, a corresponding standard voice is determined for each test command. These standard voices typically accurately reflect the expected results after the command is executed. Once the standard voices are determined, the Mel-frequency Cepstral Coefficient (MFCC) features of each standard voice can be extracted.

[0080] Mel-frequency cepstral coefficients are a widely used feature in speech processing. They can effectively describe the frequency content of speech signals, especially the part related to human auditory perception. The process of extracting MFCC features usually includes the following steps:

[0081] Preprocessing: First, the standard speech is preprocessed, including removing noise, standardizing volume, etc., to ensure that the extracted features are not affected by these factors.

[0082] Framing: Segment the speech signal into short time frames, usually between 20 and 30 milliseconds in length, in order to analyze the local characteristics of the speech.

[0083] Apply a Mel filter bank: Each frame of speech signal is passed through a set of Mel filters that simulate the human ear's perception of different frequencies.

[0084] Take the logarithm and perform discrete cosine transform (DCT): Take the logarithm of the signal after the Mel filter bank processing, and then perform discrete cosine transform to obtain the Mel cepstral coefficients.

[0085] After extracting the MFCC features for each standard speech, these features are stored in a database, forming a standard speech feature database. This database serves as a reference for comparison with actual feedback speech during subsequent testing. By comparing the MFCC features of the feedback speech played after the test command is executed with the standard features in the database, the performance of the voice interaction system in real-world applications can be evaluated.

[0086] Step 202 : Load a test case and play a test instruction voice corresponding to the test case so that the target vehicle executes the corresponding voice instruction in response to the test instruction voice.

[0087] Specifically, in step 202, predefined test cases are loaded. These test cases usually contain various types of voice commands, covering various scenarios and requirements that users may encounter in actual use. Each test case corresponds to one or more specific test command voices. Figure 3 , Figure 3 This is a schematic diagram of the framework of a voice interaction test system. The system's artificial mouth (a speaker that simulates human speech) plays test command audio. The target vehicle's onboard microphone receives the test command audio and generates voice commands through processes such as speech recognition and natural language processing.

[0088] By loading and playing the test command voice corresponding to the test case, testers can simulate the voice commands that users would issue to the target vehicle in actual use, thereby evaluating the target vehicle's performance in real-world applications. This testing method not only helps the development team identify potential problems and defects, but also provides a strong basis for subsequent optimization and improvement.

[0089] Step 203: pre-process the feedback speech and extract Mel-frequency cepstral coefficient features of the pre-processed feedback speech.

[0090] Specifically, in step 203, by extracting Mel-frequency cepstral coefficient features from the collected feedback voice, it can be determined whether the feedback voice played by the target vehicle after executing the voice command is consistent with the expected standard voice.

[0091] The process of Mel-cepstral coefficient feature extraction includes the following steps:

[0092] Preprocessing: First, the standard speech is preprocessed, including removing noise, standardizing volume, etc., to ensure that the extracted features are not affected by these factors.

[0093] Framing: Segment the speech signal into short time frames, usually between 20 and 30 milliseconds in length, in order to analyze the local characteristics of the speech.

[0094] Apply a Mel filter bank: Each frame of speech signal is passed through a set of Mel filters that simulate the human ear's perception of different frequencies.

[0095] Take the logarithm and perform discrete cosine transform (DCT): Take the logarithm of the signal after the Mel filter bank processing, and then perform discrete cosine transform to obtain the Mel cepstral coefficients.

[0096] The extracted MFCC features reflect the spectral characteristics and sound quality of the feedback speech, providing basic data for subsequent speech feature comparison. By comparing the MFCC features of the actual feedback speech with the MFCC features of the standard speech, the accuracy of the target vehicle's feedback after executing the voice command and the quality of the speech synthesis can be evaluated.

[0097] Step 204 : Calculate the Euclidean distance between the Mel-cepstral coefficient feature of the feedback speech and the Mel-cepstral coefficient feature of the standard speech to determine the similarity between the feedback speech and the standard speech, and generate the speech feature comparison result based on the similarity.

[0098] Step 205: Obtain the similarity between the feedback speech and the standard speech in the speech feature comparison result, and generate a first result indicating that the speech feature comparison is passed if the similarity exceeds a similarity threshold.

[0099] Specifically, in steps 204 to 205, in order to quantitatively evaluate the similarity between the actual feedback speech and the preset standard speech, the Euclidean distance between the Mel-frequency cepstral coefficient (MFCC) features of the two is usually calculated. The calculation formula for the Euclidean distance is the square root of the sum of the squares of the differences in each dimension between the two points. For an MFCC feature vector, each dimension represents the energy distribution in a specific Mel-frequency band. Therefore, calculating the Euclidean distance between MFCC feature vectors is to measure the difference in the energy distribution of the two vectors in each Mel-frequency band.

[0100] The smaller the Euclidean distance, the smaller the difference in MFCC features between the actual feedback speech and the standard speech, and the higher the similarity. By setting a similarity threshold (or comparing relative distances), it is possible to determine whether the actual feedback speech meets the required similarity with the standard speech, thereby evaluating the accuracy of the feedback speech provided by the target vehicle after executing the voice command. If the similarity between the feedback speech and the standard speech exceeds the similarity threshold, the voice feature comparison is determined to pass, and a first result is generated, indicating that the target vehicle is deemed to have correctly executed the voice command in terms of voice output.

[0101] Step 206: perform signal comparison between the bus signal and the standard signal.

[0102] Step 207: Based on the signal comparison result, determine whether the first identification information of the bus signal is consistent with the second identification information of the standard signal, and generate a second result indicating that the signal comparison is passed if the first identification information is consistent with the second identification information.

[0103] Specifically, in steps 206 to 207, for the comparison results of the bus signal, it can be determined that the standard signal that the target vehicle should generate when executing the instruction is consistent with the standard signal by comparing key parameters such as the timing, amplitude and waveform of the signal, but not limited to. If the bus signal actually collected is consistent with the standard signal in these parameters or the difference is within an acceptable range, then it can be determined that the working state of the internal system of the target vehicle is in line with expectations when executing the voice instruction. In the voice interaction test system, the test software in the system compares the identification information of the bus signal with the identification information of the standard signal, and determines whether the signal comparison passes by comparing the first identification information of the bus signal with the second identification information of the standard signal, and generates a second result if it is determined to pass.

[0104] For example, the voice command is "Control Adjust Left Front Window (100%)", and the corresponding control signal identification information in the bus is HuFrtLWdwPosiCtrl=100%. During the test, after the corresponding test command voice is broadcast, the target vehicle will recognize it. Under normal circumstances, the voice command received is "Control Adjust Left Front Window (100%)". However, due to errors in voice recognition and other reasons, it may be recognized as a different voice command, such as "Control Adjust Left Front Window (50%)". In this case, the target vehicle will execute according to "Control Adjust Left Front Window (50%)", and the identification information in the bus signal is HuFrtLWdwPosiCtrl=50%. However, the identification information (HuFrtLWdwPosiCtrl=100%) corresponding to the expected standard signal (Control Adjust Left Front Window (100%)) is different. Therefore, it is determined that the signal comparison fails.

[0105] Step 208 : Generate a judgment result that the target vehicle correctly executes the voice command based on the first result and the second result.

[0106] Specifically, in step 208, if the voice feature comparison passes, it means that the actual feedback voice and the preset standard voice are highly similar in acoustic characteristics. This is usually achieved by comparing the Mel-frequency cepstral coefficients (MFCC) or other acoustic features of the two. If the signal comparison passes, it means that when the target vehicle executes the voice command, its internal bus signal is consistent with the preset standard bus signal. It can be considered that when the target vehicle executes the command, the working state of its internal system is normal and no faults or errors occur. When both the voice feature comparison and the signal comparison pass, it can be comprehensively judged that the target vehicle has correctly executed the user's voice command.

[0107] Step 209: Obtain performance evaluation indicators of the target vehicle during the test, wherein the performance evaluation indicators include wake-up rate, wake-up delay, command response delay, command execution delay, the number of times the target vehicle correctly executes voice commands, and the number of times the voice commands are executed.

[0108] Specifically, in step 209, when evaluating the target vehicle's performance during voice interaction testing, a series of performance evaluation metrics are required. These metrics comprehensively reflect the target vehicle's performance in recognizing, responding to, and executing voice commands. Arousal rate is a key metric, measuring the percentage of the target vehicle that successfully recognizes and responds to user awakening commands. A higher awakening rate indicates a stronger ability of the target vehicle to recognize user intent, resulting in a smoother user experience.

[0109] Wake-up latency refers to the time delay between the user issuing a wake-up command and the target vehicle successfully waking up and beginning to process the command. This metric reflects the target vehicle's response speed and processing efficiency. A shorter wake-up latency means the target vehicle can respond to user requests more quickly, improving the smoothness of the interaction.

[0110] Command response latency and command execution latency are also key performance indicators. Command response latency refers to the time delay between a user issuing a specific command and the target vehicle recognizing and confirming it, while command execution latency refers to the time it takes from the target vehicle confirming the command to actually executing it. Together, these two metrics reflect the target vehicle's efficiency and ability in understanding and executing user commands.

[0111] The number of times a target vehicle correctly executes a voice command and the number of times it executes a voice command are also important evaluation metrics. By comparing these two data points, we can calculate the correct execution rate, which is the proportion of voice commands successfully executed by the target vehicle. This metric directly reflects the accuracy and reliability of the target vehicle in voice interaction.

[0112] Step 210 : Based on the performance evaluation index, calculate the success rate of the voice command execution and determine whether the target vehicle passes the test.

[0113] Specifically, in step 210, to calculate the voice command execution success rate, we need to focus on two key metrics: the number of times the target vehicle correctly executed the voice command and the total number of voice command executions. The voice command execution success rate is the ratio of the number of correct executions to the total number of executions. The higher this ratio, the more accurate the target vehicle is in executing the voice command. The calculation formula is as follows:

[0114]

[0115] Among them, ε2 is the number of times the target vehicle correctly executes the voice command, and ε1 is the total number of times the voice command is executed.

[0116] This calculated success rate is used to determine whether the target vehicle has passed the test. A passing threshold can be set as the judgment standard. If the calculated voice command execution success rate is greater than or equal to this threshold, the target vehicle is considered to have passed the test, meaning that its voice interaction performance meets the preset requirements.

[0117] It should be noted that the embodiments of the present disclosure may include multiple steps. For the convenience of description, these steps are numbered, but these numbers do not limit the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of the present disclosure do not limit this.

[0118] Corresponding to the above-mentioned method for voice interaction testing, the present invention also provides a device for voice interaction testing. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment and will not be repeated in this invention.

[0119] Figure 4 A schematic diagram of a structure of a voice interaction test device provided in an embodiment of the present disclosure, such as Figure 4 Shown, including:

[0120] The collecting unit 31 is used to collect the feedback voice played by the target vehicle and the bus signal of the target vehicle, wherein the feedback voice is the voice broadcast of the command execution content played after the target vehicle executes the voice command;

[0121] A first comparison unit 32 is configured to compare the speech characteristics of the feedback speech with the standard speech to generate a speech characteristic comparison result;

[0122] A second comparison unit 33 is used to perform signal comparison on the bus signal standard signal to generate a signal comparison result;

[0123] The judgment unit 34 is used to judge whether the target vehicle correctly executes the voice command based on the voice feature comparison result and the signal comparison result, and generate a judgment result.

[0124] The present disclosure provides a device for voice interaction testing, which collects feedback voice played by a target vehicle and a bus signal of the target vehicle, wherein the feedback voice is the command execution content generated after the target vehicle executes a voice command; compares the voice characteristics of the feedback voice with a standard voice to generate a voice characteristic comparison result; compares the bus signal with the standard signal to generate a signal comparison result; and determines whether the target vehicle correctly executes the voice command based on the voice characteristic comparison result and the signal comparison result, and generates a judgment result. Compared with the related art, the present disclosure compares the voice characteristics of the feedback voice with a preset standard voice. The feedback voice is the most direct manifestation of the success of the voice interaction function, and can accurately determine whether the target vehicle accurately recognizes and responds to the user's voice command; compares the bus signal with the preset standard signal to confirm whether the target vehicle executes the corresponding operation as expected; and by combining and analyzing the voice characteristic comparison result and the signal comparison result, it can ensure that the test not only covers the accuracy of voice recognition, but also considers the reliability of the vehicle execution system, thereby achieving a comprehensive and accurate test of the vehicle voice interaction system.

[0125] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the first comparison unit 32 includes:

[0126] An extraction module 321 is configured to pre-process the feedback speech and extract Mel-cepstral coefficient features of the pre-processed feedback speech;

[0127] A calculation module 322 is configured to calculate a Euclidean distance between the Mel-cepstral coefficient feature of the feedback speech and the Mel-cepstral coefficient feature of the standard speech;

[0128] The first generating module 323 is configured to determine the similarity between the feedback speech and the standard speech according to the Euclidean distance, and generate the speech feature comparison result.

[0129] In some embodiments, the apparatus further comprises:

[0130] The extraction unit 35 is used to determine the standard voice corresponding to each test command voice before collecting the feedback voice played by the target vehicle and the bus signal of the target vehicle, extract the Mel-frequency cepstral coefficient features of each standard voice and generate a standard voice feature database, wherein the test command voice is obtained after the target vehicle voice recognition processing.

[0131] In some embodiments, the determining unit 34 includes:

[0132] The second generating module 341 is configured to obtain the similarity between the feedback speech and the standard speech in the speech feature comparison result, and generate a first result indicating that the speech feature comparison has passed if the similarity exceeds a similarity threshold;

[0133] A third generating module 342 is configured to determine, based on the signal comparison result, whether the first identification information of the bus signal is consistent with the second identification information of the standard signal, and generate a second result indicating that the signal comparison is passed if the first identification information is consistent with the second identification information;

[0134] The fourth generating module 343 is configured to generate a judgment result of whether the target vehicle correctly executes the voice command based on the first result and the second result.

[0135] Furthermore, in a possible implementation of this embodiment, as Figure 5 As shown, the device also includes:

[0136] an acquisition unit 36 ​​for acquiring performance evaluation indicators of the target vehicle during the test, wherein the performance evaluation indicators include wake-up rate, wake-up delay, command response delay, command execution delay, the number of times the target vehicle correctly executes voice commands, and the number of times the voice commands are executed;

[0137] The determination unit 37 is configured to calculate the success rate of the voice command execution based on the performance evaluation index and determine whether the target vehicle has passed the test.

[0138] It should be noted that the above explanation of the method embodiment is also applicable to the device of this embodiment, and the principles are the same, which is not limited in this embodiment.

[0139] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0140] Figure 6A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0141] like Figure 6 As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 402 or a computer program loaded from a storage unit 408 into a RAM (Random Access Memory) 403. Various programs and data required for the operation of the device 400 can also be stored in the RAM 403. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An I / O (Input / Output) interface 405 is also connected to the bus 404.

[0142] Various components in device 400 are connected to I / O interface 405, including an input unit 406, such as a keyboard, mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a magnetic disk, optical disk, etc.; and a communication unit 409, such as a network card, modem, wireless communication transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0143] The computing unit 401 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units that run machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the method for voice interaction testing. For example, in some embodiments, the method for voice interaction testing can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the method described above can be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform the aforementioned voice interaction test method in any other appropriate manner (for example, by means of firmware).

[0144] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System on Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0145] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0146] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0147] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0148] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0149] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0150] It's important to note that artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). This encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0151] The various numerical numbers such as first and second involved in the present disclosure are only for the convenience of description and are not used to limit the scope of the embodiments of the present disclosure, and also indicate the order of precedence.

[0152] The at least one in the present disclosure can also be described as one or more, and the multiple can be two, three, four or more, which is not limited in the present disclosure. In the embodiments of the present disclosure, for a technical feature, the technical features in the technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", and there is no order of precedence or size between the technical features described by "first", "second", "third", "A", "B", "C" and "D".

[0153] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0154] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for voice interaction testing, characterized in that: include: Collecting feedback voice played by a target vehicle and a bus signal of the target vehicle, wherein the feedback voice is instruction execution content generated after the target vehicle executes a voice instruction; Comparing the speech characteristics of the feedback speech with the standard speech to generate a speech characteristic comparison result; Performing a signal comparison between the bus signal and a standard signal to generate a signal comparison result; Based on the voice feature comparison result and the signal comparison result, it is determined whether the target vehicle correctly executes the voice command and a determination result is generated.

2. The method according to claim 1, characterized in that The comparing the speech features of the feedback speech with the standard speech to generate a speech feature comparison result includes: Preprocessing the feedback speech and extracting Mel-frequency cepstral coefficient features of the preprocessed feedback speech; Calculating the Euclidean distance between the Mel-cespectral coefficient feature of the feedback speech and the Mel-cespectral coefficient feature of the standard speech; The similarity between the feedback speech and the standard speech is determined according to the Euclidean distance, and the speech feature comparison result is generated.

3. The method according to claim 1, characterized in that Before collecting the feedback voice played by the target vehicle and the bus signal of the target vehicle, the method further includes: Determine the standard voice corresponding to each test command voice, extract the Mel-frequency cepstral coefficient features of each standard voice and generate a standard voice feature database, wherein the test command voice is processed by the target vehicle voice recognition to obtain the voice command.

4. The method according to claim 1, wherein The determining whether the target vehicle correctly executes the voice command based on the voice feature comparison result and the signal comparison result, and generating a determination result, includes: Obtaining the similarity between the feedback speech and the standard speech in the speech feature comparison result, and generating a first result indicating that the speech feature comparison passes if the similarity exceeds a similarity threshold; determining, based on the signal comparison result, whether the first identification information of the bus signal is consistent with the second identification information of the standard signal, and generating a second result indicating that the signal comparison is passed if the first identification information is consistent with the second identification information; A determination result is generated based on the first result and the second result that the target vehicle correctly executes the voice command.

5. The method according to claim 1, wherein The method further comprises: Obtaining performance evaluation indicators of the target vehicle during the test, wherein the performance evaluation indicators include wake-up rate, wake-up delay, command response delay, command execution delay, the number of times the target vehicle correctly executes voice commands, and the number of times the voice commands are executed; Based on the performance evaluation index, the success rate of the voice command execution is calculated and it is determined whether the target vehicle passes the test.

6. A device for voice interaction testing, characterized in that: include: a collecting unit, configured to collect feedback voice played by the target vehicle and a bus signal of the target vehicle, wherein the feedback voice is instruction execution content generated after the target vehicle executes a voice instruction; A first comparison unit is configured to compare voice features of the feedback voice with a standard voice to generate a voice feature comparison result; a second comparing unit, configured to compare the bus signal with a standard signal to generate a signal comparison result; A judgment unit is used to judge whether the target vehicle correctly executes the voice command based on the voice feature comparison result and the signal comparison result, and generate a judgment result.

7. A vehicle, characterized in that: The vehicle includes: a device for voice interaction testing as described in claim 6.

8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images