A method, device, equipment and storage medium for testing a speech recognition system

Abstract

The present application provides a testing method, device, equipment and storage medium for a speech recognition system, wherein the testing method includes: inputting a test speech in a test data set corresponding to a test item of the system under test into the system under test, obtaining the operating data of the system under test on the test item, and determining the test result of the system under test on the test item according to the operating data of the system under test on the test item. Since the total test data set of the system under test is constructed on the basis of considering the actual application scenario of the system under test, with the goal of obtaining a test result that can more accurately reflect the capabilities of the system under test, therefore, based on the test data in the constructed total test data set, the system under test is tested using the test process provided by the present application, and better test results can be obtained. The present application is applicable to the design and implementation of testing of speech recognition systems or speech recognition subsystems of intelligent voice interaction applications by intelligent voice service providers, users and third-party testing agencies.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent voice interaction system testing, and in particular to a testing method, device, equipment and storage medium for a voice recognition system. Background Art

[0002] Intelligent voice interaction is widely used in smart homes, smart customer service, mobile terminals, car terminals, smart education, smart medical care, smart offices, service robots and many other fields, and has become one of the important ways of human-computer interaction.

[0003] As intelligent voice interaction penetrates deeper into all aspects of production and life, it is necessary to unify the system reference framework, basic technical requirements, Internet interface requirements, etc. of intelligent voice interaction. In this regard, the country has formulated basic national standards to support intelligent voice interaction systems. For example, GB / T 5271.29-2006 Information Technology Vocabulary Part 29: Artificial Intelligence Speech Recognition and Synthesis provides the terms and definitions of related concepts of speech recognition and synthesis in artificial intelligence.

[0004] However, with the rapid development of artificial intelligence, especially intelligent voice interaction, intelligent voice interaction systems are being used in more and more scenarios, and will be used in even more scenarios in the future. The widespread use of intelligent voice interaction systems has made it impossible for existing standards to provide good support for intelligent voice interaction systems.

[0005] Speech recognition is a basic link in intelligent speech interaction. For the application of speech recognition system, it is crucial to make it meet the requirements of application scenarios. Therefore, it is necessary to evaluate the ability of speech recognition system. At present, most of the speech recognition system evaluation schemes in the industry use temporarily collected data as test data to test the speech recognition system. However, since this evaluation scheme does not form a complete test data set, the test data is separated from the actual application scenario of the speech recognition system, which makes it impossible to accurately test the ability of the speech recognition system in actual application using these data, and the test effect is not ideal. Summary of the invention

[0006] In view of this, the present application provides a method, apparatus, device and storage medium for testing a speech recognition system, which are used to evaluate the capability of the speech recognition system. The technical solution is as follows:

[0007] A method for testing a speech recognition system, comprising:

[0008] The test speech in the test data set corresponding to the test item of the system under test is input into the system under test, wherein the test data set is obtained from the total test data set of the system under test according to the corresponding test item, the total number of test speech in the total test data set is at least 2000, the total test data set includes multiple types of test speech with different qualities, namely, type A test speech, type B test speech, type C test speech and type D test speech, the category to which the test speech belongs is determined by the speech type and / or audio type of the test speech, each type of test speech in the total test data set at least includes speech with audio type of weak noise normal and / or truncated audio, the audio type of weak noise normal refers to speech with a speech signal-to-noise ratio greater than or equal to 20dB, the proportion of type A test speech in the total test data set should not be less than 70%, and the proportion of type B test speech should not be less than 15% and should not be greater than 20%, The proportion of Class C test speech should not be less than 5% and should not be greater than 10%, the proportion of Class D test speech should not be greater than 5%, the proportion of test speech with a duration of 3 to 5 seconds in the total test data set should be more than 80%, and the number of speakers of each test speech type should not be less than 30;

[0009] Obtaining operation data of the system under test on the test item;

[0010] A test result of the system under test on the test item is determined according to the operating data of the system under test on the test item.

[0011] Optionally, the step of inputting a test speech in a test data set corresponding to a test item of the system under test into the system under test includes:

[0012] Playing the test voice in the test data set corresponding to the test item of the system under test by using a voice playback device, and inputting the test voice played by the voice playback device into the system under test;

[0013] Wherein, the speech playback device is a first playback device or a second playback device, the first playback device is a player, and the second playback device includes a power amplifier and an artificial mouth;

[0014] The first playback device meets the following parameter requirements:

[0015] Frequency response: 74Hz~18kHz; Maximum sound pressure level: 102dB(A);

[0016] The second playback device meets the following parameter requirements:

[0017] Signal-to-noise ratio: 90dB; Gain control: 0dB-25dB; Frequency response of the power amplifier: 20Hz-20kHz; Frequency response of the artificial mouth: 100Hz-10kHz; Maximum sound pressure level: 110 dB(A);

[0018] The environment in which the test voice in the test data set corresponding to the test item of the system under test is played by the voice playback device satisfies:

[0019] When the external noise does not exceed 55dB(A), the indoor background noise is ≤20dB(A);

[0020] The test method also includes:

[0021] While the test speech in the test data set corresponding to the test item of the system under test is played by the speech playback device, the background noise of the test speech is played by the noise playback device, wherein the background noise is recorded in the usage scenario of the system under test, and the noise playback device meets the following parameter requirements:

[0022] Power: 70W (peak 125W); frequency response: 50Hz~21kHz;

[0023] Sound pressure: ≤113dB SPL@1m; Input impedance: 10KΩ; Maximum input level: 22dBu;

[0024] The noise playback device is processed by frequency response equalization, and the frequency response difference between the playback noise at the voice acquisition module of the system under test and the noise in the application scenario of the system under test satisfies +3dB.

[0025] Optionally, the test items include functional test items and / or performance test items;

[0026] The functional test items include: one or more of: voice signal acquisition function, voice-to-text function, voice wake-up function, front-end signal processing function, speaker separation function, language information recognition function and voice recognition post-processing function, among which:

[0027] The voice wake-up function includes one or more of the following functions: a wake-up function based on a predefined wake-up word, a wake-up function based on a custom wake-up word, a wake-up function based on multiple wake-up words, a function of monitoring multiple audio streams while performing voice wake-up, a function of waking up based on voice interruption, and a collaborative wake-up function, wherein the collaborative wake-up function means that when multiple devices using the same wake-up word appear in the same scene, only one device responds to a wake-up operation; the front-end signal processing function includes one or more of the following functions: a voice encoding and decoding function, a voice endpoint detection function, a voice enhancement function, a sound source localization function, a sound The function of speech recognition includes one or more of the following functions: a speaker segmentation function and a speaker clustering function; the function of speech recognition includes one or more of the following functions: a one-language or multiple-language recognition function, a multi-language mixed reading recognition function, a one-language or multiple-language recognition function, a multi-language mixed reading recognition function, and a language information endpoint recognition function; the function of speech recognition post-processing includes one or more of the following functions: a character regularization function, a punctuation prediction function, a text smoothing function, and a text replacement function;

[0028] If the test items of the system under test include the functional test items, the operation data of the system under test on the functional test items include: the operation results generated by the system under test on the functional test items for the input test speech;

[0029] The step of determining the test result of the system under test on the test item according to the running data of the system under test on the test item includes:

[0030] Analyzing whether the system under test provides the function indicated by the functional test item according to the running data of the system under test on the functional test item; and generating a test result of the system under test on the test item according to the analysis result;

[0031] The performance test items include: one or more of speech recognition effect, speech recognition efficiency, speech wake-up effect, front-end signal processing effect, speaker separation effect, language information recognition effect and system stability;

[0032] If the performance test item includes a first performance test item, the operating data of the system under test on the first performance test item includes: the operating result generated by the system under test on the first functional test item for the input test voice, wherein the first performance test item includes one or more test items of speech recognition effect, speech recognition efficiency, speech wake-up effect, front-end signal processing effect, speaker separation effect, and language information recognition effect; If the system under test includes a second performance test item, the operating data of the system under test on the second performance test item includes: data characterizing the operating status and / or resource utilization of the system under test obtained during the test of the system under test based on the test voice in the test data set corresponding to the second performance test item, wherein the second performance test item includes system stability;

[0033] The step of determining the test result corresponding to the test item of the system under test according to the running data of the system under test on the test item includes:

[0034] If the performance test items include the first performance test item, then based on the operating data of the system under test on the first performance test item, the performance characterization parameters of the system under test on the first performance test item are determined, and the test results of the system under test on the first performance test item are generated based on the determined performance characterization parameters; if the performance test items include the second performance test item, then based on the operating data of the system under test on the second performance test item, the test results of the system under test on the second performance test item are generated.

[0035] Optionally, analyzing whether the system under test provides the function indicated by the functional test item according to the running data of the system under test on the functional test item includes:

[0036] If the functional test item includes the voice signal acquisition function, then according to the operating result of the system under test on the voice signal acquisition function for the input test voice, analyze whether the system under test provides the voice signal acquisition function under the set pickup distance, wherein the system under test and the voice playback device are arranged according to the set pickup distance, the pickup distance is set according to the application scenario of the system under test, and the voice playback device is used to play the test voice in the test data set corresponding to the voice signal acquisition function, so that the system under test can collect it;

[0037] If the functional test item includes the speech-to-text function, then analyzing whether the system under test provides a function of converting the received speech into text that matches the content of the received speech and outputting the text according to the operating result of the system under test on the speech-to-text function for the input test speech;

[0038] If the functional test item includes the voice wake-up function, analyzing whether the system under test provides the voice wake-up function according to the running result generated by the system under test on the voice wake-up function for the input test voice;

[0039] If the functional test item includes the front-end signal processing function, analyzing whether the system under test provides the front-end signal processing function according to the operating result generated by the system under test on the front-end signal processing function for the input test voice;

[0040] If the functional test item includes the speaker separation function, analyzing whether the system under test provides the speaker separation function according to the operating result of the system under test on the speaker separation function for the input test speech, wherein the test data set corresponding to the speaker separation function includes test speech of multiple speakers talking alternately;

[0041] If the functional test item includes the language information recognition function, analyzing whether the system under test provides the language information recognition function according to the operating result of the system under test on the language information recognition function for the input test speech;

[0042] If the functional test item includes a speech recognition post-processing function, analyzing whether the system under test provides the speech recognition post-processing function according to the running result of the system under test on the speech recognition post-processing function for the input test speech;

[0043] The step of determining the performance characterization parameter of the system under test on the first performance test item according to the operation data of the system under test on the first performance test item includes:

[0044] If the first performance test item includes the speech recognition effect, then according to the operating result of the system under test on the speech recognition effect for the test speech input and the speech content annotated text of the test speech input to the system under test, determine the parameters that can characterize the word recognition ability and / or sentence recognition ability of the system under test;

[0045] If the first performance test item includes the speech recognition efficiency, determining a parameter capable of characterizing the recognition response and / or recognition throughput of the system under test according to the recognition response time of the system under test to the input test speech;

[0046] If the first performance test item includes the voice wake-up effect, determining parameters that can characterize the voice wake-up capability, and / or voice interruption capability, and / or voice interruption wake-up capability, and / or collaborative wake-up capability of the system under test according to the wake-up condition of the system under test on the input test voice;

[0047] If the first performance test item includes the front-end signal processing effect, then according to the audio bit rate before and after the test speech encoding input to the system under test, determine the parameter that can characterize the speech encoding and decoding capability of the system under test, and / or, according to the endpoint detection result of the test speech input to the system under test, determine the parameter that can characterize the endpoint detection capability of the system under test, and / or, according to the signal-to-noise ratio before and after the test speech enhancement input to the system under test and / or the noise amplitude before and after the test speech enhancement input to the system under test, determine the parameter that can characterize the speech enhancement capability of the system under test, and / or, according to the number of correct sound source localization of the input test speech and the total number of sound source localization requests of the system under test, determine the parameter that can characterize the sound source localization capability of the system under test, and / or according to the expected echo signal and error signal corresponding to the test speech input to the system under test, determine the parameter that can characterize the echo cancellation capability of the system under test;

[0048] If the first performance test item includes the speaker separation effect, determining a parameter capable of characterizing the speaker separation capability of the system under test according to the duration of the speech segment with incorrect separation and the duration of the entire valid speech segment in the speaker separation result of the test speech input by the system under test;

[0049] If the first performance test item includes the language information recognition effect, then the parameters that can characterize the recognition ability of the system under test for the specified type of language information are determined based on the number of times the specified type of language information is correctly recognized by the system under test and the total number of times the system under test recognizes the specified type of language information.

[0050] Optionally, determining a parameter capable of characterizing a recognition response and / or a recognition throughput of the system under test according to a recognition response time of the system under test to an input test speech includes:

[0051] According to the recognition response time of all test voices input to the system under test and the total number of all test voices input to the system under test, and the calculation method shown in the following formula, the average speech recognition response time T is determined. avr :

[0052]

[0053] in, represents the time when the system under test gives the recognition result of the i-th test speech input, represents the time when the i-th test speech input into the system under test ends, and N represents the total number of test speech inputs into the system under test;

[0054] And / or, according to the clause recognition response time of all test speech input to the system under test and the total number of all test speech input to the system under test, and the calculation method shown in the following formula, determine the average clause response time T of speech recognition avsr :

[0055]

[0056] The recognition response time of a clause of a test speech is the time it takes for the system under test to give a recognition result of a clause in the test speech after receiving the test speech. represents the time when the system under test gives the recognition result of the j-th clause in the i-th test speech input, represents the time when the j-th clause in the i-th test speech input to the system under test ends, and N represents the total number of test speech inputs to the system under test;

[0057] and / or, determining a speech recognition response time distribution based on recognition response times of all test speech inputs into the system under test;

[0058] and / or, determining a speech recognition throughput rate according to a recognition response time of all test speech input to the system under test and a duration of all test speech input to the system under test;

[0059] Determining parameters capable of characterizing the cooperative wake-up capability of the system under test according to the wake-up condition of the system under test in response to the input test voice, including:

[0060] Determine the collaborative wake-up success rate according to the number of collaborative wake-up successes and the total number of collaborative wake-up operations, wherein the collaborative wake-up success means that, when the wake-up operation is performed on multiple devices, only one device is successfully woken up, and the total number of collaborative wake-up operations is the total number of collaborative wake-up test voices input into the system under test;

[0061] And / or, the collaborative wake-up success rate is determined according to the number of correct collaborative wake-up operations and the total number of collaborative wake-up operations, wherein correct collaborative wake-up means that the wake-up operation is performed on multiple devices and the device intended by the user is correctly woken up.

[0062] Optionally, if the test item of the system under test includes the voice wake-up effect, the test scenario of the voice wake-up effect includes one or more of the following scenarios:

[0063] A high-noise scene with a signal-to-noise ratio greater than or equal to a first preset signal-to-noise ratio and less than or equal to a second preset signal-to-noise ratio;

[0064] A low-noise scene with a signal-to-noise ratio greater than the second preset signal-to-noise ratio and less than or equal to a third preset signal-to-noise ratio;

[0065] A quiet scene with a signal-to-noise ratio greater than the third preset signal-to-noise ratio;

[0066] Custom noise scenes;

[0067] The third preset signal-to-noise ratio is greater than the second preset signal-to-noise ratio, and the second preset signal-to-noise ratio is greater than the first signal-to-noise ratio.

[0068] Optionally, if the test item of the system under test includes system stability, the step of inputting the test speech in the test data set corresponding to the test item of the system under test into the system under test includes:

[0069] The test speech in the test data set corresponding to the system stability is continuously and uninterruptedly input into the system under test in an online state for processing until the test duration in the online state reaches a first test duration, and the test speech in the test data set corresponding to the system stability is continuously and uninterruptedly input into the system under test in an offline state for processing until the test duration in the offline state reaches a second test duration.

[0070] A test device for a speech recognition system, comprising: a data input module, a data acquisition module and a test result determination module;

[0071] The data input module is used to input the test speech in the test data set corresponding to the test item of the system under test into the system under test, wherein the test data set is obtained from the total test data set of the system under test according to the corresponding test item, and the total amount of the test speech in the total test data set is at least 2000 Article, the test data set includes multiple types of test speech with different qualities, namely, Class A test speech, Class B test speech, Class C test speech and Class D test speech, the category to which the test speech belongs is determined by the speech type and / or audio type of the test speech, the audio type at least includes weak noise normal and / or truncated audio, each type of test speech in the test data set at least includes speech with weak noise normal and / or truncated audio, the audio type of weak noise normal refers to speech with a speech signal-to-noise ratio greater than or equal to 20dB, the proportion of Class A test speech in the test data set should not be less than 70%, the proportion of Class B test speech should not be less than 15% and should not be greater than 20%, the proportion of Class C test speech should not be less than 5% and should not be greater than 10%, the proportion of Class D test speech should not be greater than 5%, the proportion of test speech with a duration of 3 to 5 seconds in the test data set is greater than 80%, and the number of speakers of each type of test speech should not be less than 30;

[0072] The data acquisition module is used to acquire the operating data of the system under test on the test item;

[0073] The test result determination module is used to determine the test result of the system under test on the test item according to the operation data of the system under test on the test item.

[0074] A test device for a speech recognition system, comprising: a memory and a processor;

[0075] The memory is used to store programs;

[0076] The processor is used to execute the program to implement each step of the test method of the speech recognition system described in any one of the above items.

[0077] A readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements each step of the test method of the speech recognition system described in any one of the above items.

[0078] The test method, device, equipment and storage medium of the speech recognition system provided by the present application first input the test speech in the test data set corresponding to the test item of the tested system into the tested system, then obtain the running data of the tested system on the test item, and finally determine the test result of the tested system on the test item according to the running data of the tested system on the test item. Since the test data set corresponding to the test item of the tested system is obtained from the total test data set of the tested system, and the total test data set of the tested system is based on the actual application scenario of the tested system (for example, the type of speech and audio type involved in the speech recognized by the tested system in the real scenario, the duration of the speech recognized by the tested system in the real scenario, etc.), the test result is constructed with the goal of obtaining a test result that can more accurately reflect the capability of the tested system. Therefore, based on the test data in the constructed total test data set, the tested system is tested using the test process provided by the present application, and a better test effect can be obtained. The test method of the speech recognition system provided by the present application is suitable for the design and implementation of the speech recognition system or speech recognition subsystem test of the intelligent speech interaction application by intelligent speech service providers, users and third-party testing agencies. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0080] Figure 1 A flowchart of a method for testing a speech recognition system provided in an embodiment of the present application;

[0081] Figure 2 A schematic diagram of the structure of a test device for a speech recognition system provided in an embodiment of the present application;

[0082] Figure 3 A schematic diagram of the structure of a test device for a speech recognition system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] In view of the fact that there is no method for evaluating the speech recognition system (or speech recognition subsystem) in intelligent speech interaction, the inventor of this case attempts to propose a more general and unified testing method. To this end, the inventor of this case conducted research and found that in addition to the need for a more general and unified testing process, the test data used in the test is crucial to the evaluation of the speech recognition system (or speech recognition subsystem), because it directly affects the test effect. In order to obtain better test results, that is, to obtain test results that can more realistically reflect the capabilities of the system under test, the inventor of this case proposed requirements for test data based on full consideration of the actual application scenarios of the system under test, such as requirements for the type and quantity of test data. After fully studying the functions and performance of the speech recognition system (or speech recognition subsystem), the inventor of this case also proposed general test items for the speech recognition system (or speech recognition subsystem), which can be flexibly selected by the tester according to the specific test requirements of the system under test. In addition to the above, the inventor of this case also proposed requirements for the test environment, the test equipment used in the test process, and the requirements for the test equipment. These are all to restore the actual application scenario of the system under test as much as possible, so that the test results can more truly reflect the capabilities of the system under test.

[0085] It should be noted that the method of constructing the test data set of the speech recognition system in this application (i.e., the requirements for the test data of the speech recognition system) is based on a large amount of data accumulated in the process of actual use of speech recognition systems and services in various industries, and is clearly proposed after analyzing, summarizing, and refining these data. The test data set constructed by the construction method proposed after analyzing, summarizing, and refining a large amount of data accumulated in actual application scenarios can fit the actual application scenarios of the speech recognition system. Testing the speech recognition system using such a test data set can accurately test the capabilities of the speech recognition system in actual application scenarios. Similarly, the test items proposed for the speech recognition system in this application are proposed after analyzing, summarizing, and generalizing the functional and / or performance requirements of the speech recognition systems used by various industries. The test items proposed in this way can cover all walks of life, have strong versatility, and are suitable for testing speech recognition systems in various industries.

[0086] Next, the testing method of the speech recognition system proposed in this application is introduced through the following embodiments.

[0087] See also Figure 1 , which shows a flow chart of a method for testing a speech recognition system provided by an embodiment of the present application, and may include:

[0088] Step S101: input the test speech in the test data set corresponding to the test item of the system under test into the system under test.

[0089] The system under test in this embodiment is the speech recognition system or speech recognition subsystem to be tested, and the test items of the system under test are the items that need to be tested for the speech recognition system or speech recognition subsystem.

[0090] Among them, the test items of the system under test may include: functional test items and / or performance test items. Functional test items may include one or more of the following test items: voice signal acquisition function, voice-to-text function, voice wake-up function, front-end signal processing function, speaker separation function, language information recognition function and voice recognition post-processing function; performance test items include one or more of the following test items: voice recognition effect, voice recognition efficiency, voice wake-up effect, front-end signal processing effect, speaker separation effect, language information recognition effect and system stability. It should be noted that the tester can select test items according to the functional requirements and test requirements of the system under test.

[0091] Among them, the test data set is obtained from the test data set of the system under test according to the corresponding test items. It can be understood that the quality of the test effect depends largely on the test data used. If the test data used is inappropriate (for example, the number of test data is too small, the voice type and audio type of the test data are monotonous, etc.), the test results finally obtained will not reflect the true capabilities of the system under test. In order to obtain better test results, this application considers the real application scenarios of the system under test and the speech to be recognized that may appear in the real application scenarios, with the goal of obtaining test results that can reflect the true capabilities of the system under test. From the perspectives of the number of test voices, the type of test voices, the speaker of the test voices, the duration of the test voices, etc., it is proposed to construct a test data set in which the test voices in the test data set meet the following requirements:

[0092] (1) The total number of test voices in the total test data set is greater than or equal to a preset number threshold;

[0093] (2) The test data set includes multiple types of test speech, and the category to which the test speech belongs is determined according to the speech type and / or audio type of the test speech, wherein the speech type of any test speech in the test data set is one of the following types: male Mandarin speech, female Mandarin speech, child Mandarin speech, elderly Mandarin speech, male Chinese dialect speech, female Chinese dialect speech, speech without valid speech content, speech of the language information type supported by the system under test, speech of mixed language information type, and empty audio; the audio type of any test speech in the test data set is one of the following types: normal speech without noise, normal speech with weak noise, normal speech with strong noise, loud voice, fast speech, and truncated audio;

[0094] (3) The proportion of each type of test speech in the total test data set is different, and the proportion of each type of test speech meets the corresponding requirements;

[0095] (4) The proportion of test speech in the total test data set whose duration is within the preset duration range is greater than a preset proportion threshold;

[0096] (5) The number of speakers of the test speech of each speech type in the total test data set is greater than or equal to the preset number of speakers;

[0097] (6) The content of the test speech in the total test speech set is determined according to the application scenario of the system under test and the test items of the system under test.

[0098] Preferably, the test speech in the test data set meets the requirements shown in Table 1:

[0099] Table 1 Requirements for test data in the test data set

[0100]

[0101]

[0102]

[0103] It should be noted that the contents in the above table include requirements for the total amount of test data in the total test data set, requirements for the type of test data, requirements for the number of each type of test data, requirements for the duration of the test data, requirements for the number of speakers of the test data, and requirements for the content of the test data. These requirements are all proposed on the basis of fully considering the actual application scenarios of the system under test. The starting point or goal is to make the test more in line with the real scenario, so that the test results finally obtained can more truly reflect the capabilities of the system under test. It should also be noted that each type of test speech in the total test data set (i.e., Class A test speech, Class B test speech, Class C test speech, and Class D test speech) contains at least speech with audio types of weak noise normal and / or truncated audio. The inventor of this case has found through research that weak noise normal and truncated audio are two very important audio types. The recognition effect of the system under test on speech of these two audio types can largely reflect the recognition ability of the system under test.

[0104] Optionally, the test speech in the test data set corresponding to the test item of the system under test can be obtained based on the audio sampling device and the microphone device. During the test, the above-mentioned audio sampling device and the microphone device can also be used to determine whether the test environment meets the test requirements. The parameter requirements of the audio sampling device and the microphone device are shown in the following table:

[0105] Table 2 Parameter requirements for audio sampling equipment

[0106] Device Name Parameter requirements Removable sound card Supports sampling frequencies of 44.1kHz and above, 16-bit and above analog-to-digital converters and digital-to-analog converters Recording Software The waveform sampling range is ±5000smpl~±10000smpl computer Support the installation and use of recording software Sound pressure meter Support environmental sound pressure confirmation

[0107] Table 3 Parameter requirements for microphone equipment

[0108]

[0109]

[0110] In this embodiment, the process of inputting the test voice in the test data set corresponding to the test item of the system under test into the system under test may include: using a voice playback device to play the test voice in the test data set corresponding to the test item of the system under test so that the system under test can collect it, thereby realizing the input of the test voice. Optionally, the voice playback device can be a player, or a power amplifier and an artificial mouth. During the test, in order to restore the application scenario of the system under test, while the test voice in the test data set corresponding to the test item of the system under test is played by the voice playback device, the background noise of the test voice can be played by the noise playback device, wherein the background noise can be recorded in the use scenario of the system under test, and optionally, the noise playback device may include a noise playback speaker. In order to realize the test of the system under test, some other playback devices will also be used, such as computers, simulated human bodies, etc. During the test, the above-mentioned playback device meets the following parameter requirements:

[0111] Table 4 Parameter requirements for playback equipment

[0112]

[0113]

[0114] Preferably, in order to restore the noise sound field of the actual usage scenario of the system under test, 5 noise playback speakers can be used to play background noise, and the 5 speakers are preferably 4 high-fidelity speakers and 1 subwoofer, wherein the high-fidelity speakers and the subwoofer are both active speakers, the subwoofer plays 50-120Hz frequency band, and the other speakers play 120-20kHz frequency band, the gains of the 4 high-fidelity speakers are adjusted to be consistent, and the 4 high-fidelity speakers are symmetrically placed at equal intervals around the system under test, and the distance between the high-fidelity speakers and the DUT is between 0.5 and 2m.

[0115] In addition, considering that when testing certain test items of the system under test, it may be necessary to test in multiple different test scenarios, and different test scenarios may have different requirements for the setting position of the voice playback device (such as an artificial mouth), in order to realize the testing of multiple test scenarios, the voice playback device can be set on a slide rail and the system under test can be set on a turntable, wherein the slide rail is used to adjust the distance between the voice playback device and the system under test, and the turntable is used to adjust the angle between the voice playback device and the system under test. For each test scenario, the slide rail and the turntable can be controlled by a controller to adjust the distance between the voice playback device and the system under test and the angle between the voice playback device and the system under test to the distance and angle required by the scenario.

[0116] In order to obtain more accurate test results, before the test, the background noise channel and the playback device (such as an artificial mouth) channel can be amplitude calibrated (for example, calibrated to a specified amplitude), the background noise channel and the playback device (such as an artificial mouth) channel can be channel equalized, the background noise channel can be amplitude fine-tuned (the noise amplitude played by the speaker can be adjusted to be consistent with the amplitude of the recorded noise), and the amplitude of the test speech to be played can be normalized. After completing the above processing, the test scene can be set and calibrated (including calibration of the noise channel gain in the corresponding scene, calibration of the playback device (such as an artificial mouth) channel gain, etc.).

[0117] Since the system under test is tested in a test scenario rather than in the actual application scenario of the system under test, it is understandable that in order to obtain better test results, the test scenario needs to be made as close to the actual scenario as possible (for example, the noise played back by the noise playback device in the test scenario is made to restore the noise in the real scenario as much as possible, and the voice played back by the voice playback device is made to restore the voice in the real scenario as much as possible). If the test scenario deviates greatly from the actual scenario, the obtained test results will deviate from the capabilities of the system under test in the real scenario, that is, the test results are not very meaningful. In view of this, the present application proposes the above-mentioned test equipment and parameter requirements for the test equipment used through research and repeated experiments. When testing the system under test, using test equipment that meets the above-mentioned requirements can make the test scenario of the system under test fit the actual scenario of the system under test, and then the capabilities of the system under test reflected in the test results match the capabilities of the system under test in the real scenario. It should be noted that, whether it is the requirements for test data in the total test data set or the requirements for the test environment and the test equipment used, the starting point is to make the test more in line with the actual application scenario of the system under test, so as to obtain test results that can more truly reflect the capabilities of the system under test.

[0118] Step S102: Obtaining the operating data of the system under test on the test items.

[0119] The operating data of the system under test on the test item may include the operating results of the system under test on the test item for the input test speech, and may also include data characterizing the operating status and / or resource usage of the system under test. For example, if the test item includes a speech-to-text function, the operating data of the system under test on the test item includes the operating results of the system under test on the speech-to-text function for the input test speech (the converted text of the test speech input to the system under test). For another example, if the test item includes system stability, the operating data of the system under test on the test item includes data characterizing the operating status and / or resource usage of the system under test in the process of processing the input test speech.

[0120] Step S103: Determine the test result of the system under test on the test item according to the operation data of the system under test on the test item.

[0121] The test result of the system under test on the test item can represent the test situation of the system under test on the test item, such as test passed, test failed, etc.

[0122] It should be noted that before testing the system under test, it is necessary to configure the corresponding software and hardware environment according to the functional and / or performance requirements of the system under test and the application scenario of the system under test. When testing the system under test, the test tool can be used to implement the test of the system under test, that is, the test tool can be used to input the test speech in the test data set corresponding to the test item of the system under test into the system under test, and the test tool can be used to obtain the operating data of the system under test on the test item, and the test tool can be used to analyze the operating data of the system under test on the test item to determine the test result of the system under test on the test item.

[0123] The test tools in this embodiment may include programmable test tools and test statistics tools, and may also include resource monitoring tools. It should be noted that during actual testing, the test tools used may be selected according to specific test items and test requirements.

[0124] Among them, the programmable test tool meets the following requirements: it can call the open interface of the system under test; it can customize the tool configuration file; it can receive voice data and input it into the system under test; it can perform functional testing and / or performance testing; and it can obtain the operating data of the system under test in text form.

[0125] The test statistics tool satisfies the following requirements: it can automatically collect statistics and analyze system operation results of different test items; and / or it can automatically compare system operation results with standard result comparison files.

[0126] Among them, the resource monitoring tool meets the following requirements: it can monitor the resource usage of the system under test, for example, it can monitor the system resource parameters such as memory, CPU, GPU, number of handles, etc. of the system under test.

[0127] The testing method of the speech recognition system provided in the embodiment of the present application first inputs the test speech in the test data set corresponding to the test item of the system under test into the system under test, then obtains the operating data of the system under test on the test item, and finally determines the test result of the system under test on the test item according to the operating data of the system under test on the test item. Since the test data set corresponding to the test item of the system under test is obtained from the total test data set of the system under test, and the total test data set of the system under test is based on the consideration of the real application scenario of the system under test (for example, the speech type and audio type involved in the speech recognized by the system under test in the real application scenario, the duration of the speech recognized by the system under test in the real application scenario, etc.), with the goal of obtaining a test result that can more accurately reflect the capabilities of the system under test, therefore, based on the test data in the total test data set constructed by the present application, the system under test is tested using the test process provided by the present application, and better test results can be obtained.

[0128] The embodiment of the present application provides a relatively standardized and unified test method with good test effect for the speech recognition system in intelligent voice interaction. The method provides common test items for the speech recognition system, the construction method of the total test data set, and the common test method for common test items. It also provides the requirements of the test environment, the test equipment used, and the requirements for the test equipment. Through the test method provided by the embodiment of the present application, the function and / or performance of the speech recognition system can be tested. The test method of the speech recognition system provided by the embodiment of the present application can be applicable to, but not limited to, the design and implementation of the test of the speech recognition system or speech recognition subsystem of the intelligent voice interaction application by intelligent voice service providers, users, and third-party testing agencies.

[0129] The above embodiment mentioned that the test items of the system under test may include functional test items and / or performance test items, wherein the functional test items may include one or more of speech signal acquisition function, speech-to-text function, speech wake-up function, front-end signal processing function, speaker separation function, language information recognition function and speech recognition post-processing function, and the performance test items may include one or more of speech recognition effect, speech recognition efficiency, voice wake-up effect, front-end signal processing effect, speaker separation effect, language information recognition effect and system stability. Next, the test process of each test item of the system under test is introduced respectively.

[0130] First, the testing process of the functional test items of the system under test is introduced.

[0131] (1-1) Testing of the voice signal acquisition function of the system under test

[0132] Before testing the functional test item of voice signal collection of the system under test, the system under test and the voice playback device are arranged at a specified pickup distance. The pickup distance is set according to the application scenario of the system under test and the main purpose of the system under test. In an optional implementation method, the distance intervals can be divided according to the application scenario of the system under test and the main purpose of the system under test. For example, 1m, 3m and 5m are divided into 4 distance intervals. It should be noted that according to the principle of acoustic propagation, the test environment arranged in this way is more reasonable, and the test effect is better when testing based on this test environment.

[0133] After the system under test and the playback device are arranged, the voice signal acquisition function of the system under test can be tested. The test process of the voice signal acquisition function of the system under test can include:

[0134] Step S1-1-a1: input the test speech in the test data set corresponding to the test item of the speech signal acquisition function of the system under test into the system under test.

[0135] Specifically, a voice playback device is used to play the test voice in the test data set corresponding to the test item of the voice signal acquisition function so as to facilitate acquisition by the system under test.

[0136] Step S1-1-a2: Obtain the operating data of the system under test on the test item of voice signal acquisition function.

[0137] Among them, the operating data of the system under test on the test item of speech signal acquisition function includes the operating results of the system under test on the test item of speech signal acquisition function for the input test speech (the test speech here refers to the test speech in the test data set corresponding to the test item of speech signal acquisition function). Specifically, the operating results of the system under test on the test item of speech signal acquisition function for the input test speech include: data characterizing the collection status of the input test speech by the system under test, wherein the collection status of the input test speech may be that the test speech is collected or not collected.

[0138] Step S1-1-a3: Determine the test result of the system under test on the test item of voice signal acquisition function according to the operating data of the system under test on the test item of voice signal acquisition function.

[0139] Specifically, based on the operating results of the system under test on the test item of voice signal acquisition function for the input test voice, that is, data characterizing the acquisition of the input test voice by the system under test, it is analyzed whether the system under test provides the voice signal acquisition function under the set pickup distance, and the test results of the system under test on the test item of voice signal acquisition function are generated based on the analysis results.

[0140] (1-2) Testing of the speech-to-text function of the system under test

[0141] The speech-to-text function refers to the function of converting the input speech into text and outputting it, thereby obtaining the text corresponding to the input speech. The test process of the speech-to-text function of the system under test may include:

[0142] Step S1-2-b1: input the test speech in the test data set corresponding to the test item of the speech-to-text function of the system under test into the system under test.

[0143] Step S1-2-b2: Obtain the operating data of the system under test on the test item of speech-to-text function.

[0144] Among them, the operating data of the system under test on the test item of speech-to-text function is the operating result of the system under test on the test item of speech-to-text function for the input test speech. Specifically, the operating results of the system under test on the test item of speech-to-text function for the input test speech (the test speech here refers to the test speech in the test data set corresponding to the test item of speech-to-text function) include the converted text of the test speech input to the system under test, that is, the text obtained after the system under test converts the input test speech.

[0145] Step S1-1-b3: Determine the test result of the system under test on the test item of speech-to-text function according to the operating data of the system under test on the test item of speech-to-text function.

[0146] Specifically, the system under test analyzes the operating results of the input test voice on the test item of speech-to-text function, that is, the converted text of the test voice input to the system under test, to see whether the system under test provides the function of converting the received voice into text consistent with the content of the received voice and outputting it, and generates the test results of the system under test on the test item of speech-to-text function based on the analysis results.

[0147] (1-3) Testing of the voice wake-up function of the system under test

[0148] The test process of the voice wake-up function of the system under test may include:

[0149] Step S1-3-c1: input the test voice in the test data set corresponding to the test item of the voice wake-up function of the system under test into the system under test.

[0150] Among them, the test data set corresponding to the voice wake-up function test item includes: test voices with wake-up words as the content and test voices with non-wake-up words as the content.

[0151] The voice wake-up function in this embodiment may include but is not limited to one or more of the following functions: a predefined wake-up word wake-up function, a custom wake-up word wake-up function, a multiple wake-up word wake-up function, a multiple audio stream monitoring function, a voice interruption wake-up function, and a collaborative wake-up function.

[0152] It should be noted that the predefined wake-up word wake-up function is a wake-up function based on a predefined wake-up word, that is, the function of waking up the system under test with a predefined wake-up word; the custom wake-up word wake-up function is a wake-up function based on a custom wake-up word, that is, the function of waking up the system under test with a custom wake-up word; the multiple wake-up word wake-up function is a wake-up function based on multiple wake-up words, that is, the function of using multiple different wake-up words to wake up the system under test; the multiple audio stream monitoring function is a function of monitoring multiple audio streams while executing voice wake-up; the voice interruption wake-up function is a function of waking up based on a voice interruption method, that is, the function of waking up the system under test by voice interruption; the collaborative wake-up function is a function in which, when multiple devices using the same wake-up word appear in the same scene, only one device responds to a wake-up operation.

[0153] In view that the voice wake-up function may include the wake-up functions listed above, the test voice with the content of the wake-up word in the test data set corresponding to the voice wake-up function test item may include: the test voice with the content of the predefined wake-up word, and / or the test voice with the content of the custom wake-up word, and / or the test voice with the content of multiple wake-up words (for example, the test voice with the content of the wake-up word A, the test voice with the content of the wake-up word B, and the test voice with the content of the wake-up word C), and / or the test voice with the content of the voice interruption wake-up word. It should be noted that the content of the test voice in the test data set corresponding to the voice wake-up function can be determined according to the specific voice wake-up function test item.

[0154] Step S1-3-c2: Obtain the operating data of the system under test on the test item of the voice wake-up function.

[0155] Among them, the operating data of the system under test on the test item of voice wake-up function is the operating result of the system under test on the test item of voice wake-up function for the input test voice (the test voice here is the test voice in the test data set corresponding to the test item of voice wake-up function). Specifically, the operating result of the system under test on the test item of voice wake-up function for the input test voice may include data characterizing the wake-up situation of the system under test on the input test voice.

[0156] Step S1-3-c3: Determine the test result of the system under test on the test item of the voice wake-up function according to the operating data of the system under test on the test item of the voice wake-up function.

[0157] Specifically, based on the running results of the system under test on the test item of voice wake-up function for the input test voice, that is, data characterizing the wake-up situation of the system under test for the input test voice, analyze whether the system under test provides a voice wake-up function (for example, analyze whether the system under test provides a predefined wake-up word wake-up function, and / or a custom wake-up word wake-up function, and / or a multiple wake-up word wake-up function, and / or a voice interruption wake-up function, and / or a collaborative wake-up function, etc.), and generate a test result corresponding to the voice wake-up function based on the analysis result.

[0158] (1-4) Testing of the front-end signal processing function of the system under test

[0159] The testing process of the front-end signal processing function of the system under test can include:

[0160] Step S1-4-d1: Input the test speech in the test data set corresponding to the test item of the front-end signal processing function of the system under test into the system under test.

[0161] Among them, the front-end signal processing function may include but is not limited to one or more of the following functions: speech encoding and decoding function, speech endpoint detection function, speech enhancement function, sound source localization function, audio format conversion function, speech resampling function and audio quality judgment function. When testing any of the functions listed above, the test speech in the test data set corresponding to the function can be input into the system under test. For example, when testing the speech endpoint detection function of the system under test, the test speech in the test data set corresponding to the speech endpoint detection function can be input into the system under test.

[0162] It should be noted that the speech coding and decoding function is that the system under test can set the compression level of the speech coding algorithm, support the compression and decompression of speech audio under different audio coding formats, and ensure that the speech content remains unchanged; the endpoint detection function is that the system under test can detect the starting point and end point of the first or more speech segments from the continuous audio stream, and can set the endpoint detection sensitivity, that is, the function of setting the speech waiting timeout length and the tail silence length; the speech enhancement function is that the system under test can automatically improve the signal-to-noise ratio of the input speech, and can suppress the background noise and late reverberation in the input speech; the sound source localization function is , the system under test can automatically locate the spatial position of the sound source; the audio format conversion function is that the system under test can automatically convert the format of the input audio and ensure that the voice content remains unchanged; the resampling function is that the system under test can change the sampling rate of the digital voice signal and ensure that the voice content remains unchanged; the audio quality judgment function is that the system under test can automatically judge the quality of the input audio (for example, the audio with too low volume, too low signal-to-noise ratio, or front or back truncation is judged as poor audio quality); the acoustic echo cancellation function refers to the function that the system under test can cancel the echo of the input audio.

[0163] In addition, it should be noted that if the audio quality judgment function of the system under test needs to be tested, the corresponding test data set may include front-truncated audio, rear-truncated audio, low-volume audio, low-signal-to-noise ratio audio, etc.

[0164] Step S1-4-d2: Obtain the operating data of the system under test on the test item of the front-end signal processing function.

[0165] Among them, the operating data of the system under test on the test item of front-end signal processing function is the operating result of the system under test on the test item of front-end signal processing function for the input test voice (the input test voice here is the test voice in the test data set corresponding to the test item of front-end signal processing function). Specifically, the operating result of the system under test on the test item of front-end signal processing function for the input test voice includes the front-end processing result of the test voice input to the system under test, that is, the result of the front-end processing of the input test voice by the system under test.

[0166] Step S1-4-d3: Determine the test result of the system under test on the test item of the front-end signal processing function based on the operating data of the system under test on the test item of the front-end signal processing function.

[0167] Specifically, the system under test analyzes whether it provides front-end signal processing functions (for example, voice encoding and decoding functions, and / or voice endpoint detection functions, and / or voice enhancement functions, and / or sound source localization functions, and / or audio format conversion functions, and / or voice resampling functions, and / or audio quality judgment functions, etc.) based on the operating results of the input test voice on the test item of the front-end signal processing function, that is, the front-end processing results of the test voice input to the system under test, and generates test results of the system under test on the test item of the front-end signal processing function based on the analysis results.

[0168] (1-5) Test of speaker separation function of the system under test

[0169] Speaker separation refers to the process of speaker segmentation and speaker clustering for multiple speakers in an audio stream containing valid speech signals. It should be noted that the purpose of speaker separation is generally to classify and track multiple speakers in a space.

[0170] The testing process of the speaker separation function of the system under test may include:

[0171] Step S1-5-e1: Input the test speech in the test data set corresponding to the test item of the speaker separation function of the system under test into the system under test.

[0172] The test data set corresponding to the speaker separation function includes: test speech of multiple speakers' alternating conversations. Preferably, the duration of the multiple speakers' alternating conversations is 20 minutes.

[0173] The speaker separation function in this embodiment may include a speaker segmentation function and / or a speaker clustering function. The speaker segmentation function is a function of segmenting the input speech into speech segments containing only the speech content of one speaker, and the speaker clustering function is a function of clustering speech segments corresponding to the same speaker to obtain speech segments corresponding to different speakers.

[0174] Step S1-5-e2: Obtain the operating data of the system under test on the test item of speaker separation function.

[0175] Among them, the operating data of the system under test on the test item of speaker separation function is the operating result of the system under test on the test item of speaker separation function for the input test speech (the test speech in the test data set corresponding to the test item of speaker separation function). Specifically, the operating results of the system under test on the test item of speaker separation function for the input test speech include: the speaker separation result of the test speech input to the system under test, that is, the result obtained by the system under test performing speaker separation on the input test speech.

[0176] Step S1-5-e3: Determine the test result of the system under test on the test item of speaker separation function according to the operating data of the system under test on the test item of speaker separation function.

[0177] Specifically, based on the running result of the system under test on the test item of speaker separation function for the input test speech, that is, the speaker separation result of the test speech input to the system under test, analyze whether the system under test provides a speaker separation function (for example, speaker segmentation function, speaker clustering function), and generate the test result of the system under test on the test item of speaker separation function based on the analysis result.

[0178] (1-6) Testing of the language information recognition function of the system under test

[0179] The testing process of the language information recognition capability of the system under test may include:

[0180] Step S1-6-f1: Input the test speech in the test data set corresponding to the test item of the language information recognition function of the system under test into the system under test.

[0181] The test data set corresponding to the test item of language information recognition function may include: test speech containing one or more language information. The language information may be a language or a Chinese dialect. It should be noted that a Chinese dialect is a Chinese language that is different from Mandarin and is only used in a specific area.

[0182] The front-end signal processing function in this embodiment may include but is not limited to one or more of the following functions: one language or multiple language recognition function, multiple language mixed reading recognition function, one Chinese dialect or multiple Chinese dialect recognition function, multiple Chinese dialect mixed reading recognition function, language information endpoint recognition function, and generate test results corresponding to the language information recognition function based on the analysis results.

[0183] It should be noted that the one or more language recognition function is a function of judging one or more (2 or more) languages ​​to which the input speech belongs, and outputting the recognition results of one or more languages; the multi-language mixed reading recognition function is a function of judging different languages ​​and outputting the recognition results of multi-language mixed reading in the case of mixed reading of multiple languages; the one or more Chinese dialect recognition function is a function of judging one or more (2 or more) Chinese dialects to which the input speech belongs, and outputting the recognition results of one or more Chinese dialects; the multi-Chinese dialect mixed reading recognition function is a function of judging different Chinese dialects and outputting the recognition results of multi-Chinese dialect mixed reading in the case of mixed reading of multiple Chinese dialects; the language information endpoint recognition function is a function of judging the endpoints of speech segments of different languages ​​or speech segments of different Chinese dialects, and outputting the endpoint recognition results.

[0184] In view that the language information recognition function may include the functions listed above, the test data set corresponding to the test item of the language information recognition function may include: test speech in one language or multiple languages, and / or, test speech in one Chinese dialect or multiple Chinese dialects, and / or, test speech in a mixture of multiple languages, and / or, test speech in a mixture of multiple Chinese dialects. It should be noted that the language information of the test speech in the test data set corresponding to the language information recognition function can be determined according to the specific test item of the language information recognition function.

[0185] Step S1-6-f2: Obtain the operating data of the system under test on the test item of language information recognition function.

[0186] Among them, the operating data of the system under test on the test item of language information recognition function is the operating result of the system under test on the test item of language information recognition function for the input test speech (the test speech in the test data set corresponding to the test item of language information recognition function). Furthermore, the operating results of the system under test on the test item of language information recognition function for the input test speech include: the recognition result of the test speech input to the system under test, that is, the result obtained by the system under test recognizing the input test speech.

[0187] Step S1-6-f3: Determine the test result of the system under test on the test item of language information recognition function based on the operating data of the system under test on the test item of language information recognition function.

[0188] Specifically, based on the running result of the system under test on the test item of language information recognition function for the input test speech, that is, the recognition result of the test speech input to the system under test, it is analyzed whether the system under test provides a language information recognition function (for example, one language or multiple languages ​​recognition function, and / or multiple languages ​​mixed reading recognition function, and / or one Chinese dialect or multiple Chinese dialects recognition function, and / or multiple Chinese dialects mixed reading recognition function, and / or language information endpoint recognition function), and the test result of the system under test on the test item of language information recognition function is generated according to the analysis result.

[0189] (1-7) Testing of speech recognition post-processing function of the system under test

[0190] The speech recognition post-processing function of the system under test refers to the function of post-processing the recognition result of the input speech. The test process of the speech recognition post-processing function of the system under test may include:

[0191] Step S1-7-g1: Input the test speech in the test data set corresponding to the test item of the speech recognition post-processing function of the system under test into the system under test.

[0192] The post-processing function of speech recognition in this embodiment may, but is not limited to, include one or more of the following functions: character regularization function, punctuation prediction function, text smoothing function, and text replacement function.

[0193] It should be noted that the character regularization function is a function of regularizing the characters in the speech recognition result, which may, but is not limited to, include one or more of the digital regularization function, the letter regularization function, and the traditional and simplified Chinese character regularization function. Among them, the digital regularization function refers to regularizing the numbers represented by Chinese characters into a format that conforms to the reading habit as required. For example, the result of digital regularization of "one thousand two hundred and thirty-two yuan" is "1232 yuan". The letter regularization function refers to regularizing the recognition result into a format that is easier to read as required. For example, the result of letter regularization of "three W points" is "www.". The traditional and simplified Chinese character regularization function refers to regularizing the recognition result into traditional or simplified Chinese as required. For example, the result of traditional and simplified Chinese character regularization of "one, two, three, four, five" is "壹, 贰, 叁, 肆, 伍". The punctuation prediction function is a function of predicting and adding punctuation to the speech recognition result according to the speech content. For example, the result of punctuation prediction for the speech recognition result "Hello, where are you?" is "Hello, where are you?". The text smoothing function is a function of filtering out meaningless text units (such as characters and words) in the speech recognition result according to the speech content. For example, the result of text smoothing for "um... um... I'm not sure" is "I'm not sure". The text replacement function is a function of dynamically replacing or modifying text units in the speech recognition result according to the speech content. For example, the result of text replacement for the speech recognition result "They made great sacrifices" is "They made great sacrifices".

[0194] Step S1-7-g2: Obtain the operation data of the system under test on the test item of the post-processing function of speech recognition.

[0195] Among them, the operation data of the system under test on the test item of the post-processing function of speech recognition is the operation result of the system under test on the test item of the post-processing function of speech recognition for the input test speech. Specifically, the operation result of the system under test on the test item of the post-processing function of speech recognition for the input test speech includes the post-processing result of the speech recognition result of the test speech input to the system under test.

[0196] Step S1-7-g3: Determine the test result of the system under test on the test item of the post-processing function of speech recognition according to the operation data of the system under test on the test item of the post-processing function of speech recognition.

[0197] Specifically, the running result of the system under test on the test item of speech recognition post-processing function for the input test speech, that is, the post-processing result of the speech recognition result of the test speech input to the system under test, analyzes whether the system under test provides speech recognition post-processing function (for example, character regularization function, punctuation prediction function, text smoothing function, text replacement function), and generates the test result of the system under test on the test item of speech recognition post-processing function based on the analysis result.

[0198] It should be noted that when testing any of the above-mentioned functional test items, the test data in the test data set corresponding to the functional test item can be input into the system under test based on the programmable test tool mentioned above, and the operating data of the system under test in the functional test item can be obtained based on the programmable test tool, and then the operating data of the system under test in the functional test item can be analyzed based on the test statistics tool. Among them, the test data set corresponding to any functional test item is obtained from the total test data set prepared according to the requirements shown in Table 1.

[0199] Next, the testing process of the performance test items of the system under test is introduced.

[0200] (2-1) Testing of the speech recognition effect of the system under test

[0201] The testing process of the speech recognition effect of the system under test may include:

[0202] Step S2-1-a1: input the test speech in the test data set corresponding to the test item of speech recognition effect of the system under test into the system under test.

[0203] Step S2-1-a2: Obtain the operating data of the system under test on the test item of speech recognition effect.

[0204] Among them, the operating data of the system under test on the test item of speech recognition effect is the operating result of the system under test on the test item of speech recognition effect for the input test speech. Furthermore, the operating results of the system under test on the test item of speech recognition effect for the input test speech include: the speech recognition result of the test speech input into the system under test.

[0205] Step S2-1-a3: Determine the test result of the system under test on the test item of speech recognition effect according to the operating data of the system under test on the test item of speech recognition effect.

[0206] Specifically, based on the operating data of the system under test on the test item of speech recognition effect, the performance characterization parameters of the system under test on the test item of speech recognition effect are determined, and based on the performance characterization parameters of the system under test on the test item of speech recognition effect, the test results of the system under test on the test item of speech recognition effect are generated.

[0207] Among them, the process of determining the performance characterization parameters of the system under test on the test item of speech recognition effect based on the operating data of the system under test on the test item of speech recognition effect can include: determining the performance characterization parameters of the system under test on the test item of speech recognition effect based on the operating results of the system under test on the speech recognition effect for the input test speech (that is, the speech recognition result of the test speech input to the system under test) and the speech content annotated text of the test speech input to the system under test.

[0208] The performance characterization parameters of the tested system in the test item of speech recognition effect may include: parameters that can characterize the word recognition ability of the tested system and / or parameters that can characterize the sentence recognition ability of the tested system.

[0209] Furthermore, the parameters that can characterize the word recognition ability of the system under test may include one or more of the following parameters: word matching rate, substitution error rate, insertion error rate, deletion error rate, word error rate, and word accuracy rate. These parameters can be calculated with reference to the method described in GB / T 21023 (General Technical Specification for Chinese Speech Recognition Systems); the parameters that can characterize the sentence recognition ability of the system under test may include but are not limited to sentence recognition accuracy rate. The sentence recognition accuracy rate is determined based on the total number of correctly recognized sentences in the recognition results of the test speech input to the system under test and the total number of annotated sentences of the test speech input to the system under test. Specifically, the total number of correctly recognized sentences in the recognition results of the test speech input to the system under test can be divided by the total number of annotated sentences of the test speech input to the system under test, and the result obtained is the sentence recognition accuracy rate.

[0210] After obtaining the performance characterization parameters of the tested system on the test item of speech recognition effect, the test results of the tested system on the test item of speech recognition effect can be generated based on the obtained parameters. Optionally, it can be determined whether the performance characterization parameters of the tested system on the test item of speech recognition effect meet the technical requirements of the tested system or relevant standards and specifications. If so, it is determined that the test item of speech recognition effect of the tested system has passed. Otherwise, it is determined that the test item of speech recognition effect of the tested system has failed. The test results of the tested system on the test item of speech recognition effect are generated based on the analysis results. Among them, the test results corresponding to the test item of speech recognition effect of the tested system can include the test results indicating whether the test item of speech recognition effect has passed. Optionally, it can also include the name of the corresponding test data set, the number of test voices in the corresponding test data set and other related test information.

[0211] (2-2) Testing of the speech recognition efficiency of the system under test

[0212] The testing process of the speech recognition efficiency of the system under test may include:

[0213] Step S2-2-b1: Input the test speech in the test data set corresponding to the test item of speech recognition efficiency of the system under test into the system under test.

[0214] Step S2-2-b2: Obtain the operating data of the system under test on the test item of speech recognition efficiency.

[0215] Among them, the operating data of the tested system on the test item of speech recognition efficiency is the operating result of the tested system on the test item of speech recognition efficiency for the input test speech. Further, the operating result of the tested system on the test item of speech recognition efficiency for the input test speech includes: the recognition response time of the tested system to the input test speech. It should be noted that the recognition response time of the tested system to an input test speech is the time when the tested system gives the recognition result of the test speech after receiving the test speech.

[0216] Step S2-2-b3: Determine the test result corresponding to the test item of speech recognition efficiency of the system under test based on the operating data of the system under test on the test item of speech recognition efficiency.

[0217] Specifically, based on the operating data of the system under test on the test item of speech recognition efficiency, that is, the recognition response time of the system under test to the input test speech, determine the performance characterization parameters of the system under test on the test item of speech recognition efficiency; based on the performance characterization parameters of the system under test on the test item of speech recognition efficiency, generate the test results of the system under test on the test item of speech recognition efficiency.

[0218] Among them, the performance characterization parameters of the system under test on the test item of speech recognition efficiency may include one or more of the following parameters: average speech recognition response time, average sentence response time of speech recognition, speech recognition response time distribution, and speech recognition throughput.

[0219] The average response time of speech recognition is determined according to the recognition response time of all test speech input to the system under test and the total number of all test speech input to the system under test, wherein the recognition response time of a test speech is determined according to the moment when the system under test gives the recognition result of the test speech and the moment when the input of the test speech ends. Specifically, as shown in the following formula (1), the ratio of the sum of the recognition response time of all test speech input to the system under test to the total number of all test speech input to the system under test can be calculated as the average response time of speech recognition:

[0220]

[0221] In formula (1), It indicates the time when the system under test gives the recognition result of the i-th test speech. represents the time when the i-th test speech input to the system under test ends, N represents the total number of test speech inputs to the system under test, T avr Indicates the average speech recognition response time.

[0222] The average clause response time of speech recognition is determined based on the clause recognition response time of all test speech inputs to the system under test and the total number of all test speech inputs to the system under test. It should be noted that the clause recognition response time of a test speech is the time when the system under test gives a recognition result of a clause in the test speech after receiving the test speech. Specifically, the clause recognition response time of a test speech is determined based on the moment when the system under test gives the recognition result of the clause and the moment when the input of the clause ends. Specifically, as shown in the following formula (2), the ratio of the sum of the clause recognition response times of all test speech inputs to the system under test to the total number of all test speech inputs to the system under test can be calculated as the average clause response time of speech recognition:

[0223]

[0224] In formula (2), It indicates the time when the system under test gives the recognition result of the jth clause in the i-th test speech. represents the time when the jth clause in the i-th test speech input to the system under test ends, N represents the total number of test speech inputs to the system under test, T avsr It should be noted that the average clause response time of speech recognition calculated by formula (2) can more truly reflect the recognition response of the tested system to the input speech.

[0225] It should be noted that the average response time of speech recognition reflects the recognition response of the system under test to the entire test speech. It is understandable that a test speech may not only contain one sentence, but may contain multiple sentences. From this point of view, the average response time of speech recognition can only roughly characterize the speech recognition efficiency of the system under test. In order to more accurately characterize the speech recognition efficiency of the system under test, this application introduces the average clause response time of speech recognition that can reflect the recognition response of the system under test to the clauses in the test speech. Compared with the average response time of speech recognition, the average clause response time of speech recognition can reflect the recognition response of the system under test for speech segments with smaller granularity (i.e., clauses). For the system under test with higher real-time requirements, the average clause response time of speech recognition can better characterize the speech recognition efficiency of the system under test.

[0226] Among them, the speech recognition response time distribution is determined according to the recognition response time of all test speech input to the system under test. Optionally, multiple response time distribution intervals can be preset to determine the distribution of the recognition response time of all test speech input to the system under test in the preset multiple response time distribution intervals. For example, three response time distribution intervals are preset, namely interval A, interval B and interval C. It can be determined how many test speech with recognition response time is in interval A, how many test speech with recognition response time is in interval B, and how many test speech with recognition response time is in interval C, so as to obtain the recognition response time distribution of all test speech input to the system under test. In addition, the proportion of the number of test speech with recognition response time in each response time distribution interval in all test speech input to the system under test can also be counted.

[0227] Among them, the speech recognition throughput describes the efficiency of inputting a recorded (long) speech at one time and giving the overall speech recognition result at one time, that is, the length of time the system under test recognizes the speech audio within a unit response time. The speech recognition throughput is determined based on the recognition response time of all test speech input to the system under test and the duration of all test speech input to the system under test. Specifically, the sum of the duration of all test speech input to the system under test can be divided by the sum of the recognition response time of all test speech input to the system under test, and the result obtained can be used as the speech recognition throughput.

[0228] It should be noted that if the average response time of sound recognition, the average clause response time of speech recognition, and the distribution of speech recognition response time of the system under test are to be obtained, it is preferred that the test data set corresponding to the test item of speech recognition efficiency contain a test speech with a duration of 10s±0.1s and no silence after the end of the speech; if the speech recognition throughput of the system under test is to be obtained, it is preferred that the duration of the test speech in the test data set corresponding to the test item of speech recognition efficiency be greater than 10 hours.

[0229] After obtaining the performance characterization parameters of the tested system on the test item of speech recognition efficiency, the test results of the tested system on the test item of speech recognition efficiency can be generated based on the obtained parameters. Specifically, it can be determined whether the performance characterization parameters of the tested system on the test item of speech recognition efficiency meet the technical requirements of the tested system or relevant standards and specifications. If so, it is determined that the test item of speech recognition efficiency of the tested system has passed. Otherwise, it is determined that the test item of speech recognition efficiency of the tested system has failed. The test results of the tested system on the test item of speech recognition efficiency are generated based on the analysis and judgment results. Among them, the test results of the tested system on the test item of speech recognition efficiency can include the test results indicating whether the test item of speech recognition efficiency has passed. Optionally, it can also include the name of the corresponding test data set, the number of test voices in the corresponding test data set, and other related test information.

[0230] (2-3) Test of voice wake-up effect of the system under test

[0231] The test process of the voice wake-up effect of the system under test may include:

[0232] Step S2-3-c1: input the test speech in the test data set corresponding to the test item of the voice wake-up effect of the system under test into the system under test.

[0233] Among them, the test data set corresponding to the test item of voice wake-up effect may include: test voices with wake-up words as the content and test voices with non-wake-up words as the content, wherein the test voices with wake-up words as the content may include test voices with predefined wake-up words as the content, and / or test voices with custom wake-up words as the content, and / or test voices with non-wake-up words as the content, and / or test voices with different wake-up words as the content (for example, test voices with wake-up words A as the content, test voices with wake-up words B as the content, and test voices with wake-up words C as the content), and / or test voices with voice interruption wake-up words as the content. Preferably, the number of test voices in the test data set corresponding to the test item of voice wake-up effect is not less than 200.

[0234] Step S2-3-c2: Obtain the operating data of the system under test on the test item of voice wake-up effect.

[0235] Among them, the operating data of the system under test on the test item of voice wake-up effect is the operating result of the system under test on the test item of voice recognition efficiency for the input test voice. Specifically, the operating results of the system under test on the test item of voice recognition efficiency for the input test voice include: the wake-up status of the system under test on the input test voice.

[0236] Step S2-3-c3: Determine the test result corresponding to the test item of voice wake-up effect of the system under test based on the operating data of the system under test on the test item of voice wake-up effect.

[0237] Specifically, based on the operating data of the system under test on the test item of voice wake-up effect, that is, the wake-up situation of the system under test in response to the input test voice, the performance characterization parameters of the system under test on the test item of voice wake-up effect are determined; based on the performance characterization parameters of the system under test on the test item of voice wake-up effect, the test results of the system under test on the test item of speech recognition efficiency are generated.

[0238] In this embodiment, the performance characterization parameters of the system under test on the test item of voice wake-up effect may include but are not limited to one or more of the following parameters: the wake-up rate characterizing the voice wake-up capability of the system under test, the false wake-up frequency characterizing the voice wake-up capability of the system under test, the voice interruption success rate characterizing the voice interruption capability of the system under test, the voice interruption wake-up rate characterizing the voice interruption wake-up capability of the system under test, the voice interruption false wake-up frequency characterizing the voice interruption wake-up capability of the system under test, the collaborative wake-up success rate characterizing the voice wake-up capability of multiple devices in the same scene, and the collaborative wake-up accuracy rate characterizing the voice wake-up capability of multiple devices in the same scene.

[0239] The wake-up rate is determined based on the number of successful wake-ups and the total number of voice wake-up operations. Specifically, the number of successful wake-ups within a certain period of time can be divided by the total number of voice wake-up operations, and the result obtained is the wake-up rate. It should be noted that the total number of voice wake-up operations is the total number of voice wake-up test voices input into the system under test, that is, each time a voice wake-up test voice is input into the system under test, a voice wake-up operation is performed.

[0240] Among them, the false wake-up frequency is determined according to the number of false wake-ups and the total duration of the voice wake-up test. Specifically, the number of false wake-ups in a certain period of time can be divided by the total duration of the voice wake-up test, and the result obtained is used as the false wake-up frequency. It can be seen that the false wake-up frequency is the number of times the system under test is falsely awakened per unit time.

[0241] The voice interruption success rate is determined based on the number of successful voice interruptions and the total number of voice interruption operations. Specifically, the number of successful voice interruptions can be divided by the total number of voice interruption operations, and the result is the voice interruption success rate, that is, the ratio of the system under test to successful voice interruption. It should be noted that the total number of voice interruption operations is the total number of voice interruption test voices input into the system under test.

[0242] The voice interruption wake-up rate is determined based on the number of successful interruption wake-ups and the total number of voice interruption wake-up operations. Specifically, the number of successful interruption wake-ups within a certain period of time can be divided by the total number of voice interruption wake-up operations, and the result obtained is the voice interruption wake-up rate. It should be noted that the total number of voice interruption wake-up operations is the total number of voice interruption wake-up test voices input into the system under test.

[0243] The frequency of false wake-up due to voice interruption is determined according to the number of false wake-up due to voice interruption and the total duration of the voice interruption wake-up test. Specifically, the number of false wake-up due to voice interruption is divided by the total duration of the voice interruption wake-up test, and the result is taken as the frequency of false wake-up due to voice interruption.

[0244] The collaborative wake-up success rate is determined based on the number of collaborative wake-up successes and the total number of collaborative wake-up operations. Specifically, the number of collaborative wake-up successes can be divided by the total number of collaborative wake-up operations, and the result is used as the collaborative wake-up success rate. It should be noted that collaborative wake-up success means that when a wake-up operation is performed on multiple devices, only one device is successfully awakened, and the total number of collaborative wake-up operations is the total number of collaborative wake-up test voices input into the system under test.

[0245] The collaborative wake-up accuracy rate is determined based on the number of collaborative wake-up accuracy times and the total number of collaborative wake-up operations. Specifically, the number of collaborative wake-up accuracy times can be divided by the total number of collaborative wake-up operations to obtain the result as the collaborative wake-up accuracy rate. It should be noted that collaborative wake-up accuracy means that when wake-up operations are performed on multiple devices, the device intended by the user is correctly woken up.

[0246] After obtaining the performance characterization parameters of the system under test on the test item of voice wake-up effect, the test results of the system under test on the test item of voice wake-up effect can be generated based on the obtained parameters. Specifically, it can be determined whether the performance characterization parameters of the system under test on the test item of voice wake-up effect meet the technical requirements of the system under test or relevant standards and specifications. If so, it is determined that the test item of voice wake-up effect of the system under test has passed. Otherwise, it is determined that the test item of voice wake-up effect of the system under test has failed. The test results of the system under test on the test item of voice wake-up effect are generated based on the analysis and judgment results. Among them, the test results of the system under test on the test item of voice wake-up effect may include a test result indicating whether the test item of voice wake-up effect has passed, and optionally, it may also include relevant test information such as the name of the corresponding test data set and the number of test voices in the corresponding test data set.

[0247] It should be noted that when testing the voice wake-up effect test item, the test can be performed in one or more of the following four scenarios: high noise scene, low noise scene, quiet scene, and custom noise scene.

[0248] The high noise scene is a scene where the signal-to-noise ratio is greater than or equal to the first preset signal-to-noise ratio and less than or equal to the second preset signal-to-noise ratio. The low noise scene is a scene where the signal-to-noise ratio is greater than the second preset signal-to-noise ratio and less than or equal to the third preset signal-to-noise ratio. The quiet scene is a scene where the signal-to-noise ratio is greater than the third preset signal-to-noise ratio. The custom noise scene is implemented in accordance with GB / T 36464. Among them, the third preset signal-to-noise ratio is greater than the second preset signal-to-noise ratio, and the second preset signal-to-noise ratio is greater than the first signal-to-noise ratio.

[0249] The following table shows an example of the four scenarios described above:

[0250] Table 5 Four noise scenarios of the tested system

[0251]

[0252] Considering that in real scenarios, there may be noise-free situations or noisy situations, and the noisy situations may be divided into multiple types, such as low-noise situations, high-noise situations, etc., if the system under test is tested only in a single scenario (such as a quiet scene), the test results obtained cannot reflect the true capabilities of the system under test. In order to ultimately obtain more comprehensive test results that truly reflect the voice wake-up effect of the system under test, the system under test can be tested in the above four scenarios.

[0253] (2-4) Testing of the front-end signal processing effect of the system under test

[0254] The testing process of the front-end signal processing effect of the system under test can include:

[0255] Step S2-4-d1: Input the test speech in the test data set corresponding to the test item of the front-end signal processing effect of the system under test into the system under test.

[0256] Step S2-4-d2: Obtain the operating data of the system under test on the test item of front-end signal processing effect.

[0257] Among them, the operating data of the system under test on the test item of front-end signal processing effect is the operating result of the system under test on the test item of front-end signal processing effect for the input test speech. Furthermore, the operating results of the system under test on the test item of speech recognition efficiency for the input test speech include: the front-end processing result of the test speech input to the system under test.

[0258] Step S2-4-d3: Determine the test result corresponding to the test item of the front-end signal processing effect of the system under test based on the operating data of the system under test on the test item of the front-end signal processing effect.

[0259] Specifically, based on the operating data of the system under test on the test item of front-end signal processing effect, that is, the front-end processing result of the test speech input into the system under test, the performance characterization parameters of the system under test on the test item of front-end signal processing effect are determined, and based on the performance characterization parameters of the system under test on the test item of front-end signal processing effect, the test results of the system under test on the test item of front-end signal processing effect are generated.

[0260] In this embodiment, the performance characterization parameters of the system under test on the test item of front-end signal processing effect may include but are not limited to one or more of the following parameters: the speech coding and decoding compression rate characterizing the speech coding and decoding capability of the system under test, the endpoint detection accuracy characterizing the endpoint detection capability of the system under test, the signal-to-noise ratio improvement rate and / or noise suppression amount characterizing the speech enhancement capability of the system under test, the sound source localization accuracy characterizing the sound source localization capability of the system under test, and the echo return attenuation gain characterizing the echo cancellation capability of the system under test.

[0261] Among them, the speech encoding and decoding compression rate can be determined according to the audio bit stream rate before and after the test speech is encoded input into the system under test. Specifically, for the test speech input into the system under test, the audio bit stream rate before speech encoding can be divided by the audio bit stream rate after speech encoding, and the result obtained is the speech encoding and decoding compression rate.

[0262] The endpoint detection accuracy can be determined based on the endpoint detection result of the tested system on the input test speech. Specifically, the endpoint detection accuracy can be calculated based on the calculation method shown in the following formula (3):

[0263]

[0264] In formula (3), T valid Indicates the effective voice audio duration, T silence Indicates the duration of silent audio, T dev To detect the error audio duration, A VAD is the endpoint detection accuracy. The endpoint detection accuracy calculated by formula (3) can more truly reflect the endpoint detection capability of the system under test.

[0265] It should be noted that the test data set corresponding to the endpoint detection accuracy test item includes a test speech of no less than 3 seconds.

[0266] The signal-to-noise ratio improvement rate can be determined based on the signal-to-noise ratio of the test speech input to the system under test before and after enhancement. Specifically, the signal-to-noise ratio of the test speech input to the system under test can be divided by the signal-to-noise ratio of the enhanced speech obtained by performing speech enhancement on the test speech by the system under test, and the result obtained is used as the signal-to-noise ratio improvement rate.

[0267] The noise suppression amount can be determined according to the noise amplitude before and after the test speech enhancement input to the system under test. Specifically, the noise suppression amount can be calculated according to the calculation method shown in the following formula (4):

[0268]

[0269] In formula (4), v in (n) represents the amplitude of the nth noise signal in the speech signal input to the system under test, v out(n) is the amplitude of the nth noise signal in the output signal of the system under test (i.e., the signal after speech enhancement of the speech signal input to the system under test), N represents the total number of spectral frequency components of the speech signal input to the system under test, and D NR The noise suppression amount calculated by formula (4) can truly reflect the speech enhancement capability of the system under test.

[0270] Among them, the sound source localization accuracy can be determined based on the number of correct sound source localizations of the input test speech by the system under test and the total number of sound source localization requests. Specifically, the number of correct sound source localizations can be divided by the total number of sound source localization requests, and the result obtained is the sound source localization accuracy.

[0271] The echo return attenuation gain may be determined according to the expected echo signal and the error signal corresponding to the test speech input into the system under test. Specifically, the echo return attenuation gain may be calculated according to the calculation method shown in the following formula (5):

[0272]

[0273] In formula (5), y(n) represents the expected echo signal, e(n) represents the error signal, ERLE represents the echo return attenuation gain, and the unit of the echo return attenuation gain is dB. The echo return attenuation gain calculated by formula (5) can more truly reflect the echo cancellation capability of the system under test.

[0274] After obtaining the performance characterization parameters of the system under test on the test item of front-end signal processing effect, the test result of the system under test on the test item of front-end signal processing effect can be generated according to the obtained parameters. Specifically, it can be determined whether the performance characterization parameters of the system under test on the test item of front-end signal processing effect meet the technical requirements of the system under test or relevant standards and specifications. If so, it is determined that the test item of front-end signal processing effect of the system under test has passed. Otherwise, it is determined that the test item of front-end signal processing effect of the system under test has failed. The test result of the system under test on the test item of front-end signal processing effect is generated according to the analysis and judgment results. Among them, the test result of the system under test on the test item of front-end signal processing effect may include a test result indicating whether the test item of front-end signal processing effect has passed, and optionally, it may also include relevant test information such as the name of the corresponding test data set and the number of test voices in the corresponding test data set.

[0275] (2-5) Test of speaker separation effect of the system under test

[0276] The testing process of the speaker separation effect of the system under test may include:

[0277] Step S2-5-e1: input the test speech in the test data set corresponding to the test item of speaker separation effect of the system under test into the system under test.

[0278] Among them, the test data set corresponding to the speaker separation effect test item includes test speech containing at least two speakers' alternating conversations, and the conversation duration is preferably 20 minutes.

[0279] Step S2-5-e2: Obtain the operating data of the system under test on the test item of speaker separation effect.

[0280] Among them, the operating data of the system under test on the test item of speaker separation effect is the operating result of the system under test on the test item of speaker separation effect for the input test speech. Specifically, the operating results of the system under test on the test item of speaker separation effect for the input test speech include: the speaker separation result of the test speech input to the system under test, that is, the result obtained by the system under test performing speaker separation on the input test speech.

[0281] Step S2-5-e3: Determine the test result corresponding to the test item of speaker separation effect of the system under test based on the operating data of the system under test on the test item of front-end signal processing effect.

[0282] Specifically, based on the operating result of the system under test on the test item of speaker separation effect for the input test speech, that is, the speaker separation result of the test speech input to the system under test, the performance characterization parameters of the system under test on the test item of speaker separation effect are determined, and based on the performance characterization parameters of the system under test on the test item of speaker separation effect, the test result of the system under test on the test item of speaker separation effect is generated.

[0283] In this embodiment, the performance characterization parameters of the tested system in the test item of speaker separation effect may include, but are not limited to: error separation rate. The error separation rate is the ratio of the duration of the speech segment of the input test speech that is incorrectly separated by the tested system to the duration of the entire valid speech segment. The error separation rate can be calculated by the calculation method shown in the following formula (6):

[0284]

[0285] In formula (6), S represents the number of speaker segments that both the actual speaker separation result and the speaker separation result output by the tested system contain the same speaker (pair), dur(s) represents the duration of segment s, and N ref (s) represents the number of actual results in segment s, N hyp (s) represents the number of system output results in segment s, N correct(s) represents the number of correct correspondences between the system output results and the actual results in segment s, and DER represents the separation error rate. The error separation rate calculated by formula (6) can more realistically reflect the speaker separation ability of the tested system.

[0286] After obtaining the performance characterization parameters of the tested system on the test item of speaker separation effect, the test result of the tested system on the test item of speaker separation effect can be generated according to the obtained parameters. Specifically, it can be determined whether the performance characterization parameters of the tested system on the test item of speaker separation effect meet the technical requirements of the tested system or relevant standards and specifications. If so, it is determined that the test item of speaker separation effect of the tested system has passed. Otherwise, it is determined that the test item of speaker separation effect of the tested system has failed. The test result of the tested system on the test item of speaker separation effect is generated according to the judgment result. The test result of the tested system on the test item of speaker separation effect can include a test result indicating whether the test item of speaker separation effect has passed. Optionally, it can also include relevant test information such as the name of the corresponding test data set and the number of test voices in the corresponding test data set.

[0287] (2-6) Testing of the language information recognition effect of the system under test

[0288] The testing process of the language information recognition effect of the system under test may include:

[0289] Step S2-5-e1: Input the test speech in the test data set corresponding to the test item of the language information recognition effect of the system under test into the system under test.

[0290] The test data set corresponding to the test item of language information recognition effect of the system under test contains test data of one or more language information, and the number of test speech items in the test data set is not less than 200.

[0291] Step S2-5-e2: Obtain the operating data of the system under test on the test item of language information recognition effect.

[0292] Among them, the operating data of the system under test on the test item of language information recognition effect is the operating result of the system under test on the test item of language information recognition effect for the input test voice. Specifically, the operating results of the system under test on the test item of language information recognition effect for the input test voice include: the recognition result of the test voice input into the system under test.

[0293] Step S2-5-e3: Determine the test result corresponding to the test item of language information recognition effect of the system under test based on the operating data of the system under test on the test item of language information recognition effect.

[0294] Specifically, based on the operating results of the system under test on the test item of language information recognition effect for the input test speech, that is, the recognition result of the test speech input to the system under test, determine the performance characterization parameters of the system under test on the test item of language information recognition effect; based on the performance characterization parameters of the system under test on the test item of language information recognition effect, generate the test results of the system under test on the test item of language information recognition effect.

[0295] Among them, the performance characterization parameters of the tested system in the test item of language information recognition effect may include, but are not limited to: language information recognition recall rate. The language information recognition recall rate is determined based on the number of times the specified type of language information is correctly recognized and the total number of times that it should be correctly recognized. Specifically, the number of times the specified type of language information is correctly recognized divided by the total number of times that it should be correctly recognized can be used as the language information recognition recall rate.

[0296] After obtaining the performance characterization parameters of the tested system on the test item of language information recognition effect, the test result of the tested system on the test item of language information recognition effect can be generated according to the obtained parameters. Specifically, it can be determined whether the performance characterization parameters of the tested system on the test item of language information recognition effect meet the technical requirements of the tested system or relevant standards and specifications. If so, it is determined that the tested system has passed the test item of language information recognition effect. Otherwise, it is determined that the tested system has failed the test item of language information recognition effect. The test result of the tested system on the test item of language information recognition effect can include a test result indicating whether the tested item of language information recognition effect has passed, and optionally, it can also include relevant test information such as the name of the corresponding test data set and the number of test voices in the corresponding test data set.

[0297] (2-7) Test of system stability of the system under test

[0298] The stability test of the system under test mainly tests whether the system under test can run stably under the conditions of given software and hardware configuration and system concurrent paths, and / or whether the resource utilization rate of the system under test is stable. It should be noted that the given software and hardware configuration and system concurrent paths must meet the ability of the system under test to operate normally. Among them, the stable operation of the system under test means that the system under test does not crash, freeze or malfunction, and meets the performance requirements and can continue to operate normally. The stable resource utilization rate of the system under test means that the utilization rate of various resources such as the system physical memory, virtual memory, CPU, GPU, handles, network resources, etc. of the system under test is continuously stable.

[0299] The testing process of system stability of the system under test can include:

[0300] Step S2-7-e1: Input the test speech in the test data set corresponding to the test item of system stability of the system under test into the system under test.

[0301] In this embodiment, the test voice can be input into the system under test in the following manner: the test voice in the test data set corresponding to the test item of system stability is continuously and uninterruptedly input into the system under test in an online state for processing (the test voice in the test data set can be input cyclically) until the test duration in the online state reaches a first test duration (for example, 7 days); the test voice in the test data set corresponding to the system stability is continuously and uninterruptedly input into the system under test in an offline state for processing (the test voice in the test data set can be input cyclically) until the test duration in the offline state reaches a second test duration (for example, 3 days).

[0302] Taking into account that in real scenarios, the system under test may process voice data continuously for a long time, and the system under test is mostly in an online state. In addition, the system under test may enter an offline state from an online state for some reason. In order to evaluate the true performance of the system under test (whether it can run stably and whether the resource utilization rate is stable), the present application is proposed based on the operating conditions that may occur when the system under test is applied in real scenarios, so that the system under test continuously processes the test voice, and first continuously processes the test voice for a period of time in an online state, and then continuously processes the test voice for a period of time in an offline state. In this process, the operating conditions and resource usage of the system under test are detected. In addition, when testing the operation and resource usage of the system under test, it can be decided which functions the system under test should run for the input test voice based on the main functions running in the real scenario. For example, the functions that the system under test runs more in the real scenario include voice signal acquisition function, voice-to-text function, voice wake-up function, front-end signal processing function, and speaker separation function. When testing the operation and resource usage of the system under test, the system under test can run the voice signal acquisition function, voice-to-text function, voice wake-up function, front-end signal processing function, and speaker separation function for the input test voice.

[0303] Step S2-5-e2: Obtain the operating data of the system under test on the test item of system stability.

[0304] During the test of the system stability of the system under test, the operation status of the system under test (such as whether there is a crash, pseudo-death, malfunction, etc.) and / or resource utilization (such as changes in the utilization rate of various resources such as system physical memory, virtual memory, CPU, GPU, handles, network resources, etc.) can be continuously monitored. The operation data of the system under test on the test item of system stability in this step is the data that characterizes the operation status and / or resource utilization of the system under test.

[0305] Step S2-5-e3: Determine the test result of the system under test on the test item of system stability according to the operating data of the system under test on the test item of system stability.

[0306] Specifically, based on the operating data of the system under test on the test item of system stability, that is, data characterizing the operating status and / or resource utilization of the system under test, it is possible to analyze whether the system under test can run stably and / or whether the resource utilization rate of the system under test is stable, and then generate the test results of the system under test on the test item of system stability according to the analysis results. Among them, the test results of the system under test on the test item of system stability may include test results characterizing whether the system under test can run stably and / or whether the resource utilization rate of the system under test is stable, and optionally, may also include the name of the corresponding test data set, the number of test voices in the corresponding test data set, the hardware and software configuration of the system under test, the number of concurrent paths of the system, etc.

[0307] It should be noted that the test data set corresponding to any of the above functional test items is obtained from the total test data set produced according to the requirements shown in Table 1. When testing any of the above performance test items, the test data in the test data set corresponding to the performance test item can be input into the system under test based on the programmable test tool mentioned above, and the operating data of the system under test in the functional test item can be obtained based on the programmable test tool, and then the operating data of the system under test in the functional test item can be analyzed based on the test statistics tool. It should be noted that for the test item of system stability, the resource usage of the system under test can be monitored based on the resource monitoring tool.

[0308] An embodiment of the present application also provides a testing device for a speech recognition system. The testing device for the speech recognition system provided in the embodiment of the present application is described below. The testing device for the speech recognition system described below and the testing method for the speech recognition system described above can be referenced to each other.

[0309] See also Figure 2 , shows a schematic diagram of the structure of a test device for a speech recognition system provided in an embodiment of the present application, which may include: a data input module 201, a data acquisition module 202 and a test result determination module 203.

[0310] The data input module 201 is used to input the test speech in the test data set corresponding to the test item of the system under test into the system under test.

[0311] The test data set is obtained from the total test data set of the system under test according to the corresponding test items, the total number of test voices in the total test data set is at least 2000, and the total test data set includes multiple types of test voices with different qualities, namely, Class A test voices, Class B test voices, Class C test voices, and Class D test voices. Class A test speech and Class D test speech, the category to which the test speech belongs is determined by the speech type and / or audio type of the test speech, each category of test speech in the test data set at least includes speech with audio type of weak noise normal and / or truncated audio, speech with audio type of weak noise normal refers to speech with a speech signal-to-noise ratio greater than or equal to 20dB, the proportion of Class A test speech in the test data set should not be less than 70%, the proportion of Class B test speech should not be less than 15% and should not be greater than 20%, the proportion of Class C test speech should not be less than 5% and should not be greater than 10%, the proportion of Class D test speech should not be greater than 5%, the proportion of test speech with a duration of 3 to 5 seconds in the test data set is more than 80%, and the number of speakers of each speech category of the test speech should not be less than 30.

[0312] Data acquisition module 202, for obtaining the operating data of the system under test on the test item.

[0313] The test result determination module 203 is used to determine the test result of the system under test on the test item according to the operation data of the system under test on the test item.

[0314] The speech recognition system device provided by the embodiment of the present application first inputs the test speech in the test data set corresponding to the test item of the system under test into the system under test, then obtains the operating data of the system under test on the test item, and finally determines the test result of the system under test on the test item according to the operating data of the system under test on the test item. Since the test data set corresponding to the test item of the system under test is obtained from the total test data set of the system under test, and the total test data set of the system under test is based on the actual application scenario of the system under test (for example, the type of speech and audio type involved in the speech recognized by the system under test in the actual scenario, the duration of the speech recognized by the system under test in the actual scenario, etc.), the test result is constructed with the goal of obtaining a test result that can more accurately reflect the capability of the system under test. Therefore, the system under test is tested based on the test data in the constructed total test data set, and a better test effect can be obtained. The test device of the speech recognition system provided by the embodiment of the present application is suitable for the design and implementation of the speech recognition system or speech recognition subsystem test of the intelligent speech interaction application by intelligent speech service providers, users and third-party testing agencies.

[0315] In a possible implementation, the data input module 201 is specifically configured to use a voice playback device to play a test voice in a test data set corresponding to a test item of the system under test, and input the test voice played by the voice playback device into the system under test;

[0316] Wherein, the voice playback device is a first playback device or a second playback device;

[0317] The first playback device is a player, and the first playback device meets the following parameter requirements:

[0318] Frequency response: 74Hz~18kHz; Maximum sound pressure level: 102dB(A);

[0319] The second playback device includes a power amplifier and an artificial mouth, and the second playback device meets the following parameter requirements:

[0320] Signal-to-noise ratio: 90dB; Gain control: 0dB-25dB; Frequency response of the power amplifier: 20Hz-20kHz; Frequency response of the artificial mouth: 100Hz-10kHz; Maximum sound pressure level: 110 dB(A);

[0321] The environment in which the test voice in the test data set corresponding to the test item of the system under test is played by the voice playback device satisfies:

[0322] When the external noise does not exceed 55dB(A), the indoor background noise is ≤20dB(A);

[0323] The test device of the speech recognition system may further include: a noise playing module;

[0324] A noise playback module, configured to play the background noise of the test voice by using a noise playback device while playing the test voice in the test data set corresponding to the test item of the system under test by using a voice playback device, wherein the background noise is recorded in the use scenario of the system under test;

[0325] The noise playback device meets the following parameter requirements:

[0326] Power: 70W (peak 125W); frequency response: 50Hz~21kHz;

[0327] Sound pressure: ≤113dB SPL@1m; Input impedance: 10KΩ; Maximum input level: 22dBu;

[0328] The noise playback device is processed by frequency response equalization, and the frequency response difference between the playback noise at the voice acquisition module of the system under test and the noise in the use scenario of the system under test satisfies +3dB.

[0329] In a possible implementation manner, the test items include functional test items and / or performance test items.

[0330] The functional test items include one or more of the voice signal acquisition function, voice-to-text function, voice wake-up function, front-end signal processing function, speaker separation function, language information recognition function and voice recognition post-processing function; the performance test items include one or more of the voice recognition effect, voice recognition efficiency, voice wake-up effect, front-end signal processing effect, speaker separation effect, language information recognition effect and system stability;

[0331] Among them, the voice wake-up function includes one or more of the following functions: a wake-up function based on a predefined wake-up word, a wake-up function based on a custom wake-up word, a wake-up function based on multiple wake-up words, a function of monitoring multiple audio streams while performing voice wake-up, a function of waking up based on voice interruption, and a collaborative wake-up function. The collaborative wake-up function means that when multiple devices using the same wake-up word appear in the same scene, only one device responds to a wake-up operation; the front-end signal processing function includes one or more of the following functions: voice encoding and decoding function, voice endpoint detection function, voice enhancement function, sound source localization function , audio format conversion function, voice resampling function, audio quality judgment function, acoustic echo cancellation function; the speaker separation function includes one or more of the following functions: speaker segmentation function, speaker clustering function; the language information recognition function includes one or more of the following functions: one language or multiple languages ​​recognition function, multiple languages ​​mixed reading recognition function, one Chinese dialect or multiple Chinese dialects recognition function, multiple Chinese dialects mixed reading recognition function, language information endpoint recognition function; the speech recognition post-processing function includes one or more of the following functions: character regularization function, punctuation prediction function, text smoothing function, text replacement function;

[0332] If the test items of the system under test include the functional test items, the operation data of the system under test on the functional test items include: the operation results generated by the system under test on the functional test items for the input test speech;

[0333] The test result determination module 203 determines the test result of the system under test on the test item according to the operation data of the system under test on the test item, including:

[0334] Analyzing whether the system under test provides the function indicated by the functional test item according to the running data of the system under test on the functional test item; and generating a test result of the system under test on the test item according to the analysis result;

[0335] The performance test items include: one or more of speech recognition effect, speech recognition efficiency, speech wake-up effect, front-end signal processing effect, speaker separation effect, language information recognition effect and system stability;

[0336] If the performance test item includes a first performance test item, the operating data of the system under test on the first performance test item includes: the operating result generated by the system under test on the first functional test item for the input test voice, wherein the first performance test item includes one or more test items of speech recognition effect, speech recognition efficiency, speech wake-up effect, front-end signal processing effect, speaker separation effect, and language information recognition effect; If the system under test includes a second performance test item, the operating data of the system under test on the second performance test item includes: data characterizing the operating status and / or resource utilization of the system under test obtained during the test of the system under test based on the test voice in the test data set corresponding to the second performance test item, wherein the second performance test item includes system stability;

[0337] The test result determination module 203 determines the test result corresponding to the test item of the system under test according to the running data of the system under test on the test item, including:

[0338] If the performance test items include the first performance test item, then based on the operating data of the system under test on the first performance test item, the performance characterization parameters of the system under test on the first performance test item are determined, and the test results of the system under test on the first performance test item are generated based on the determined performance characterization parameters; if the performance test items include the second performance test item, then based on the operating data of the system under test on the second performance test item, the test results of the system under test on the second performance test item are generated.

[0339] In a possible implementation, the test result determination module 203 analyzes whether the system under test provides the function indicated by the functional test item according to the running data of the system under test on the functional test item, including:

[0340] If the functional test item includes the voice signal acquisition function, then according to the operating result of the system under test on the voice signal acquisition function for the input test voice, analyze whether the system under test provides the voice signal acquisition function under the set pickup distance, wherein the system under test and the voice playback device are arranged according to the set pickup distance, the pickup distance is set according to the application scenario of the system under test, and the voice playback device is used to play the test voice in the test data set corresponding to the voice signal acquisition function, so that the system under test can collect it;

[0341] If the functional test item includes the speech-to-text function, then analyzing whether the system under test provides a function of converting the received speech into text that matches the content of the received speech and outputting the text according to the operating result of the system under test on the speech-to-text function for the input test speech;

[0342] If the functional test item includes the voice wake-up function, analyzing whether the system under test provides the voice wake-up function according to the running result generated by the system under test on the voice wake-up function for the input test voice;

[0343] If the functional test item includes the front-end signal processing function, analyzing whether the system under test provides the front-end signal processing function according to the operating result generated by the system under test on the front-end signal processing function for the input test voice;

[0344] If the functional test item includes the speaker separation function, analyzing whether the system under test provides the speaker separation function according to the operating result of the system under test on the speaker separation function for the input test speech, wherein the test data set corresponding to the speaker separation function includes test speech of multiple speakers talking alternately;

[0345] If the functional test item includes the language information recognition function, analyzing whether the system under test provides the language information recognition function according to the operating result of the system under test on the language information recognition function for the input test speech;

[0346] If the functional test item includes a speech recognition post-processing function, analyzing whether the system under test provides the speech recognition post-processing function according to the running result of the system under test on the speech recognition post-processing function for the input test speech;

[0347] The test result determination module 203 determines the performance characterization parameters of the system under test on the first performance test item according to the operation data of the system under test on the first performance test item, including:

[0348] If the first performance test item includes the speech recognition effect, then according to the operating result of the system under test on the speech recognition effect for the test speech input and the speech content annotated text of the test speech input to the system under test, determine the parameters that can characterize the word recognition ability and / or sentence recognition ability of the system under test;

[0349] If the first performance test item includes the speech recognition efficiency, determining a parameter capable of characterizing the recognition response and / or recognition throughput of the system under test according to the recognition response time of the system under test to the input test speech;

[0350] If the first performance test item includes the voice wake-up effect, determining parameters that can characterize the voice wake-up capability, and / or voice interruption capability, and / or voice interruption wake-up capability, and / or collaborative wake-up capability of the system under test according to the wake-up condition of the system under test on the input test voice;

[0351] If the first performance test item includes the front-end signal processing effect, then according to the audio bit rate before and after the test speech encoding input to the system under test, determine the parameter that can characterize the speech encoding and decoding capability of the system under test, and / or, according to the endpoint detection result of the test speech input to the system under test, determine the parameter that can characterize the endpoint detection capability of the system under test, and / or, according to the signal-to-noise ratio before and after the test speech enhancement input to the system under test and / or the noise amplitude before and after the test speech enhancement input to the system under test, determine the parameter that can characterize the speech enhancement capability of the system under test, and / or, according to the number of correct sound source localization of the input test speech and the total number of sound source localization requests of the system under test, determine the parameter that can characterize the sound source localization capability of the system under test, and / or according to the expected echo signal and error signal corresponding to the test speech input to the system under test, determine the parameter that can characterize the echo cancellation capability of the system under test;

[0352] If the first performance test item includes the speaker separation effect, determining a parameter capable of characterizing the speaker separation capability of the system under test according to the duration of the speech segment with incorrect separation and the duration of the entire valid speech segment in the speaker separation result of the test speech input by the system under test;

[0353] If the first performance test item includes the language information recognition effect, then the parameters that can characterize the recognition ability of the system under test for the specified type of language information are determined based on the number of times the specified type of language information is correctly recognized by the system under test and the total number of times the system under test recognizes the specified type of language information.

[0354] In a possible implementation, the test result determination module 203 determines a parameter capable of characterizing a recognition response and / or a recognition throughput of the system under test according to a recognition response time of the system under test to an input test speech, including:

[0355] Determining parameters that can characterize the recognition response and / or recognition throughput of the system under test according to the recognition response time of the system under test to the input test speech includes:

[0356] According to the recognition response time of all test voices input to the system under test and the total number of all test voices input to the system under test, and the calculation method shown in the following formula, the average speech recognition response time T is determined. avr :

[0357]

[0358] in, represents the time when the system under test gives the recognition result of the i-th test speech input, represents the time when the i-th test speech input into the system under test ends, and N represents the total number of test speech inputs into the system under test;

[0359] And / or, according to the clause recognition response time of all test speech input to the system under test and the total number of all test speech input to the system under test, and the calculation method shown in the following formula, determine the average clause response time T of speech recognition avsr :

[0360]

[0361] The recognition response time of a clause of a test speech is the time it takes for the system under test to give a recognition result of a clause in the test speech after receiving the test speech. represents the time when the system under test gives the recognition result of the j-th clause in the i-th test speech input, represents the time when the j-th clause in the i-th test speech input to the system under test ends, and N represents the total number of test speech inputs to the system under test;

[0362] and / or, determining a speech recognition response time distribution based on recognition response times of all test speech inputs into the system under test;

[0363] and / or, determining a speech recognition throughput rate according to a recognition response time of all test speech input to the system under test and a duration of all test speech input to the system under test;

[0364] The test result determination module 203 determines the parameters that can characterize the cooperative wake-up capability of the system under test according to the wake-up condition of the system under test on the input test voice, including:

[0365] Determine the collaborative wake-up success rate according to the number of collaborative wake-up successes and the total number of collaborative wake-up operations, wherein the collaborative wake-up success means that, when the wake-up operation is performed on multiple devices, only one device is successfully woken up, and the total number of collaborative wake-up operations is the total number of collaborative wake-up test voices input into the system under test;

[0366] And / or, the collaborative wake-up success rate is determined according to the number of correct collaborative wake-up operations and the total number of collaborative wake-up operations, wherein correct collaborative wake-up means that the wake-up operation is performed on multiple devices and the device intended by the user is correctly woken up.

[0367] Determine the collaborative wake-up success rate according to the number of collaborative wake-up successes and the total number of collaborative wake-up operations, wherein the collaborative wake-up success means that, when the wake-up operation is performed on multiple devices, only one device is successfully woken up, and the total number of collaborative wake-up operations is the total number of collaborative wake-up test voices input into the system under test;

[0368] And / or, the collaborative wake-up success rate is determined according to the number of correct collaborative wake-up operations and the total number of collaborative wake-up operations, wherein correct collaborative wake-up means that the wake-up operation is performed on multiple devices and the device intended by the user is correctly woken up.

[0369] In a possible implementation, if the test item of the system under test includes the voice wake-up effect, the test scenario of the voice wake-up effect includes one or more of the following scenarios:

[0370] A high-noise scene with a signal-to-noise ratio greater than or equal to a first preset signal-to-noise ratio and less than or equal to a second preset signal-to-noise ratio;

[0371] A low-noise scene with a signal-to-noise ratio greater than the second preset signal-to-noise ratio and less than or equal to a third preset signal-to-noise ratio;

[0372] A quiet scene with a signal-to-noise ratio greater than the third preset signal-to-noise ratio;

[0373] Custom noise scenes;

[0374] The third preset signal-to-noise ratio is greater than the second preset signal-to-noise ratio, and the second preset signal-to-noise ratio is greater than the first signal-to-noise ratio.

[0375] In a possible implementation, if the test items of the system under test include system stability, then the test items of the system under test include system stability;

[0376] The data input module 201 inputs the test speech in the test data set corresponding to the test item of the system under test into the system under test, including:

[0377] The test speech in the test data set corresponding to the system stability is continuously and uninterruptedly input into the system under test in an online state for processing until the test duration in the online state reaches a first test duration, and the test speech in the test data set corresponding to the system stability is continuously and uninterruptedly input into the system under test in an offline state for processing until the test duration in the offline state reaches a second test duration.

[0378] The present application also provides a test device for a speech recognition system. Figure 3 , shows a schematic diagram of the structure of the test device of the speech recognition system, the device may include: at least one processor 301, at least one communication interface 302, at least one memory 303 and at least one communication bus 304;

[0379] In the embodiment of the present application, the number of the processor 301, the communication interface 302, the memory 303, and the communication bus 304 is at least one, and the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304;

[0380] The processor 301 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0381] The memory 303 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0382] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0383] The test speech in the test data set corresponding to the test item of the system under test is input into the system under test, wherein the test data set is obtained from the test data set of the system under test according to the corresponding test item, the total number of test speech in the test data set is at least 2000, and the test data set includes multiple categories of test speech with different qualities, namely category A test speech, category B test speech, category C test speech and category D test speech, the category to which the test speech belongs is determined by the speech type and / or audio type of the test speech, the audio type at least includes weak noise normal and / or truncated audio, each category of test speech in the test data set includes at least speech with audio type of weak noise normal and / or truncated audio, and speech with audio type of weak noise normal means that the speech signal-to-noise ratio is greater than or equal to 20dB The proportion of Class A test voices in the total test data set should not be less than 70%, the proportion of Class B test voices should not be less than 15% and should not be greater than 20%, the proportion of Class C test voices should not be less than 5% and should not be greater than 10%, the proportion of Class D test voices should not be greater than 5%, the proportion of test voices with a duration of 3 to 5 seconds in the total test data set should be greater than 80%, and the number of speakers of each test voice type should not be less than 30; obtain the operating data of the system under test on the test items;

[0384] A test result of the system under test on the test item is determined according to the operating data of the system under test on the test item.

[0385] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0386] The embodiment of the present application further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0387] The test speech in the test data set corresponding to the test item of the system under test is input into the system under test, wherein the test data set is obtained from the test data set of the system under test according to the corresponding test item, the total number of test speech in the test data set is at least 2000, and the test data set includes multiple categories of test speech with different qualities, namely category A test speech, category B test speech, category C test speech and category D test speech, the category to which the test speech belongs is determined by the speech type and / or audio type of the test speech, the audio type at least includes weak noise normal and / or truncated audio, each category of test speech in the test data set includes at least speech with audio type of weak noise normal and / or truncated audio, and speech with audio type of weak noise normal means that the speech signal-to-noise ratio is greater than or equal to 20dB The proportion of Class A test voices in the total test data set should not be less than 70%, the proportion of Class B test voices should not be less than 15% and should not be greater than 20%, the proportion of Class C test voices should not be less than 5% and should not be greater than 10%, the proportion of Class D test voices should not be greater than 5%, the proportion of test voices with a duration of 3 to 5 seconds in the total test data set should be greater than 80%, and the number of speakers of each test voice type should not be less than 30; obtain the operating data of the system under test on the test items;

[0388] A test result of the system under test on the test item is determined according to the operating data of the system under test on the test item.

[0389] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0390] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0391] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for testing a speech recognition system, characterized in that: include: Input the test speech in the test data set corresponding to the test item of the system under test into the system under test, wherein the test data set is obtained from the test data set of the system under test according to the corresponding test item, the total amount of test speech in the test data set is at least 2000, the test data set includes multiple types of test speech, namely, type A test speech, type B test speech, type C test speech and type D test speech, the category to which the test speech belongs is determined by the speech type and / or audio type of the test speech, each type of test speech in the test data set at least includes speech with audio type of weak noise normal and / or truncated audio, the audio type of weak noise normal refers to speech with a speech signal-to-noise ratio greater than or equal to 20dB, the proportion of type A test speech in the test data set should not be less than 70%, the proportion of type B test speech should not be less than 15% and should not be greater than 20%, the proportion of type C test speech should not be less than 5% and should not be greater than 10%, the proportion of type D test speech should not be greater than 5%, and the proportion of test speech with a duration of 3 to 5 seconds in the test data set is more than 80%; Obtaining operation data of the system under test on the test item; Determining a test result of the system under test on the test item according to the operating data of the system under test on the test item; The test items include functional test items; The functional test items include: one or more of: voice signal acquisition function, voice-to-text function, voice wake-up function, front-end signal processing function, speaker separation function, language information recognition function and voice recognition post-processing function; Among them, the voice wake-up function includes one or more of the following functions: a wake-up function based on a predefined wake-up word, a wake-up function based on a custom wake-up word, a wake-up function based on multiple wake-up words, a function of monitoring multiple audio streams while performing voice wake-up, a function of waking up based on voice interruption, and a collaborative wake-up function. The collaborative wake-up function means that when multiple devices using the same wake-up word appear in the same scene, only one device responds to a wake-up operation; the front-end signal processing function includes one or more of the following functions: voice encoding and decoding function, voice endpoint detection function, voice enhancement function, sound source localization function , audio format conversion function, speech resampling function, audio quality judgment function, acoustic echo cancellation function; the speaker separation function includes one or more of the following functions: speaker segmentation function, speaker clustering function; the language information recognition function includes one or more of the following functions: one language or multiple languages ​​recognition function, multiple languages ​​mixed reading recognition function, one Chinese dialect or multiple Chinese dialects recognition function, multiple Chinese dialects mixed reading recognition function, language information endpoint recognition function; the speech recognition post-processing function includes one or more of the following functions: character regularization function, punctuation prediction function, text smoothing function, text replacement function.

2. The testing method according to claim 1, characterized in that: The step of inputting a test speech in a test data set corresponding to a test item of the system under test into the system under test comprises: Using a voice playback device to play a test voice in a test data set corresponding to a test item of the system under test, and inputting the test voice played by the voice playback device into the system under test; Wherein, the speech playback device is a first playback device or a second playback device, the first playback device is a player, and the second playback device includes a power amplifier and an artificial mouth; The first playback device meets the following parameter requirements: Frequency response: 74Hz~18kHz; Maximum sound pressure level: 102dB(A); The second playback device meets the following parameter requirements: Signal-to-noise ratio: 90dB; gain control: 0dB-25dB; frequency response of the power amplifier: 20Hz-20kHz; frequency response of the artificial mouth: 100Hz-10kHz; maximum sound pressure level: 110dB(A); The environment in which the test voice in the test data set corresponding to the test item of the system under test is played by the voice playback device satisfies: When the external noise does not exceed 55dB(A), the indoor background noise is ≤20dB(A); The test method also includes: While the test speech in the test data set corresponding to the test item of the system under test is played by the speech playback device, the background noise of the test speech is played by the noise playback device, wherein the background noise is recorded in the usage scenario of the system under test, and the noise playback device meets the following parameter requirements: Power: 70W (peak 125W); frequency response: 50Hz~21kHz; Sound pressure: ≤113dB SPL@1m; Input impedance: 10KΩ; Maximum input level: 22dBu; The noise playback device is processed by frequency response equalization, and the frequency response difference between the playback noise at the voice acquisition module of the system under test and the noise in the use scenario of the system under test satisfies +3dB.

3. The testing method according to claim 1 or 2, characterized in that: If the test items of the system under test include the functional test items, the operation data of the system under test on the functional test items include: the operation results generated by the system under test on the functional test items for the input test speech; The step of determining the test result of the system under test on the test item according to the running data of the system under test on the test item includes: Analyzing whether the system under test provides the function indicated by the functional test item according to the running data of the system under test on the functional test item; and generating a test result of the system under test on the test item according to the analysis result; The test items also include performance test items; The performance test items include: one or more of speech recognition effect, speech recognition efficiency, speech wake-up effect, front-end signal processing effect, speaker separation effect, language information recognition effect and system stability; If the performance test item includes a first performance test item, the operating data of the system under test on the first performance test item includes: the operating result generated by the system under test on the first performance test item for the input test voice, wherein the first performance test item includes one or more test items of speech recognition effect, speech recognition efficiency, speech wake-up effect, front-end signal processing effect, speaker separation effect, and language information recognition effect; if the system under test includes a second performance test item, the operating data of the system under test on the second performance test item includes: data characterizing the operating status and / or resource utilization status of the system under test obtained during the testing of the system under test based on the test voice in the test data set corresponding to the second performance test item, wherein the second performance test item includes system stability; The step of determining the test result corresponding to the test item of the system under test according to the running data of the system under test on the test item includes: If the performance test items include the first performance test item, then based on the operating data of the system under test on the first performance test item, the performance characterization parameters of the system under test on the first performance test item are determined, and the test results of the system under test on the first performance test item are generated based on the determined performance characterization parameters; if the performance test items include the second performance test item, then based on the operating data of the system under test on the second performance test item, the test results of the system under test on the second performance test item are generated.

4. The testing method according to claim 3, characterized in that: The analyzing, according to the operation data of the system under test on the functional test item, whether the system under test provides the function indicated by the functional test item includes: If the functional test item includes the voice signal acquisition function, then according to the operating result of the system under test on the voice signal acquisition function for the input test voice, analyze whether the system under test provides the voice signal acquisition function under the set pickup distance, wherein the system under test and the voice playback device are arranged according to the set pickup distance, the pickup distance is set according to the application scenario of the system under test, and the voice playback device is used to play the test voice in the test data set corresponding to the voice signal acquisition function, so that the system under test can collect it; If the functional test item includes the speech-to-text function, then analyzing whether the system under test provides a function of converting the received speech into text that matches the content of the received speech and outputting the text according to the operating result of the system under test on the speech-to-text function for the input test speech; If the functional test item includes the voice wake-up function, analyzing whether the system under test provides the voice wake-up function according to the running result generated by the system under test on the voice wake-up function for the input test voice; If the functional test item includes the front-end signal processing function, analyzing whether the system under test provides the front-end signal processing function according to the operating result generated by the system under test on the front-end signal processing function for the input test voice; If the functional test item includes the speaker separation function, analyzing whether the system under test provides the speaker separation function according to the operating result of the system under test on the speaker separation function for the input test speech, wherein the test data set corresponding to the speaker separation function includes test speech of multiple speakers talking alternately; If the functional test item includes the language information recognition function, analyzing whether the system under test provides the language information recognition function according to the operating result of the system under test on the language information recognition function for the input test speech; If the functional test item includes a speech recognition post-processing function, analyzing whether the system under test provides the speech recognition post-processing function according to the running result of the system under test on the speech recognition post-processing function for the input test speech; The step of determining the performance characterization parameter of the system under test on the first performance test item according to the operation data of the system under test on the first performance test item includes: If the first performance test item includes the speech recognition effect, then according to the operating result of the system under test on the speech recognition effect for the test speech input and the speech content annotated text of the test speech input to the system under test, determine the parameters that can characterize the word recognition ability and / or sentence recognition ability of the system under test; If the first performance test item includes the speech recognition efficiency, determining a parameter capable of characterizing the recognition response and / or recognition throughput of the system under test according to the recognition response time of the system under test to the input test speech; If the first performance test item includes the voice wake-up effect, determining parameters that can characterize the voice wake-up capability, and / or voice interruption capability, and / or voice interruption wake-up capability, and / or collaborative wake-up capability of the system under test according to the wake-up condition of the system under test on the input test voice; If the first performance test item includes the front-end signal processing effect, then according to the audio bit rate before and after the test voice encoding input to the system under test, determine the parameters that can characterize the voice encoding and decoding capability of the system under test, and / or, according to the endpoint detection result of the test voice input to the system under test, determine the parameters that can characterize the endpoint detection capability of the system under test, and / or, according to the signal-to-noise ratio before and after the test voice enhancement input to the system under test and / or the noise amplitude before and after the test voice enhancement input to the system under test, determine the parameters that can characterize the voice enhancement capability of the system under test, and / or, according to the number of correct sound source localization of the input test voice by the system under test and the total number of sound source localization requests, determine the parameters that can characterize the sound source localization capability of the system under test, and / or according to the expected echo signal and error signal corresponding to the test voice input to the system under test, determine the parameters that can characterize the echo cancellation capability of the system under test; If the first performance test item includes the speaker separation effect, determining a parameter capable of characterizing the speaker separation capability of the system under test according to the duration of the speech segment with incorrect separation and the duration of the entire valid speech segment in the speaker separation result of the test speech input by the system under test; If the first performance test item includes the language information recognition effect, then the parameters that can characterize the recognition ability of the system under test for the specified type of language information are determined based on the number of times the specified type of language information is correctly recognized by the system under test and the total number of times the system under test recognizes the specified type of language information.

5. The testing method according to claim 4, characterized in that: The step of determining, according to the recognition response time of the system under test to the input test speech, a parameter capable of characterizing the recognition response and / or recognition throughput of the system under test comprises: According to the recognition response time of all test voices input to the system under test and the total number of all test voices input to the system under test, and the calculation method shown in the following formula, the average speech recognition response time T is determined. avr : in, represents the time when the system under test gives the recognition result of the i-th test speech input, represents the time when the i-th test speech input into the system under test ends, and N represents the total number of test speech inputs into the system under test; and / or, According to the clause recognition response time of all test speech input to the system under test and the total number of all test speech input to the system under test, and the calculation method shown in the following formula, the average clause response time T of speech recognition is determined. avsr : The recognition response time of a clause of a test speech is the time it takes for the system under test to give a recognition result of a clause in the test speech after receiving the test speech. represents the time when the system under test gives the recognition result of the j-th clause in the i-th test speech input, represents the time when the j-th clause in the i-th test speech input to the system under test ends, and N represents the total number of test speech inputs to the system under test; and / or, determining a speech recognition response time distribution based on recognition response times of all test speech inputs into the system under test; and / or, determining a speech recognition throughput rate according to a recognition response time of all test speech input to the system under test and a duration of all test speech input to the system under test; Determining parameters capable of characterizing the cooperative wake-up capability of the system under test according to the wake-up condition of the system under test in response to the input test voice, including: Determine the collaborative wake-up success rate according to the number of collaborative wake-up successes and the total number of collaborative wake-up operations, wherein the collaborative wake-up success means that, when the wake-up operation is performed on multiple devices, only one device is successfully woken up, and the total number of collaborative wake-up operations is the total number of collaborative wake-up test voices input into the system under test; And / or, the collaborative wake-up success rate is determined according to the number of correct collaborative wake-up operations and the total number of collaborative wake-up operations, wherein correct collaborative wake-up means that the wake-up operation is performed on multiple devices and the device intended by the user is correctly woken up.

6. The testing method according to claim 3, characterized in that: If the test item of the system under test includes the voice wake-up effect, the test scenario of the voice wake-up effect includes one or more of the following scenarios: A high-noise scene with a signal-to-noise ratio greater than or equal to a first preset signal-to-noise ratio and less than or equal to a second preset signal-to-noise ratio; A low-noise scene with a signal-to-noise ratio greater than the second preset signal-to-noise ratio and less than or equal to a third preset signal-to-noise ratio; A quiet scene with a signal-to-noise ratio greater than the third preset signal-to-noise ratio; Custom noise scenes; The third preset signal-to-noise ratio is greater than the second preset signal-to-noise ratio, and the second preset signal-to-noise ratio is greater than the first preset signal-to-noise ratio.

7. The testing method according to claim 3, characterized in that: If the test item of the system under test includes system stability, inputting the test speech in the test data set corresponding to the test item of the system under test into the system under test includes: The test speech in the test data set corresponding to the system stability is continuously and uninterruptedly input into the system under test in an online state for processing until the test duration in the online state reaches a first test duration, and the test speech in the test data set corresponding to the system stability is continuously and uninterruptedly input into the system under test in an offline state for processing until the test duration in the offline state reaches a second test duration.

8. A testing device for a speech recognition system, characterized in that: include: Data input module, data acquisition module and test result determination module; The data input module is used to input the test speech in the test data set corresponding to the test item of the system under test into the system under test, wherein the test data set is obtained from the test data set of the system under test according to the corresponding test item, the total amount of test speech in the test data set is at least 2000, and the test data set includes multiple categories of test speech, namely, category A test speech, category B test speech, category C test speech and category D test speech, the category to which the test speech belongs is determined by the speech type and / or audio type of the test speech, and the audio type at least includes weak noise normal and / or Truncated audio, each type of test speech in the test data set at least includes speech with weak noise normal audio type and / or truncated audio, and the audio type of weak noise normal speech refers to speech with a speech signal-to-noise ratio greater than or equal to 20dB, the proportion of Class A test speech in the test data set should not be less than 70%, the proportion of Class B test speech should not be less than 15% and should not be greater than 20%, the proportion of Class C test speech should not be less than 5% and should not be greater than 10%, the proportion of Class D test speech should not be greater than 5%, and the proportion of test speech with a duration of 3 to 5 seconds in the test data set is greater than 80%; The data acquisition module is used to acquire the operating data of the system under test on the test item; The test result determination module is used to determine the test result of the system under test on the test item according to the operation data of the system under test on the test item; The test items include functional test items; The functional test items include: one or more of: voice signal acquisition function, voice-to-text function, voice wake-up function, front-end signal processing function, speaker separation function, language information recognition function and voice recognition post-processing function; Among them, the voice wake-up function includes one or more of the following functions: a wake-up function based on a predefined wake-up word, a wake-up function based on a custom wake-up word, a wake-up function based on multiple wake-up words, a function of monitoring multiple audio streams while performing voice wake-up, a function of waking up based on voice interruption, and a collaborative wake-up function. The collaborative wake-up function means that when multiple devices using the same wake-up word appear in the same scene, only one device responds to a wake-up operation; the front-end signal processing function includes one or more of the following functions: voice encoding and decoding function, voice endpoint detection function, voice enhancement function, sound source localization function , audio format conversion function, speech resampling function, audio quality judgment function, acoustic echo cancellation function; the speaker separation function includes one or more of the following functions: speaker segmentation function, speaker clustering function; the language information recognition function includes one or more of the following functions: one language or multiple languages ​​recognition function, multiple languages ​​mixed reading recognition function, one Chinese dialect or multiple Chinese dialects recognition function, multiple Chinese dialects mixed reading recognition function, language information endpoint recognition function; the speech recognition post-processing function includes one or more of the following functions: character regularization function, punctuation prediction function, text smoothing function, text replacement function.

9. A test device for a speech recognition system, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the method for testing a speech recognition system according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method for testing a speech recognition system according to any one of claims 1 to 7 is implemented.

Images