A test method, apparatus, device, and storage medium

Abstract

The present disclosure provides a test method, device, equipment and storage medium, computer technology field, especially relates to voice recognition, audio conference and voice coding and the like technical field. The specific implementation scheme is: based on the target VAD algorithm to be tested, voice region identification is carried out on test audio, and identification result is obtained;Using the identification result and the artificial marking result of the test audio, the index value of a plurality of specified test indexes corresponding to the target VAD algorithm is calculated;The reference index value of the plurality of specified test indexes determined for the specified test requirement is obtained;Based on the index value of the plurality of specified test indexes corresponding to the target VAD algorithm and the obtained reference index value, the test result of the target VAD algorithm under the specified test requirement is determined. It can be seen that through the present scheme, the performance of the target VAD algorithm under various test requirements can be more comprehensively evaluated.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer technology, in particular to the technical field of speech recognition, audio conference and speech coding, and specifically relates to a testing method, device, apparatus and storage medium. BACKGROUND

[0002] A VAD (Voice activity detection) algorithm is a key technology in speech signal processing, and is used to identify valid speech regions and non-speech regions in a speech signal.

[0003] In the testing process of a VAD algorithm, a testing method based on an error rate is usually used, that is, the performance of a VAD algorithm is evaluated by detecting the error rate of misjudging a non-speech region as a speech region by the VAD algorithm. SUMMARY

[0004] The present disclosure provides a testing method, device, apparatus and storage medium.

[0005] According to an aspect of the present disclosure, a testing method is provided, comprising:

[0006] Based on a target VAD algorithm to be tested, performing speech region identification on test audio to obtain an identification result; wherein the identification result represents the start endpoint and the end endpoint of each speech region identified;

[0007] Using the identification result and a manual annotation result of the test audio, calculating the index values of a plurality of specified test indicators corresponding to the target VAD algorithm; wherein the manual annotation result represents the start endpoint and the end endpoint of each speech region annotated;

[0008] Obtaining reference index values of the plurality of specified test indicators determined for a specified test requirement;

[0009] Based on the index values of the plurality of specified test indicators corresponding to the target VAD algorithm and the obtained reference index values, determining a test result of the target VAD algorithm under the specified test requirement.

[0010] According to another aspect of the present disclosure, a testing device is provided, comprising:

[0011] An identification module, configured to perform speech region identification on test audio based on a target VAD algorithm to be tested, to obtain an identification result; wherein the identification result represents the start endpoint and the end endpoint of each speech region identified;

[0012] The computing module is configured to calculate index values of the plurality of specified test indexes corresponding to the target VAD algorithm by using the recognition result and a manual annotation result of the test audio, wherein the manual annotation result represents starting endpoints and ending endpoints of each speech region obtained by annotation.

[0013] The obtaining module is configured to obtain reference index values of the plurality of specified test indexes determined for a specified test requirement.

[0014] The determining module is configured to determine a test result of the target VAD algorithm under the specified test requirement based on the index values of the plurality of specified test indexes corresponding to the target VAD algorithm and the obtained reference index values.

[0015] According to another aspect of the present disclosure, an electronic device is provided, comprising at least one processor, and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the test method according to any one of the above.

[0016] According to another aspect of the present disclosure, a non-transitory computer readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the test method according to any one of the above.

[0017] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the test method according to any one of the above.

[0018] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0020] FIG. 1A is a schematic diagram of test audio recognized by a VAD algorithm;

[0021] FIG. 1B is another schematic diagram of test audio recognized by a VAD algorithm;

[0022] FIG. 1C is another schematic diagram of test audio recognized by a VAD algorithm;

[0023] FIG. 2is a flowchart of a testing method according to the present disclosure;

[0024] FIG. 3 is a flowchart of step S204 in the embodiments of the present disclosure;

[0025] FIG. 4A is a schematic diagram of test audio after manual annotation;

[0026] FIG. 4B is a schematic diagram of test audio after manual annotation after being recognized by a VAD algorithm;

[0027] FIG. 5 is a structural schematic diagram of a testing device according to the present disclosure;

[0028] FIG. 6 is a block diagram of an electronic device for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0029] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Thus, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted from the following description.

[0030] For a piece of test audio, the following three situations can exist after being recognized by a VAD algorithm:

[0031] The first situation: the start and end points of each speech region recognized by the VAD algorithm are aligned with the start and end points of each speech region obtained by manual annotation;

[0032] The second situation: the start and end points of each speech region recognized by the VAD algorithm cannot be aligned with the start and end points of each speech region obtained by manual annotation, and division is performed at a place where it should not be divided;

[0033] The third situation: the start and end points of each speech region recognized by the VAD algorithm cannot be aligned with the start and end points of each speech region obtained by manual annotation, and division is not performed at a place where it should be divided.

[0034] Among them, FIGS. 1A-1C The above three situations are respectively shown in the schematic diagrams. In the diagrams, "s" is the position of the start point of manual annotation, "e" is the position of the end point of manual annotation; "s'" is the position of the start point recognized by the VAD algorithm, and "e'" is the position of the end point recognized by the VAD algorithm.

[0035] In the related art, the test on the VAD algorithm usually adopts an error rate based test method to evaluate the performance of the VAD algorithm, that is, the algorithm performance is evaluated by calculating the error rate of the VAD algorithm for misjudging the non-speech region as the speech region. Since the error rate based test method can only reflect part of the performance of the VAD algorithm, that is, can only reflect the error frequency of the VAD algorithm for misjudging the non-speech region as the speech region, the error rate based test method cannot comprehensively evaluate the performance of the VAD algorithm under different test requirements.

[0036] Based on the above, in order to more comprehensively evaluate the performance of the VAD algorithm under different test requirements, the embodiment of the present disclosure provides a test method, device, equipment and storage medium.

[0037] Next, first, a test method provided by the embodiment of the present disclosure is introduced.

[0038] It should be noted that in specific applications, the test method provided by the embodiment of the present disclosure can be applied to various electronic devices, such as personal computers, servers, and other devices with data processing capabilities. In addition, it can be understood that the test method provided by the embodiment of the present disclosure can be realized by software, hardware or a combination of software and hardware.

[0039] The test method provided by the embodiment of the present disclosure can include the following steps:

[0040] Based on the target VAD algorithm to be tested, the test audio is subjected to speech region recognition to obtain a recognition result; wherein the recognition result represents the start point and end point of each speech region obtained by recognition;

[0041] The recognition result and the artificial annotation result of the test audio are used to calculate the index values of a plurality of specified test indicators corresponding to the target VAD algorithm; wherein the artificial annotation result represents the start point and end point of each speech region obtained by annotation;

[0042] The reference index values of the plurality of specified test indicators determined for the specified test requirement are obtained;

[0043] Based on the index values of the plurality of specified test indicators corresponding to the target VAD algorithm and the obtained reference index values, the test result of the target VAD algorithm under the specified test requirement is determined.

[0044] In the scheme provided by the present disclosure, since the plurality of specified test indicators can be used to reflect the differences in multiple dimensions between the recognition result and the manual annotation result, the index values of the plurality of specified test indicators of the target VAD algorithm can be calculated; and the index values of the plurality of specified test indicators are analyzed in combination with the specified test requirements, that is, according to different test requirements, the index values of the plurality of specified test indicators are compared with the reference index values of the plurality of specified test indicators under the test requirements, so as to determine the test result of the target VAD algorithm under the test requirements, so that the performance of the VAD algorithm under the test requirements can be reflected from multiple dimensions. It can be seen that, by the present scheme, the performance of the target VAD algorithm under various test requirements can be more comprehensively evaluated.

[0045] The test method provided by the embodiment of the present disclosure will be introduced below in combination with the drawings.

[0046] As shown in the figure, FIG. 2 The test method provided by the embodiment of the present disclosure can include the following steps:

[0047] S201, performing speech region recognition on the test audio based on a target VAD algorithm to be tested, to obtain a recognition result; wherein the recognition result represents the start point and the end point of each speech region recognized;

[0048] In the embodiment, for the target VAD algorithm to be tested, the target VAD algorithm is used to perform speech region recognition on the test audio, that is, to recognize the start point and the end point of each speech region in the test audio as the recognition result of the VAD algorithm. It should be noted that the VAD algorithm can include various algorithms such as energy-based VAD algorithm and signal-to-noise ratio-based VAD algorithm.

[0049] For example, the test audio can be pre-acquired audio stored in an electronic device, or audio acquired in a real-time communication process, which are all reasonable, and the present embodiment does not limit the source of the test audio.

[0050] S202, calculating the index values of the plurality of specified test indicators corresponding to the target VAD algorithm by using the recognition result and a manual annotation result of the test audio; wherein the manual annotation result represents the start point and the end point of each speech region annotated;

[0051] In the embodiment, the specified test indicator can be an indicator for reflecting the difference between the recognition result and the manual annotation result, and the plurality of specified test indicators can be used to reflect the differences in multiple dimensions between the recognition result and the manual annotation result.

[0052] Optionally, in an implementation, the plurality of specified test indicators comprises a plurality of indicators among a cutoff rate indicator, a matching rate indicator, a time matching indicator, and a correctness rate indicator.

[0053] The cutoff rate indicator is used to represent granularity of speech region recognition by the target VAD algorithm.

[0054] The matching rate indicator is used to represent matching degree in quantity between the recognition result and the specified endpoint in the manual annotation result, the specified endpoint being a start endpoint or an end endpoint.

[0055] The time matching indicator is used to represent time difference between the recognition result and the specified endpoint in the manual annotation result.

[0056] The correctness rate indicator is used to represent accuracy of speech region recognition by the target VAD algorithm.

[0057] It can be understood that, since the cutoff rate indicator, the matching rate indicator, the time matching indicator, and the correctness rate indicator can represent differences between the recognition result of the target VAD algorithm and the manual annotation result in different dimensions, by calculating the indicator values of the plurality of indicators among the cutoff rate indicator, the matching rate indicator, the time matching indicator, and the correctness rate indicator corresponding to the target VAD algorithm, the manual annotation result of the test audio can be used as a reference of the recognition result, and differences between the recognition result and the manual annotation result of the test audio can be more comprehensively analyzed.

[0058] It should be emphasized that the plurality of specified test indicators described above are only exemplary descriptions and should not constitute a limitation on the embodiments of the present disclosure.

[0059] S203, obtaining reference indicator values of the plurality of specified test indicators determined for a specified test requirement;

[0060] Exemplarily, the specified test requirement can be a test requirement for performance comparison with a historical version of the target VAD algorithm, a test requirement for performance comparison with other competitive VAD algorithms, and the like. It can be understood that, when analyzing the performance of the target VAD algorithm, the reference indicator values of the specified test indicators can be obtained according to the specified test requirement of the target VAD algorithm, so that the test results of the target VAD algorithm under the specified test requirement can be determined by comparing the indicator values of the specified test indicators with the reference indicator values.

[0061] For example, if the specified test requirement is a test requirement of performance comparison with a historical version of the target VAD algorithm, the index values of the specified test indexes corresponding to the historical version of the target VAD algorithm can be obtained, i.e., the test audio is subjected to speech region recognition by using the historical version of the target VAD algorithm, and the index values of the specified test indexes corresponding to the historical version are calculated based on the recognition result corresponding to the historical version and the artificial labeling result, as the reference index values of the specified test indexes.

[0062] For the sake of clearness of the scheme and layout, the exemplary content about the specified test requirement and the specific content of the reference index values of the specified test indexes under different specified test requirements are described in combination with another embodiment.

[0063] In S204, the test result of the target VAD algorithm under the specified test requirement is determined based on the index values of the specified test indexes corresponding to the target VAD algorithm and the obtained reference index values.

[0064] In this embodiment, the test result of the target VAD algorithm under the specified test requirement can be determined by comparing the index values of the specified test indexes corresponding to the target VAD algorithm with the reference index values of the specified test indexes, and then determining the test result of the target VAD algorithm under the specified test requirement according to the obtained comparison results corresponding to each specified test index. For example, if the specified test indexes are the time matching index and the accuracy index, and the specified test requirement is a test requirement of performance comparison with a historical version of the target VAD algorithm, the comparison result corresponding to the time matching index can be obtained by comparing the index value of the time matching index with the reference index value of the time matching index, and the comparison result corresponding to the accuracy index can be obtained by comparing the index value of the accuracy index with the reference index value of the accuracy index. Then, the test result of the target VAD algorithm under the test requirement is determined according to the comparison result of the time matching index and the comparison result of the accuracy index.

[0065] For example, in actual application, according to the comparison results of the respective specified test indicators, the manner of determining the test result of the target VAD algorithm under the specified test requirement can be: scoring the target VAD algorithm according to the result content represented by the comparison results of the respective specified test indicators. For example, for an indicator whose performance is better with a larger indicator value, if the comparison result of the indicator represents that the accuracy indicator value of the target VAD algorithm is greater than the reference indicator value, a preset score can be given to the target VAD algorithm, otherwise, no score is given. For an indicator whose performance is better with a smaller indicator value, if the comparison result of the indicator represents that the accuracy indicator value of the target VAD algorithm is less than the reference indicator value, a preset score can be given to the target VAD algorithm, otherwise, no score is given. Then, whether the sum of the scores determined according to the respective comparison results exceeds the preset score threshold under the specified test requirement is determined to determine the test result of the target VAD algorithm under the specified test requirement.

[0066] In addition, for different test requirements, the type of test result can be different. Optionally, if the specified test requirement is a test requirement for performance comparison with a historical version of the target VAD algorithm, the test result can be a result representing that the performance of the target VAD algorithm is better than, or worse than, the historical version of the target VAD algorithm. If the specified test requirement is a test requirement for testing whether the target VAD algorithm meets the performance standard, the test result can be a test result representing that the target VAD algorithm meets the performance standard, or does not meet the performance standard. For example, if the specified test requirement is a test requirement for performance comparison with a historical version of the target VAD algorithm, and the score of the target VAD algorithm according to the comparison results of the respective specified test indicators is higher than the preset score threshold under the test requirement, the test result of the target VAD algorithm can be that the performance of the target VAD algorithm is better than the historical version of the target VAD algorithm.

[0067] In the scheme provided by the present disclosure, since the plurality of specified test indicators can be used to reflect the differences between the recognition result and the manual annotation result in multiple dimensions, the indicator values of the plurality of specified test indicators of the target VAD algorithm can be calculated. In addition, the indicator values of the plurality of specified test indicators are analyzed in combination with the specified test requirement, that is, according to different test requirements, the indicator values of the plurality of specified test indicators are compared with the reference indicator values of the plurality of specified test indicators under the test requirement, so as to determine the test result of the target VAD algorithm under the test requirement. In this way, the performance of the VAD algorithm under the test requirement can be reflected from multiple dimensions. It can be seen that, through the scheme, the performance of the target VAD algorithm under various test requirements can be more comprehensively evaluated.

[0068] Optionally, in another embodiment of the present disclosure, the step of calculating the index values of the plurality of specified test indexes corresponding to the target VAD algorithm by using the recognition result and the manual annotation result of the test audio can include:

[0069] calculating the index values of the plurality of specified test indexes corresponding to the target VAD algorithm according to the index value calculation manners corresponding to the plurality of specified test indexes respectively by using the recognition result and the manual annotation result of the test audio;

[0070] In the embodiment, the index values of the plurality of specified test indexes corresponding to the target VAD algorithm are calculated according to the index value calculation manners corresponding to each specified test index, and the plurality of specified test indexes can include a plurality of indexes in the cut-off rate index, the matching rate index, the time matching index, and the accuracy rate index.

[0071] The index value calculation manner of the cut-off rate index includes:

[0072] calculating the index value of the cut-off rate index based on the ratio of the number of the specified endpoints in the recognition result to the number of the specified endpoints in the manual annotation result;

[0073] The specified endpoint can be a start endpoint or an end endpoint. For example, if the number of start endpoints in the recognition result is 19 and the number of start endpoints in the manual annotation result is 17, the index value of the cut-off rate index is 19 / 17. It can be understood that the greater the index value of the cut-off rate index, the finer the granularity of the target VAD algorithm in recognizing the speech region, that is, the finer the segmentation of the target VAD algorithm to the speech region in the test audio.

[0074] The index value calculation manner of the matching rate index includes:

[0075] calculating the index value of the matching rate index based on the ratio of the difference between the total number of the specified endpoints in the recognition result and the total number of the specified endpoints in the manual annotation result to the total number of the specified endpoints in the manual annotation result;

[0076] It can be understood that since the matching rate index is used to represent the matching degree of the specified endpoints in the recognition result and the manual annotation result in terms of quantity, the index value of the matching rate index can be calculated based on the ratio of the difference between the total number of the specified endpoints in the recognition result and the total number of the specified endpoints in the manual annotation result to the total number of the specified endpoints in the manual annotation result, and the difference can be an absolute difference. Optionally, the calculation formula of the index value of the matching rate index can be: matching rate = 1- |total number of specified endpoints in recognition result - total number of specified endpoints in manual annotation result| / total number of specified endpoints in manual annotation result.

[0077] For example, if the number of start endpoints in the recognition result is 19 and the number of start endpoints in the manual annotation result is 17, the difference is 2, and the index value of the matching rate index is 15 / 17.

[0078] The index value of the time matching index is calculated based on the time difference between the specified endpoints in the target endpoint pair, wherein the target endpoint pair includes a specified endpoint in the manual annotation result and a specified endpoint in the recognition result that meets the first preset condition, and the first preset condition is a matching condition representing that the specified endpoint in the manual annotation result meets the minimum time difference.

[0079] The index value of the time matching index is calculated based on the time difference between the specified endpoints in the target endpoint pair, wherein the target endpoint pair includes a specified endpoint in the manual annotation result and a specified endpoint in the recognition result that meets the first preset condition, and the first preset condition is a matching condition representing that the specified endpoint in the manual annotation result meets the minimum time difference.

[0080] It can be understood that since the time matching index is used to represent the difference in time between the specified endpoints in the recognition result and the manual annotation result, the index value of the time matching index can be obtained by calculating the sum of the time differences between the specified endpoints identified for each speech region. Alternatively, in actual applications, the specified endpoints in the recognition result and the manual annotation result can be aligned first, i.e., using the matching method with the minimum time difference to match the specified endpoints as a target endpoint pair. For example, taking each specified endpoint in the manual annotation result as a standard, for each specified endpoint in the manual annotation result, a specified endpoint with the minimum time difference from the specified endpoint in the recognition result is selected as a target endpoint pair. For example, as shown in FIG. 1, s1 and s1' are a target endpoint pair, and e1 and e2' are a target endpoint pair. Then, the index value of the time matching index is obtained by calculating the sum of the time differences between the specified endpoints in each target endpoint pair. For example, the time difference can be a variance or an absolute difference, which are both reasonable. FIG. 4B

[0081] The index value of the time matching index is calculated based on the time difference between the specified endpoints in the target endpoint pair, wherein the target endpoint pair includes a specified endpoint in the manual annotation result and a specified endpoint in the recognition result that meets the first preset condition, and the first preset condition is a matching condition representing that the specified endpoint in the manual annotation result meets the minimum time difference.

[0082] The index value of the time matching index is calculated based on the time difference between the specified endpoints in the target endpoint pair, wherein the target endpoint pair includes a specified endpoint in the manual annotation result and a specified endpoint in the recognition result that meets the first preset condition, and the first preset condition is a matching condition representing that the specified endpoint in the manual annotation result meets the minimum time difference.

[0083] The second preset condition represents a condition for identifying errors.

[0084] ​It can be understood that in actual application, since the specified endpoints in the recognition result have insertion errors, inaccurate insertion positions or lack of insertion compared with the specified endpoints in the manual annotation result, the condition of insertion errors, inaccurate insertion positions or lack of insertion can be taken as the second preset condition. That is, the second preset condition representing the recognition error can include: a condition representing that the target VAD algorithm performs splitting at a position that should not be split; or a condition representing that the target VAD algorithm does not perform splitting at a position that should be split; or a condition representing that the position of the endpoint recognized by the target VAD algorithm is inaccurate.

[0085] Optionally, in an implementation manner, the determination manner of the number of the specified endpoints satisfying the second preset condition can include steps A1-A2:

[0086] A1, identify the number of target endpoint pairs satisfying a first sub-condition and the number of target endpoint pairs satisfying a second sub-condition; wherein the first sub-condition is that the sequence number of the specified endpoint belonging to the recognition result in the current target endpoint to be analyzed is not adjacent to the sequence number of the specified endpoint belonging to the recognition result in the previous target endpoint pair in endpoint naming; and the second sub-condition is that the time difference between the specified endpoints in the current target endpoint pair to be analyzed exceeds a preset threshold value;

[0087] In the implementation manner, if the sequence number of the specified endpoint belonging to the recognition result in each target endpoint pair is not adjacent to the sequence number of the specified endpoint belonging to the recognition result in the previous target endpoint pair in endpoint naming, there is a specified endpoint in the recognition result that splits at a position that should not be split; wherein the previous target endpoint pair is the previous target endpoint pair in the time dimension. If the time difference between the specified endpoints in the current target endpoint pair to be analyzed exceeds the preset threshold value, that is, the time difference between the specified endpoint belonging to the recognition result and the specified endpoint belonging to the manual annotation result in the target endpoint pair exceeds the preset threshold value, it can be considered that the specified endpoint belonging to the recognition result in the target endpoint pair is an endpoint with inaccurate insertion position. Exemplarily, the preset threshold value can be 50 ms, 100 ms, etc.

[0088] A2, calculate the sum of the identified number and a specified number to obtain the number of the specified endpoints satisfying the second preset condition; wherein the specified number is the number of the specified endpoints in the recognition result that are reduced relative to the manual annotation result.

[0089] In the present implementation, the specified number is the number of specified endpoints reduced in the recognition result relative to the manual annotation result, i.e., the number of specified endpoints missing insertion. It can be understood that after the number of target endpoint pairs meeting the first sub-condition and the number of target endpoint pairs meeting the second sub-condition are identified through step A1, the number of specified endpoints meeting the second preset condition can be obtained by calculating the sum of the identified number and the specified number.

[0090] It can be understood that after the number of specified endpoints meeting the second preset condition is obtained, the difference between the total number of specified endpoints in the recognition result and the number of specified endpoints meeting the second preset condition can be calculated, and the index value of the correctness index can be determined based on the ratio of the difference to the total number of specified endpoints in the manual annotation result. For example, if the total number of specified endpoints in the recognition result is 17 and the number of specified endpoints meeting the second preset condition is 11, the index value of the correctness is 6 / 17.

[0091] It can be seen that through the present solution, the differences between the recognition result and the manual annotation result are calculated from multiple dimensions, and the performance of the target VAD algorithm can be reflected from multiple dimensions, thereby improving the comprehensiveness of the test performance.

[0092] Optionally, in another embodiment of the present disclosure, the specified test requirement includes a first type of test requirement, and / or a second type of test requirement.

[0093] The first type of test requirement is a test requirement for performance comparison with a specified algorithm object, and the specified algorithm object is a historical version of the target VAD algorithm or another VAD algorithm.

[0094] The second type of test requirement is a test requirement for testing whether the target VAD algorithm meets a performance standard.

[0095] It can be understood that in actual applications, the test requirement for testing the target VAD algorithm can be a test requirement for comparing with other algorithms to analyze whether the algorithm performance of the target VAD algorithm is better than that of other algorithms, or a test requirement for analyzing whether the performance of the target VAD algorithm meets a performance standard according to the index values of the specified test indicators of the target VAD algorithm. Therefore, the specified test requirement can include a first type of test requirement and / or a second type of test requirement.

[0096] Correspondingly, in the present embodiment, the reference index values of the plurality of specified test indicators determined for the first type of test requirement include the index values of the plurality of specified test indicators corresponding to the specified algorithm object.

[0097] The reference indicator value of the plurality of specified test indicators determined for the second type of test requirement includes:

[0098] The indicator threshold value of the plurality of specified test indicators set based on the performance standard.

[0099] In this embodiment, if the specified test requirement is the first type of test requirement, the indicator values of the plurality of specified test indicators corresponding to the specified algorithm object are calculated based on the recognition result of the test audio recognized by the specified algorithm object and the artificial annotation result, as the reference indicator values of the plurality of specified test indicators determined for the first type of test requirement. If the specified test requirement is the second type of test requirement, the indicator threshold values of the plurality of specified test indicators set in advance for the performance standard of the target VAD algorithm can be used as the reference indicator values of the plurality of specified test indicators determined for the second type of test requirement.

[0100] In addition, it should be noted that if the specified test requirement is the first type of test requirement, the test result can be a result indicating that the performance of the target VAD algorithm is better than or worse than the specified algorithm object; if the specified test requirement is the second type of test requirement, the test result can be a test result indicating that the target VAD algorithm meets the performance standard or does not meet the performance standard.

[0101] For example, if the plurality of specified test indicators are the time matching indicator and the accuracy indicator, the specified test requirement is the first type of test requirement, the preset score corresponding to the time matching indicator is 30, the preset score corresponding to the accuracy indicator is 70, and the preset score threshold value under the first type of test requirement is 60, if the indicator value of the accuracy indicator is greater than the reference indicator value of the accuracy indicator, and the indicator value of the time matching indicator is less than the reference indicator value of the time matching indicator, the score of the VAD algorithm is 70, which is greater than the preset score threshold value under the first type of test requirement, and thus a test result indicating that the performance of the target VAD algorithm is better than the specified algorithm object can be obtained.

[0102] It can be seen that, by the present scheme, the indicator values of the plurality of specified test indicators are compared with the reference indicator values under different test requirements, and test results of the target VAD algorithm under different test requirements can be obtained, so that the target VAD algorithm can be tested more flexibly.

[0103] Optionally, in another embodiment of the present disclosure, as shown in FIG. 3 The step S204 of determining the test result of the target VAD algorithm under the specified test requirement based on the indicator values of the plurality of specified test indicators corresponding to the target VAD algorithm and the obtained reference indicator values can include steps S301-S304:

[0104] S301, determining a primary index and a secondary index from the plurality of specified test indexes based on an application scenario in which the target VAD algorithm is to be applied; wherein the primary index is more important than the secondary index in the application scenario;

[0105] For example, if the application scenario in which the target VAD algorithm is to be applied is a speech recognition scenario, and the speech recognition algorithm used in the speech recognition scenario has a certain bottleneck, for example, the length of the speech region to be recognized is required to be small, that is, the speech recognition algorithm cannot process a long speech region at a time, then the cut-off rate index in the plurality of specified test indexes of the target VAD algorithm is more important in the application scenario, and therefore, the cut-off rate index can be determined as the primary index, and the other specified test indexes can be determined as the secondary indexes. In addition, it should be noted that the number of the primary index or the secondary index can be one or more.

[0106] S302, comparing the index value of the primary index with the reference index value of the primary index to obtain a first comparison result;

[0107] For example, the first comparison result can be a result indicating that the index value of the primary index is greater than or less than the reference index value of the primary index.

[0108] S303, comparing the index value of the secondary index with the reference index value of the secondary index to obtain a second comparison result;

[0109] For example, the second comparison result can be a result indicating that the index value of the secondary index is greater than or less than the reference index value of the secondary index.

[0110] S304, scoring the target VAD algorithm based on the result content indicated by the first comparison result and the result content indicated by the second comparison result, and according to the weights set for the primary index and the secondary index, and determining a test result of the target VAD algorithm under the specified test requirement based on the score obtained by scoring.

[0111] In this embodiment, the result content indicated by the first comparison result can be that the index value of the primary index is greater than or less than the reference index value of the primary index, and the result content indicated by the second comparison result can be that the index value of the secondary index is greater than or less than the reference index value of the secondary index. It can be understood that, since the primary index is more important than the secondary index in the application scenario, when scoring, the target VAD algorithm can be scored according to the result content indicated by the first comparison result and the result content indicated by the second comparison result, and according to the weights set for the primary index and the secondary index.

[0112] Optionally, in actual application, the score of the target VAD algorithm in the main index and the secondary index dimensions can be determined according to the result content represented by the first comparison result and the result content represented by the second comparison result, respectively. Then, the scores in the main index and the secondary index dimensions are weighted and summed according to the weights set for the main index and the secondary index, to obtain the scoring score of the target VAD algorithm. Finally, according to the scoring score and the preset score threshold under the specified test requirement, the test result of the target VAD algorithm under the specified test requirement is determined.

[0113] For example, if the main index is the accuracy index, the secondary index is the time matching index, the greater the index values of the main index and the secondary index represent the higher performance of the algorithm, and the weights set for the main index and the secondary index are 0.7 and 0.3 respectively, and the preset score threshold under the specified test requirement is 60; if the first comparison result represents that the index value of the main index is greater than the reference index value of the main index, and the second comparison result represents that the index value of the secondary index is less than the reference index value of the secondary index, then the main index can be scored 100 points, and the secondary index can be scored 30 points. By calculating the weighted sum of the scores corresponding to the main index and the secondary index, the score of scoring the target VAD algorithm is 79 points, which is greater than the preset score threshold under the specified test requirement, so it can be determined that the test result of the target VAD algorithm under the specified test requirement. If the specified test requirement is the first type of test requirement, the test result is the result representing that the performance of the target VAD algorithm is better than the specified algorithm object; if the specified test requirement is the second type of test requirement, the test result is the test result representing that the target VAD algorithm reaches the performance standard.

[0114] It can be seen that, by the scheme, the test result of the target VAD algorithm under the specified test requirement is determined according to the application scene to which the target VAD algorithm is to be applied, so that the test result can more comprehensively reflect the performance of the target VAD algorithm in the application scene to be applied, so that in subsequent use of VAD algorithms, the VAD algorithm with better performance in the application scene can be selected according to the application scene.

[0115] In order to better understand the content of the embodiments of the present disclosure, a specific example is described below.

[0116] The present example uses the cutoff rate index, the matching rate index, the time matching index and the accuracy index to test the VAD algorithm to evaluate and compare the performance of different VAD algorithms. As shown in FIG. 1, the VAD algorithm is tested under the first type of test requirement and the second type of test requirement. FIG. 4AAs shown, the artificial labeling result of the starting point and the ending point of each speech region of a test audio is shown, wherein "s" is the position of the starting point of the artificial labeling, and "e" is the position of the ending point of the artificial labeling. The speech region recognition of the test audio is performed by using the target VAD algorithm, and the recognition result is as shown in FIG. 3. FIG. 4B As shown, "s'" is the position of the starting point recognized by the VAD algorithm, and "e'" is the position of the ending point recognized by the VAD algorithm.

[0117] The plurality of specified test indexes corresponding to the target VAD algorithm include a cutoff rate index, a matching rate index, a time matching index, and an accuracy rate index, and the calculation formulas of the indexes are as follows.

[0118] (1) Cutoff rate index

[0119] Cutoff rate = actual cutoff times / labeled cutoff times, wherein:

[0120] actual cutoff times = the number of pairs of starting and ending points actually returned by the target VAD algorithm;

[0121] labeled cutoff times = the number of pairs of starting and ending points in the artificial labeling result;

[0122] The actual cutoff times can be greater than the labeled cutoff times, or can be less than the labeled cutoff times.

[0123] (2) Matching rate index

[0124] Matching rate = 1 - | actual cutoff times - labeled cutoff times | / labeled cutoff times;

[0125] For example, if there are 17 labeled segments in the artificial labeling result, i.e., the labeled cutoff times is 17, and there are 19 labeled segments in the recognition result, then the matching rate is 15 / 17.

[0126] (3) Time matching index

[0127] For an audio file, if each specified endpoint in the artificial labeling result is one-to-one aligned with each specified endpoint returned by the target VAD algorithm, then the alignment rate is 100%, and at this time, the time matching index can be referred to as the time matching index. FIG. 1A That is, the recognition result is completely coincident with the artificial labeling result, at this time, the index value of the time matching index is 0, indicating complete matching.

[0128] For example, FIG. 4BAs shown, if the recognition result does not completely coincide with each specified endpoint in the manual annotation result, each specified endpoint can be aligned first to obtain each target endpoint pair. That is, taking the time of each specified endpoint annotated manually as a reference, the specified endpoint with the smallest time difference from the each specified endpoint annotated manually is selected from the recognition result to form each target endpoint pair. For example, FIG. 4B s1 and s1' in s1 and s1' are a target endpoint pair, and s2 and s3' are a target endpoint pair. At this time, the calculation formula of the index value of the time matching index can be the sum of the variances between each target endpoint pair in time, for example, the index value of the time matching index = (s1-s1') 2 +(s2-s3') 2 +…, the smaller the index value, the better the effect of the VAD algorithm.

[0129] (4) Accuracy index

[0130] As shown in Table 1 below, the alignment of the specified endpoints in the manual annotation result of a test audio and the specified endpoints in the recognition result returned by the VAD algorithm is shown. Among them, the annotation sequence number is the sequence number of the specified endpoint in the manual annotation result, the recognition sequence number is the sequence number of the specified endpoint in the recognition result, the same column corresponds to the aligned sequence number of the mark sequence number and the recognition sequence number, and the specified endpoint pair corresponding to the aligned sequence number is a target endpoint pair; the annotation dot time is the dot time corresponding to the mark sequence number, the recognition dot time is the dot time corresponding to the recognition sequence number; the threshold matching is the result of whether the time difference between the target endpoint pair exceeds the preset threshold, if not, the threshold matching result is TRUE, and the preset threshold is 0.5s.

[0131] Table 1

[0132]

[0133] From the above Table 1, it can be analyzed that the number of target endpoint pairs that successfully match the threshold is 8, and the number of target endpoint pairs that do not match successfully is 9. In addition, the number of insertion errors (corresponding to the target endpoint pairs meeting the first sub-condition in the above) is 2, indicating that the VAD algorithm has divided 2 segments between the annotation sequence numbers 10 to 11. The number of deletion errors (corresponding to the specified number in the above) is 0, i.e. there is no less division in the above table. The number of replacement errors (corresponding to the target endpoint pairs meeting the second sub-condition in the above) is 9, i.e. the number of target endpoint pairs that do not match successfully is 9. Therefore, the accuracy is: (17-(2+9)) / 17=35.3%.

[0134] It can be seen that, by the scheme, the performance of the target VAD algorithm can be more comprehensively evaluated by calculating the index values of multiple test indexes; by evaluating and comparing the performances of different VAD algorithms based on a unified standard, the algorithm with the best performance can be found, thereby providing more accurate and stable VAD services for products; the time and cost in the manual evaluation process can be reduced, the evaluation efficiency is improved, and the R&D team can optimize and improve the VAD algorithm more quickly; by applying the test method of the present application, the product can achieve better performance in the VAD technology, thereby standing out in the market competition and improving the competitiveness of the product in the field of speech recognition, speech signal processing, etc.

[0135] Based on the above-mentioned embodiments of the test method, the embodiments of the present disclosure also provide a test device, as shown in FIG. 5 The device comprises:

[0136] The identification module 510 is configured to perform speech region identification on the test audio based on the target VAD algorithm to be tested, and obtain an identification result; wherein the identification result represents the start and end points of each identified speech region.

[0137] The calculation module 520 is configured to calculate the index values of multiple specified test indexes corresponding to the target VAD algorithm by using the identification result and the manual annotation result of the test audio; wherein the manual annotation result represents the start and end points of each annotated speech region.

[0138] The acquisition module 530 is configured to obtain reference index values of the multiple specified test indexes determined for a specified test requirement.

[0139] The determination module 540 is configured to determine a test result of the target VAD algorithm under the specified test requirement based on the index values of the multiple specified test indexes corresponding to the target VAD algorithm and the obtained reference index values.

[0140] Optionally, the multiple specified test indexes include multiple indexes among a cutoff rate index, a matching rate index, a time matching index, and an accuracy rate index.

[0141] The cutoff rate index is used to represent the granularity of speech region identification performed by the target VAD algorithm.

[0142] The matching rate index is used to represent the matching degree of the specified endpoints in the identification result and the manual annotation result in quantity, and the specified endpoints are start or end points.

[0143] The time matching index is used to represent the time difference of the specified endpoints in the identification result and the manual annotation result.

[0144] The correctness rate index is used to represent the accuracy of the target VAD algorithm in speech region recognition.

[0145] Optionally, the computing module is specifically configured to:

[0146] According to the index value calculation manner corresponding to each of the plurality of specified test indexes, the index values of the plurality of specified test indexes corresponding to the target VAD algorithm are calculated by using the recognition result and the manual annotation result of the test audio.

[0147] The index value calculation manner of the cutoff rate index comprises:

[0148] The index value of the cutoff rate index is calculated based on the ratio of the number of the specified endpoints in the recognition result to the number of the specified endpoints in the manual annotation result.

[0149] The index value calculation manner of the matching rate index comprises:

[0150] The index value of the matching rate index is calculated based on the ratio of the difference between the total number of the specified endpoints in the recognition result and the total number of the specified endpoints in the manual annotation result to the total number of the specified endpoints in the manual annotation result.

[0151] The index value calculation manner of the time matching index comprises:

[0152] The index value of the time matching index is calculated based on the time difference between the specified endpoints in the target endpoint pair, wherein the target endpoint pair contains a specified endpoint in the manual annotation result and a specified endpoint in the recognition result that meets a first preset condition, and the first preset condition represents a time difference minimum matching condition with the specified endpoint in the manual annotation result.

[0153] The index value calculation manner of the correctness rate index comprises:

[0154] The difference between the total number of the specified endpoints in the recognition result and the number of the specified endpoints that meet a second preset condition is calculated, and the index value of the correctness rate index is determined based on the ratio of the difference to the total number of the specified endpoints in the manual annotation result.

[0155] The second preset condition represents a recognition error condition.

[0156] Optionally, the number of the specified endpoints that meet the second preset condition comprises:

[0157] The number of target endpoint pairs that meet a first sub-condition and the number of target endpoint pairs that meet a second sub-condition are identified.

[0158] After the identified quantity and the specified quantity are calculated, a quantity of the specified endpoints satisfying the second preset condition is obtained;

[0159] The first sub-condition is that a serial number of the specified endpoint belonging to the recognition result in the current target endpoint pair to be analyzed is not adjacent to a serial number of the specified endpoint belonging to the recognition result in the previous target endpoint pair in endpoint naming;

[0160] The second sub-condition is that a time difference between the specified endpoints in the current target endpoint pair to be analyzed exceeds a preset threshold;

[0161] The specified quantity is a quantity of the specified endpoints in the recognition result reduced relative to a manual annotation result.

[0162] Optionally, the specified test requirement includes a first type of test requirement, and / or a second type of test requirement;

[0163] The first type of test requirement is a test requirement for performance comparison with a specified algorithm object, and the specified algorithm object is a historical version of the target VAD algorithm or another VAD algorithm.

[0164] The second type of test requirement is a test requirement for testing whether the target VAD algorithm meets a performance standard.

[0165] Optionally, the reference indicator value of the plurality of specified test indicators determined for the first type of test requirement includes an indicator value of the plurality of specified test indicators corresponding to the specified algorithm object.

[0166] The reference indicator value of the plurality of specified test indicators determined for the second type of test requirement includes:

[0167] An indicator threshold value of the plurality of specified test indicators set based on the performance standard.

[0168] Optionally, the determination module includes:

[0169] A determination sub-module, configured to determine a primary indicator and a secondary indicator from the plurality of specified test indicators based on an application scenario to which the target VAD algorithm is to be applied, wherein the primary indicator is more important than the secondary indicator in the application scenario.

[0170] A first comparison sub-module, configured to compare a size of an indicator value of the primary indicator with a reference indicator value of the primary indicator to obtain a first comparison result.

[0171] A second comparison sub-module, configured to compare a size of an indicator value of the secondary indicator with a reference indicator value of the secondary indicator to obtain a second comparison result.

[0172] A scoring module is configured to score the target VAD algorithm based on the result content characterized by the first comparison result and the result content characterized by the second comparison result, and according to the weights set for the primary index and the secondary index, and determine the test result of the target VAD algorithm under the specified test requirement based on the score obtained by the scoring.

[0173] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0174] According to embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium and a computer program product.

[0175] The electronic device provided by the present disclosure can comprise:

[0176] at least one processor; and

[0177] The memory is in communication connection with the at least one processor, and the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the steps of any of the test methods described above.

[0178] The computer readable storage medium provided by the present disclosure stores a computer program, and the computer program is executed by a processor to implement the steps of any of the test methods described above.

[0179] In another embodiment provided by the present disclosure, a computer program product containing instructions is also provided, which, when executed on a computer, causes the computer to execute the steps of any of the test methods in the above embodiments.

[0180] FIG. 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present disclosure described and / or claimed in this document.

[0181] As FIG. 6As shown, the device 600 includes a computing unit 601 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from the storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0182] Various components in the device 600 are connected to the I / O interface 605, including an input unit 606 such as a keyboard, a mouse, etc., an output unit 607 such as various types of displays, speakers, etc., a storage unit 608 such as a magnetic disk, an optical disk, etc., and a communication unit 609 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0183] The computing unit 601 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the test method. For example, in some embodiments, the test method can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the test method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the test method by other any appropriate means, such as by means of firmware.

[0184] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0185] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, or entirely on a remote machine or server.

[0186] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0187] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0188] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0189] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server can arise by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0190] It should be understood that various forms of flow shown above can be used, with steps reordered, added, or removed. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, without limitation herein, so long as the desired results of the technology disclosed in the present disclosure are achieved.

[0191] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the spirit and principles of the disclosure. Accordingly, the disclosure is not limited to the specific embodiments described above.

Claims

1. A test method, comprising: performing speech region recognition on test audio based on a target VAD algorithm to be tested to obtain a recognition result, wherein the recognition result represents start and end endpoints of each speech region identified; calculating, using the recognition result and a manual annotation result of the test audio, an index value of a plurality of specified test indexes corresponding to the target VAD algorithm; wherein the manual annotation result represents start and end endpoints of each speech region annotated; obtaining reference index values of the plurality of specified test indexes determined for a specified test requirement; determining a test result of the target VAD algorithm under the specified test requirement based on the index values of the plurality of specified test indexes corresponding to the target VAD algorithm and the obtained reference index values; wherein the plurality of specified test indexes include a plurality of indexes among a cutoff rate index, a matching rate index, a time matching index, and an accuracy rate index; wherein the cutoff rate index is used to represent a granularity of speech region recognition performed by the target VAD algorithm; the matching rate index is used to represent a matching degree in quantity between the recognition result and specified endpoints in the manual annotation result, the specified endpoints being start endpoints or end endpoints; the time matching index is used to represent a time difference between the recognition result and the specified endpoints in the manual annotation result; the accuracy rate index is used to represent an accuracy of speech region recognition performed by the target VAD algorithm.

2. The method of claim 1, wherein, The calculating, using the recognition result and the manual annotation result of the test audio, of the index value of the plurality of specified test indexes corresponding to the target VAD algorithm includes: calculating, using the recognition result and the manual annotation result of the test audio, the index value of the plurality of specified test indexes corresponding to the target VAD algorithm according to an index value calculation manner corresponding to each of the plurality of specified test indexes; wherein the index value calculation manner corresponding to the cutoff rate index includes: calculating the index value of the cutoff rate index based on a ratio of the number of the specified endpoints in the recognition result to the number of the specified endpoints in the manual annotation result; the index value calculation manner corresponding to the matching rate index includes: calculating the index value of the matching rate index based on a difference between a total number of the specified endpoints in the recognition result and a total number of the specified endpoints in the manual annotation result, and a ratio of the total number of the specified endpoints in the manual annotation result; the index value calculation manner corresponding to the time matching index includes: calculating the index value of the time matching index based on a time difference between specified endpoints in a target endpoint pair, wherein the target endpoint pair includes a specified endpoint in the manual annotation result and a specified endpoint in the recognition result that meets a first preset condition representing a time difference minimum matching condition with the specified endpoint in the manual annotation result; the index value calculation manner corresponding to the accuracy rate index includes: calculating a difference between the total number of the specified endpoints in the recognition result and the number of the specified endpoints satisfying the second preset condition, determining the index value of the correctness rate index based on a ratio of the difference to the total number of the specified endpoints in the artificial labeling result; wherein the second preset condition is a condition representing a recognition error.

3. The method of claim 2, wherein, The determination manner of the number of the specified endpoints satisfying the second preset condition comprises: identifying the number of target endpoint pairs meeting a first sub-condition and the number of target endpoint pairs meeting a second sub-condition; calculating the identified number and a specified number to obtain the number of the specified endpoints satisfying the second preset condition; The first sub-condition is that the serial number of the specified endpoint belonging to the recognition result in the current target endpoint to be analyzed is not adjacent to the serial number of the specified endpoint belonging to the recognition result in the previous target endpoint pair in endpoint naming; The second sub-condition is that the time difference between the specified endpoints in the current target endpoint pair to be analyzed exceeds a preset threshold; The specified number is the number of the specified endpoints reduced in the recognition result relative to the artificial labeling result.

4. The method of claim 1, wherein, The specified test requirements include first type test requirements and / or second type test requirements; wherein the first type test requirements are test requirements for performance comparison with a specified algorithm object, and the specified algorithm object is a historical version of the target VAD algorithm or another VAD algorithm; The second type test requirements are test requirements for testing whether the target VAD algorithm meets a performance standard.

5. The method of claim 4, wherein, The reference index values of the plurality of specified test indexes determined for the first type test requirements include index values of the plurality of specified test indexes corresponding to the specified algorithm object; The reference index values of the plurality of specified test indexes determined for the second type test requirements include: index thresholds of the plurality of specified test indexes set based on the performance standard.

6. The method of claim 1, wherein, The determination of the test result of the target VAD algorithm under the specified test requirements based on the index values of the plurality of specified test indexes corresponding to the target VAD algorithm and the obtained reference index values comprises: determining a main index and a secondary index from the plurality of specified test indexes based on an application scenario to which the target VAD algorithm is to be applied, wherein the main index is more important than the secondary index in the application scenario; comparing the index value of the main index with the reference index value of the main index to obtain a first comparison result; comparing the index value of the secondary index with the reference index value of the secondary index to obtain a second comparison result; based on the result content represented by the first comparison result and the result content represented by the second comparison result, and according to the weights set for the main index and the secondary index, scoring the target VAD algorithm, and determining the test result of the target VAD algorithm under the specified test requirements based on the score.

7. A testing device, comprising: An identification module is configured to perform voice region identification on the test audio based on a target VAD algorithm to be tested, to obtain an identification result, wherein the identification result represents start and end points of each identified voice region. A calculation module is configured to calculate index values of a plurality of specified test indexes corresponding to the target VAD algorithm by using the identification result and a manual labeling result of the test audio, wherein the manual labeling result represents start and end points of each labeled voice region. An obtaining module is configured to obtain reference index values of the plurality of specified test indexes determined for a specified test requirement. A determination module is configured to determine a test result of the target VAD algorithm under the specified test requirement based on the index values of the plurality of specified test indexes corresponding to the target VAD algorithm and the obtained reference index values. The plurality of specified test indexes include a plurality of indexes from among a cutoff rate index, a matching rate index, a time matching index, and an accuracy rate index. The cutoff rate index is used to represent a granularity of voice region identification performed by the target VAD algorithm. The matching rate index is used to represent a matching degree in quantity between the identification result and specified end points in the manual labeling result, the specified end points being start or end points. The time matching index is used to represent a time difference between the identification result and the specified end points in the manual labeling result. The accuracy rate index is used to represent an accuracy of voice region identification performed by the target VAD algorithm.

8. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-6.

10. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-6.

Images