Voice transcription system evaluation method and device, related equipment and program product

Abstract

The invention discloses a voice transcription system evaluation method and device, related equipment and a program product, and relates to the technical field of voice recognition. When the voice transcription system is evaluated, the semantic consistency index is introduced, and the index can measure the semantic consistency between the first transcription text of the test audio and the reference text, namely, the degree of semantic expression meaning consistency of the first transcription text and the reference text is measured; according to the invention, the method can accurately measure the capability of the voice transliteration system for the semantic transmission of the test audio, further determines the evaluation result of the voice transliteration system at least according to the score of the semantic consistency index, and improves the objectivity of the evaluation result. Meanwhile, the semantic consistency evaluation index is more in line with the real reading feeling of human beings, so that the obtained evaluation result is more in line with the real feeling of human beings.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and more specifically, to a method, device, related equipment and program product for evaluating a speech transcription system. Background Art

[0002] Speech transcription, as an important case of artificial intelligence technology, is widely used in various fields. It is crucial to accurately evaluate the performance of a speech transcription system.

[0003] Currently, the mainstream speech transcription systems in the industry generally only consider the accuracy of character-for-character transcription of the speech transcription system. If the character-for-character transcription accuracy is high, it is considered that the transcription effect of the speech transcription system is good. However, in actual business scenarios, it is often the case that the character-for-character transcription accuracy is already very high, but there are still problems such as unclear meaning and deviation in understanding when people read the transcribed text. That is to say, the evaluation results of the current speech transcription system evaluation methods are not objective enough and inconsistent with the real feelings of humans. Summary of the Invention

[0004] In view of the above problems, this application is proposed to provide a method, device, related equipment and program product for evaluating a speech transcription system, so as to improve the objectivity of the evaluation results of the speech transcription system and the consistency with the real feelings of humans. The specific solutions are as follows:

[0005] In a first aspect, a method for evaluating a speech transcription system is provided, including:

[0006] Obtain a test audio and its corresponding reference text;

[0007] Obtain a first transcribed text of the test audio, where the first transcribed text is obtained by transcribing the test audio through a speech transcription system to be evaluated;

[0008] Determine the semantic consistency between the first transcribed text and the reference text to obtain a semantic consistency index score;

[0009] Determine the evaluation result of the speech transcription system according to the score of the set evaluation index, where the set evaluation index at least includes the semantic consistency index.

[0010] In a possible design, in another implementation manner of the first aspect of the embodiments of this application, it further includes:

[0011] Obtain a noise-added test audio, where the noise-added test audio is an audio obtained by adding noise to the test audio;

[0012] Obtain a second transcribed text of the noise-added test audio, where the second transcribed text is obtained by transcribing the noise-added test audio through the speech transcription system;

[0013] Based on the first transcription text, the second transcription text, and the reference text, determine a noise robustness index score, where the noise robustness index is used to measure the stability of the transcription text of the audio before and after adding noise;

[0014] The set evaluation metrics further include the noise robustness index.

[0015] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, it further includes:

[0016] According to the reference text and the first transcription text, calculate a recognition rate index score, where the recognition rate index represents the correct rate of character-to-sound transcription;

[0017] The set evaluation metrics further include the recognition rate index.

[0018] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the process of determining the semantic consistency between the first transcription text and the reference text to obtain a semantic consistency index score includes:

[0019] Call a large model to instruct the large model to evaluate the semantic consistency between the first transcription text and the reference text, and output a semantic consistency index score.

[0020] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the process of determining a noise robustness index score based on the first transcription text, the second transcription text, and the reference text includes:

[0021] Using the first transcription text as the source text and the second transcription text as the generated text, calculate the SARI index score based on the source text, the generated text, and the reference text as the noise robustness index score.

[0022] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the process of calculating a recognition rate index score according to the reference text and the first transcription text includes:

[0023] Perform edit distance alignment on the reference text and the first transcription text;

[0024] Based on the alignment result, calculate the correct recognition rate COR index score and the accuracy ACC index score;

[0025] Integrate the COR index score and the ACC index score to obtain the recognition rate index score.

[0026] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of integrating the COR metric score and the ACC metric score to obtain the recognition rate metric score includes:

[0027] Performing weighted addition on the COR metric score and the ACC metric score to obtain the recognition rate metric score;

[0028] When the test audio is from a short audio test set, the first weight corresponding to the COR metric is less than the second weight corresponding to the ACC metric, and the duration of the test audio included in the short audio test set is less than the first set duration threshold;

[0029] When the test audio is from a long audio test set, the first weight corresponding to the COR metric is greater than the second weight corresponding to the ACC metric, and the duration of the test audio included in the long audio test set is greater than the second set duration threshold.

[0030] In a second aspect, a speech-to-text system evaluation device is provided, including:

[0031] A test audio and reference text acquisition unit, configured to acquire a test audio and its corresponding reference text;

[0032] A first transcribed text acquisition unit, configured to acquire a first transcribed text of the test audio, where the first transcribed text is obtained by transcribing the test audio through a speech-to-text system to be evaluated;

[0033] A semantic consistency metric calculation unit, configured to determine the semantic consistency between the first transcribed text and the reference text to obtain a semantic consistency metric score;

[0034] An evaluation result determination unit, configured to determine the evaluation result of the speech-to-text system according to the scores of the set evaluation metrics, where the set evaluation metrics at least include the semantic consistency metric.

[0035] In a third aspect, an electronic device is provided, including: a memory and a processor;

[0036] The memory is configured to store a program;

[0037] The processor is configured to execute the program to implement each step of the speech-to-text system evaluation method described in any one of the foregoing first aspects of the present application.

[0038] In a fourth aspect, a readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, each step of the speech-to-text system evaluation method described in any one of the foregoing first aspects of the present application is implemented.

[0039] In a fifth aspect, a computer program product is provided, including a computer program which, when executed by a processor, implements each step of the speech transcription system evaluation method described in any one of the foregoing first aspects of the present application.

[0040] By means of the above technical solution, when evaluating a speech transcription system, the present application introduces a semantic consistency index, which can measure the semantic consistency between the first transcription text of a test audio and a reference text, that is, measure the degree to which the first transcription text and the reference text express the same meaning at the semantic level, and can accurately measure the ability of the speech transcription system to transmit the semantics of the test audio. Furthermore, at least based on the score of the semantic consistency index, the evaluation result of the speech transcription system is determined, improving the objectivity of the evaluation result. At the same time, the semantic consistency evaluation index is also more in line with the real reading feeling of humans, so the obtained evaluation result is also more in line with the real feeling of humans. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0042] Figure 1 A schematic diagram of the alignment of a reference text and several different transcription texts is provided;

[0043] Figure 2 A schematic diagram of the processing flow of a speech transcription system integrated with a speech transcription evaluation system is illustrated;

[0044] Figure 3 A schematic diagram of the flow of a speech transcription system evaluation method provided by an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the calculation process of an identification rate index is illustrated;

[0046] Figure 5 A schematic diagram of the edit distance alignment of a reference text and a first transcription text is provided;

[0047] Figure 6 A schematic diagram of the evaluation process of the human-machine consistency effect of a speech transcription evaluation system is illustrated;

[0048] Figure 7 A schematic diagram of the evaluation process of the human-machine consistency effect of another speech transcription evaluation system is illustrated;

[0049] Figure 8Schematic diagram of the structure of an evaluation device for a speech-to-text transcription system disclosed in an embodiment of the present application;

[0050] Figure 9 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Detailed implementation manners

[0051] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0052] Currently, the mainstream speech-to-text transcription systems in the industry generally use the correct rate of character-to-character transcription to measure the transcription effect. For example, the COR (Correct Recognition Rate) or ACC (Accuracy) indicators are used to reflect the quality of the speech-to-text transcription system. The higher the score of the COR indicator or ACC indicator, the higher the transcription effect of the speech-to-text transcription system is considered. The prior art does not consider whether the actual semantics of the transcribed text and the audio are consistent, nor does it consider the robustness of the transcription system in various complex noise scenarios. Therefore, a problem is often encountered that although the COR indicator and ACC indicator are already very high, whether reading the transcribed text or using the transcribed text for downstream tasks, problems such as information transmission deviation or downstream cascade task failure due to transcription errors always occur frequently.

[0053] The COR indicator strictly statistically analyzes the character consistency between the transcribed text and the reference text, ignoring character-to-character conversion errors such as insertions, deletions, and substitutions. This indicator is calculated by the formula (1):

[0054] COR = H / N × 100% (1)

[0055] Wherein, N represents the total number of characters in the reference text, and H represents the number of correctly transcribed characters.

[0056] The ACC indicator represents the overall proportion of the correct part in the transcribed text, taking into account the influence of insertions, deletions, and substitution errors. The specific calculation formulas are as shown in (2) and (3) below.

[0057] WER = (S + I + D) / N (2)

[0058] ACC = 1 - WER (3)

[0059] Refer to Figure 1, an example is provided. The transcription results of the same audio in three speech transcription systems are REC1, REC2, and REC3 respectively, and the alignment results with the reference text LAB are as Figure 1 shown.

[0060] The reference text of the audio has 15 characters. There are no insertion or deletion errors in all three transcription systems, and the substitution error is 1 for each. According to formulas (1), (2), and (3), the COR and ACC indicators of the three systems can be calculated to be 93.33%. According to the existing technology, it can be concluded that there is no difference in the quality of the three systems.

[0061] However, through the analysis of the transcription results of the three systems, it can be found that:

[0062] The error in REC1 is the transcription error of the characters in the entity words, which has the greatest impact on the conveyance of the correct semantics. The error in REC2 is the error of the modal particles, and there is basically no loss to the semantic conveyance in the current context. The error in REC3 is the transcription error of the verbs, which has a certain impact on the semantic transmission in the current context, but the degree of impact is less than that of the entity word error in REC1. Therefore, although the transcription accuracy indicators of the three systems are the same, from an objective perspective and combined with people's subjective feelings, the effect of REC2 is the best, REC3 is the second, and the effect of REC1 is the worst. Therefore, it is not robust enough to use only the COR and ACC indicators to evaluate the transcription system. It only focuses on the literal errors, including insertion, deletion, and substitution errors, and completely ignores the use of words in a specific context. In some cases, even if the literal errors are few, it may still cause inappropriate context due to local errors, resulting in information transmission errors, and there is a certain deviation from people's subjective feelings.

[0063] In addition, the type and intensity of the noise have a significant impact on the transcription accuracy of speech. A robust speech transcription system should keep the text transcribed from the same audio stable in various noise scenarios. The current evaluation method for speech transcription systems also does not consider the impact of noise on the speech transcription system.

[0064] In view of this, the present application provides a method for evaluating a speech transcription system, which can at least solve some of the defects existing in the prior art. The method for evaluating a speech transcription system of the present application can be applied to evaluate the transcription effect of speech transcription systems in various industries and scenarios.

[0065] The present application provides an evaluation system for a speech transcription system (hereinafter referred to as the speech transcription evaluation system), which can be deployed on a terminal or a server. Exemplarily, the speech transcription evaluation system can be deployed on the same device as the speech transcription system, or the two can be deployed on different devices.

[0066] Such as Figure 2, which exemplifies the processing flow of a speech transcription system integrated with a speech transcription evaluation system.

[0067] Transcribe the test audio through a speech transcription system (ASR system) to obtain the transcribed text of the test audio (defined as the first transcribed text).

[0068] In addition, obtain the reference text of the test audio, which is the standard text corresponding to the content of the test audio. The test audio and the reference text can be obtained from a publicly available dataset, or an expert can annotate the test audio to obtain the reference text. In addition, it can also be to perform speech synthesis on the reference text and use the synthesized speech as the test audio.

[0069] By sending the reference text and the first transcribed text into the speech transcription evaluation system, perform specific evaluation processes, such as determining the semantic consistency between the first transcribed text and the reference text, obtaining the semantic consistency index score, and obtaining the evaluation result of the speech transcription system based on this semantic consistency index score. Examples are in the forms of the evaluation score and evaluation level of the speech transcription system.

[0070] In some possible implementations, as Figure 2 shown in the right branch in, for the test audio, it is also possible to obtain the noise-added test audio through automatic noise addition processing. Further transcribe it through the ASR system to obtain the transcribed text of the noise-added test audio (defined as the second transcribed text). On this basis, the second transcribed text can also be sent into the speech transcription evaluation system for the evaluation system to calculate the noise robustness index score. Then, the evaluation result of the speech transcription system can be determined based on the scores of various set evaluation metrics such as the semantic consistency index score and the noise robustness index score.

[0071] Next, from the perspective of the speech transcription evaluation system, the embodiments of the present application provide a method for evaluating a speech transcription system. Referring to Figure 3 , this method for evaluating a speech transcription system specifically includes the following steps:

[0072] Step S100, obtain the test audio and its corresponding reference text.

[0073] Specifically, when testing the speech transcription system, the target scenario (field) to be tested can be specified, that is, this test task is: to test the transcription effect of the speech transcription system in the target scenario.

[0074] The test audio obtained in this step can be from an audio test set related to the target scenario. Exemplarily, if the target scenario is a human-computer interaction scenario, the test audio can be from a short audio test set related to the human-computer interaction scenario. Another example is that if the target scenario is a meeting scenario, the test audio can be from a long audio test set related to meeting transcription in the meeting scenario.

[0075] The reference text corresponding to the test audio is the standard text corresponding to the content of the test audio. The test audio and the reference text can be obtained from a publicly available dataset, or the reference text can be obtained by an expert annotating the test audio. In addition, it can also be that the reference text is synthesized into speech, and the synthesized speech is used as the test audio.

[0076] Step S110: Obtain the first transcription text of the test audio, where the first transcription text is obtained by transcribing the test audio through the speech transcription system to be evaluated.

[0077] Step S120: Determine the semantic consistency between the first transcription text and the reference text to obtain the semantic consistency index score.

[0078] In this embodiment, the semantic consistency index is used as an evaluation index for evaluating the effect of the speech transcription system. The semantic consistency index is used to measure the semantic consistency between the first transcription text and the reference text, that is, to measure the ability of the speech transcription system to transmit the semantics of the test audio.

[0079] The semantic consistency index score represents the degree of semantic consistency between the first transcription text and the reference text. The higher the semantic consistency index score, the higher the degree of semantic consistency between the first transcription text and the reference text, and the better the transcription effect of the speech transcription system.

[0080] Step S130: Determine the evaluation result of the speech transcription system according to the score of the set evaluation index, where the set evaluation index includes at least the semantic consistency index.

[0081] This application can pre-define one or more evaluation indexes for evaluating the speech transcription system, and at least the semantic consistency index is included in these evaluation indexes.

[0082] In this step, according to the score of the set evaluation index, the evaluation effect of the speech transcription system can be calculated. The evaluation effect can be reflected in various forms. For example, the evaluation score, evaluation level, etc. are used to represent the evaluation effect of the speech transcription system.

[0083] When evaluating a speech-to-text system, the speech-to-text system evaluation method provided by the embodiments of this application introduces a semantic consistency index. This index can measure the semantic consistency between the first transcribed text of the test audio and the reference text, that is, measure the degree to which the first transcribed text and the reference text express the same meaning at the semantic level, and can accurately measure the ability of the speech-to-text system to transmit the semantics of the test audio. Furthermore, at least based on the score of the semantic consistency index, the evaluation result of the speech-to-text system is determined, which improves the objectivity of the evaluation result. At the same time, the semantic consistency evaluation index is also more in line with the real reading experience of humans, so the obtained evaluation result is also more in line with the real feelings of humans.

[0084] In some possible implementations, the evaluation index of the speech-to-text system may only include the semantic consistency index, and then the calculated score of the semantic consistency index can be directly used as the evaluation score of the speech-to-text system.

[0085] In other possible implementations, the recognition rate index can be further added to the evaluation index of the speech-to-text system. This recognition rate index represents the correct rate of character-for-sound transcription, that is, the correct rate of transcribing the test audio into characters. On this basis, the method of this application may further include:

[0086] Calculate the score of the recognition rate index according to the reference text and the first transcribed text.

[0087] Then the evaluation score of the speech-to-text system can be comprehensively obtained based on the score of the recognition rate index and the score of the semantic consistency index. For example, the scores of the recognition rate index and the semantic consistency index are weighted and added, and the result is used as the evaluation score T of the speech-to-text system:

[0088] T = s1×wgt1 + s2×wgt2;

[0089] Wherein, s1 represents the score of the recognition rate index, s2 represents the score of the semantic consistency index, and wgt1 and wgt2 are two weight values, which can be flexibly adjusted according to the user's focus on the recognition rate index and the semantic consistency index, and the sum of the two weight values is equal to 1.

[0090] An optional calculation process for the score of the recognition rate index is introduced in this embodiment.

[0091] Specifically, the score of the recognition rate index can be determined based on the score of the correct recognition rate COR index and / or the score of the accuracy rate ACC index. For example, the score of the correct recognition rate COR index or the score of the accuracy rate ACC index can be used as the score of the recognition rate index. Or, the scores of the COR index and the ACC index can be integrated to obtain the score of the recognition rate index. The following formula exemplifies a calculation method for the score of the recognition rate index:

[0092] s1 = α × COR + β × ACC;

[0093] Where α and β represent two weight values, and their sum is equal to 1. COR and ACC respectively represent the scores of two metrics.

[0094] The COR metric and the ACC metric each have their own applicable scenarios.

[0095] The COR metric focuses on measuring the correct proportion at the character level, ignoring character-level phonetic transcription errors such as insertions, deletions, and substitutions, and can reflect local correctness. Therefore, it is more applicable to long audio, such as meeting record transcription texts, long speeches, or blog content generation, tasks that require statistical local accuracy (such as subtitle generation), etc.

[0096] The ACC metric is considered correct only when the transcribed text is exactly the same as the reference text, ensuring absolute accuracy of the text. Short audio usually corresponds to short texts (such as "Turn on the air conditioner", "Navigate to the company", etc.), with short sentences and key semantics. Minor errors may lead to serious consequences. For example, "Turn off the light" being mis-transcribed as "Turn on the light", even if only one character is replaced, may completely change the intention. Therefore, the ACC metric is more applicable to short audio, such as voice assistant commands (such as smart home control), key information extraction (such as phone numbers, verification codes), short sentence translation, or command execution, etc. The ACC metric better meets the "zero tolerance for errors" requirement of short audio.

[0097] According to the respective applicable scenarios of the COR metric and the ACC metric, in the method for evaluating the performance of a speech transcription system in this embodiment, different weights can be set for the COR metric and the ACC metric when testing the performance of the speech transcription system in different scenarios. For example:

[0098] When the test audio is from a short audio test set, the first weight α corresponding to the COR metric is less than the second weight β corresponding to the ACC metric;

[0099] When the test audio is from a long audio test set, the first weight α corresponding to the COR metric is greater than the second weight β corresponding to the ACC metric.

[0100] Wherein, the duration of the test audio included in the short audio test set is less than a first set duration threshold, and the duration of the test audio included in the long audio test set is greater than a second set duration threshold. The first set duration threshold and the second set duration threshold can be the same or different. When their values are different, the first set duration threshold is less than the second set duration threshold.

[0101] Combined with Figure 4 as shown, a calculation process of an identification rate metric is introduced, specifically including:

[0102] For the test audio, obtain its reference text and the first transcription text of the test audio obtained through an automatic speech recognition (ASR) system.

[0103] Furthermore, perform an edit distance alignment on the reference text and the first transcription text.

[0104] Calculate the COR metric score and the ACC metric score based on the alignment result.

[0105] Perform a weighted addition on the COR metric score and the ACC metric score to obtain the recognition rate score.

[0106] Combined Figure 5 , which provides an example of the edit distance alignment between the reference text LAB and the first transcription text REC.

[0107] It is counted that there are 16 characters in total in the reference text, i.e., N = 16. Among them, 13 characters are correctly transcribed, i.e., H = 13, the number of inserted characters is 2, i.e., I = 2, the number of substitution errors is 2, i.e., S = 2, and the number of deletion errors is 1, i.e., D = 1. Then, according to the aforementioned formulas (1), (2), and (3), COR can be calculated as 81.25% and ACC as 68.75%. Define α = 0.2 and β = 0.8, then the recognition rate metric score s1 = 71.25 is calculated.

[0108] In some embodiments of the present application, the determination process of the semantic consistency metric score is introduced.

[0109] In one possible implementation, a semantic consistency discriminant model can be pre-trained, and this model can adopt a deep neural network model structure. First, obtain the training data, where the training data includes text pairs, and these text pairs are labeled with semantic consistency scores. Use this training data to train the semantic consistency discriminant model to obtain the trained semantic consistency discriminant model. Furthermore, the semantic consistency discriminant model can be used to process the first transcription text and the reference text to output the semantic consistency metric score of the two.

[0110] In another possible implementation, this embodiment can call a large model and, by virtue of the natural language understanding and processing capabilities of the large model, output the semantic consistency metric score of the reference text and the first transcription text. That is to say, in this embodiment, a large model can be called to instruct the large model to evaluate the semantic consistency between the first transcription text and the reference text and output the semantic consistency metric score.

[0111] When calling a large model to evaluate the semantic consistency between the first transcription text and the reference text, the first transcription text and the reference text can be concatenated with a preset prompt to form a prompt, and this prompt is sent into the large model to obtain the score of the semantic consistency index output by the large model. In the prompt, the thinking mode of the large model can be specified to guide the large model to more accurately evaluate the score of the semantic consistency index. The following is an example of some content of the prompt:

[0112] We are doing a speech transcription task and need to evaluate the semantic change of the text after speech transcription relative to the reference text. The output is the degree of semantic change (an integer between 0 and 100, the higher the degree, the larger the value).

[0113] When judging whether the semantics has changed, the following steps can be used for thinking:

[0114] step1: First, judge whether only repeated words, colloquial words, and redundant word expressions in the reference text are lost, added, or corrected after speech recognition, and there is basically no semantic change. For example, changing "It's okay too" to "Sure", then output the tag {"No semantic change": 0} and end the determination.

[0115] step2: If the conditions of step1 are not met, then try to judge whether there are the following semantic inconsistency situations:

[0116] (1) "Sentence pattern change":

[0117] …

[0118] Output format:

[0119] {

[0120] "reason": "<Scoring reason>"

[0121] "score": ""

[0122] }

[0123] Please strictly evaluate the following data according to the above format:

[0124] Reference text: [Reference text]

[0125] Speech transcription text: [First transcription text].

[0126] This embodiment utilizes the capabilities of a large model without the need for a pre-trained model, and can generate a more accurate semantic consistency index score by leveraging the large model's natural language understanding and processing capabilities.

[0127] In some embodiments of the present application, the speech-to-text evaluation system may also have the ability to evaluate the credibility of the semantic consistency index scores given by the large model. That is, it can receive the calibration results of the semantic consistency index scores between the reference text and the first transcribed text by experts, and then compare the expert calibration results with the large model output results to give the comparison results of the human-machine consistency effect. This comparison result can further verify whether the large model can directly replace experts for semantic consistency evaluation work. In a possible application scenario, before the speech-to-text evaluation system of the present application is officially applied, the similarity between the semantic consistency evaluation results given by the large model and the results marked by experts can be verified through the above processing process (such as the similarity exceeding the set threshold). If the similarity exceeds the set threshold, it means that the output results of the large model are reliable enough, and the speech-to-text evaluation system can be applied to the evaluation stage; if the similarity does not exceed the set threshold, it means that the output results of the large model are not reliable enough, and the network parameters and prompt words of the large model can be further optimized until the output results of the large model are reliable enough.

[0128] Referring Figure 6 As shown, an evaluation process for the human-machine consistency effect of a speech-to-text evaluation system is introduced. A possible implementation process may include:

[0129] S1. The test audio is processed by a speech-to-text system (ASR system) to obtain the corresponding first transcribed text, and the reference text corresponding to the test audio is obtained.

[0130] S2. A parallel sentence pair is formed by the reference text and the first transcribed text of the same test audio, and a set of parallel sentence pairs can be constructed for multiple test audios in the test case.

[0131] S3. The large model is called to score the machine semantic consistency index for each parallel sentence pair in the set of parallel sentence pairs, and finally a machine scoring vector A at the test case dimension is obtained.

[0132] S4. The set of parallel sentence pairs obtained in S2 is provided to a language expert for manual semantic consistency index scoring, and finally a manual scoring vector B at the test case dimension is obtained.

[0133] S5. Calculate the similarity between vector A and vector B to obtain the evaluation result of the human-machine semantic scoring consistency.

[0134] In this step, the similarity between the two score vectors can use the Pearson correlation coefficient.

[0135] The method provided in this embodiment obtains the evaluation result of the consistency between the human and machine scores (the score for measuring the semantic consistency between the reference text and the first transcription text) by comparing the expert scoring results and the machine scoring results. Referring to this evaluation result, the reliability of the large model can be analyzed. So that when it is found that the large model is unreliable, the structure of the large model or the prompt words can be adjusted in time. After ensuring that the output result of the large model is reliable enough, the speech transcription evaluation system is applied to specific evaluation tasks to ensure the accuracy of the evaluation results of the speech transcription evaluation system.

[0136] In some embodiments of the present application, another method for evaluating a speech transcription system is introduced. In the method of this embodiment, a noise robustness index can be further added to the evaluation index of the speech transcription system to measure the stability of the transcribed text of the audio before and after adding noise. Then the evaluation index of the speech transcription system can include both a semantic consistency index and a noise robustness index. Or, the evaluation index of the speech transcription system can include both a recognition rate index, a semantic consistency index, and a noise robustness index.

[0137] By adding a noise robustness index to the evaluation index, the anti-noise ability of the speech transcription system can be evaluated.

[0138] Taking the example that the evaluation index includes both a recognition rate index, a semantic consistency index, and a noise robustness index, the evaluation score of the speech transcription system can be obtained comprehensively according to the scores of the recognition rate index, the semantic consistency index, and the noise robustness index. For example, the scores of the recognition rate index, the semantic consistency index, and the noise robustness index are weighted and added, and the result is used as the evaluation score T of the speech transcription system:

[0139] T = s1×wgt1 + s2×wgt2 + s3×wgt3.

[0140] Wherein, s1 represents the score of the recognition rate index, s2 represents the score of the semantic consistency index, s3 represents the score of the noise robustness index, and wgt1, wgt2, and wgt3 are three weight values, which can be flexibly adjusted according to the user's focus on the recognition rate index, the semantic consistency index, and the noise robustness index. The sum of the three weight values is equal to 1. Generally, the maximum value of a single weight does not exceed 0.5 to avoid a single index having too much influence on the evaluation result.

[0141] In the speech-to-text system evaluation method proposed in this embodiment, two types of indicators, semantic consistency and noise robustness, are added on the basis of the recognition rate indicator, effectively making up for the limitation that the current speech-to-text system evaluation method only focuses on the correctness of words. The speech-to-text system evaluation method of this embodiment comprehensively evaluates the speech-to-text system from multiple dimensions of the glyph, semantics, and noise resistance of the transcribed text, and the evaluation reliability is more robust than the existing solutions. Semantic consistency pays more attention to the subjective feelings of users, and noise robustness pays more attention to the special complex scenarios that the speech-to-text system may face. The speech-to-text system evaluation method provided in the embodiments of this application is more reasonable to evaluate the speech-to-text effect from the user's perspective than just looking at the recognition rate.

[0142] In some embodiments of this application, the calculation process of the noise robustness indicator score is introduced:

[0143] S1. Obtain the noise-added test audio, which is the audio obtained by adding noise to the test audio.

[0144] The types and intensities of the noise used during noise addition can be flexibly set. In this way, by adding noise to the test audio, noisy audio in a wider range of scenarios can be obtained, thereby shortening the construction time and cost of the noise-added test audio, enriching the diversity of the test set to a certain extent, and improving the robustness of the speech-to-text system effect evaluation indicators.

[0145] S2. Obtain the second transcribed text of the noise-added test audio, which is obtained by transcribing the noise-added test audio through the speech-to-text system.

[0146] S3. Based on the first transcribed text, the second transcribed text, and the reference text, determine the noise robustness indicator score.

[0147] In some possible implementations, the noise robustness indicator score can be a score obtained based on the statistics of the n-gram changes between the second transcribed text and the first transcribed text and the reference text.

[0148] Exemplarily, the SARI indicator score can be used as the noise robustness indicator score, or other statistical indicators can also be used to calculate the noise robustness indicator score.

[0149] Taking the calculation process of the SARI indicator score as an example, the first transcribed text is used as the source text, the second transcribed text is used as the generated text, and the SARI indicator score is calculated based on the source text, the generated text, and the reference text.

[0150] SARI (Sentence-level Automatic Relevance Indicator) is a metric used to evaluate text generation tasks such as text simplification and abstract generation. It is mainly used for tasks like text simplification. By comparing the n-gram edit operations between the generated text and the source text and reference text, it comprehensively calculates the scores for keep, delete, and add operations.

[0151] The core of SARI is to measure the accuracy of the following three types of edit operations:

[0152] KEEP: The generated text correctly retains the important content in the source text.

[0153] DELETE: The generated text correctly deletes the redundant content in the source text.

[0154] ADD: The generated text reasonably adds new content that exists in the reference text.

[0155] The final score is the weighted average of the three, and the formula is: SARI = (Keep score + Delete score + Add score) / 3

[0156] The following provides a calculation process for the SARI metric:

[0157] Step 1: Count n-grams.

[0158] Count the n-gram set of the source text: sgrams;

[0159] Count the n-gram set of the generated text: cgrams;

[0160] Count the n-gram set of the reference text: rgramsall.

[0161] Step 2: Count word frequencies.

[0162] sgramcounter = Counter(sgrams) (source text word frequency);

[0163] cgramcounter = Counter(cgrams) (generated text word frequency);

[0164] rgramcounter = Counter(rgramsall) (reference text word frequency).

[0165] Step 3: Calculate the scores of three types of editing operations.

[0166] (a) KEEP:

[0167] Kept n-grams: The n-grams shared by the generated text and the source text (intersection), i.e., text{keepgramcounter_rep} = sgram ∩ cgram.

[0168] Correct keep: The kept n-grams appear in the reference text, i.e.,

[0169] text{keepgramcountergood_rep} = text{keepgramcounter_rep} ∩ rgramsall.

[0170] (b) DELETE:

[0171] N-grams to be deleted: The n-grams that exist in the source text but are not kept in the generated text, i.e., text{delgramcounter_rep}=sgram – cgram.

[0172] Correct delete: The n-grams to be deleted do not appear in the reference text, i.e., text{delgramcountergood_rep} = text{delgramcounter_rep} – rgramsall.

[0173] (c) ADD:

[0174] Added n-grams: The n-grams that exist in the generated text but do not appear in the source text, i.e.,

[0175] addgramcounter=cgramcounter−sgramcounter.

[0176] Correct add: The added n-grams appear in the reference text, i.e.,

[0177] addgramcountergood=addgramcounter∩rgramcounter.

[0178] This embodiment further provides an example of calculating the SARI index.

[0179] Source text: ["the", "cat", "sat"], corresponding grams = {1: {'the', 'cat','sat'}}.

[0180] Candidate text: ["a", "cat", "sat"], corresponding cgrams = {1: {'a', 'cat','sat'}}.

[0181] Reference text: ["the", "cat", "sat"], corresponding rgrams = {1: {'the', 'cat','sat'}}.

[0182] Calculation process:

[0183] KEEP:

[0184] 1-grams common to the candidate and source texts: {'cat','sat'}.

[0185] Correct keep: {'cat','sat'} (all kept appear in the reference).

[0186] Precision P = 2 / 2 = 1.0, Recall R = 2 / 3 ≈ 0.666.

[0187] F1 score = (2×P×R) / (P + R) ≈ 0.8.

[0188] DELETE:

[0189] 1-grams to be deleted: those in the source but not in the candidate: {'the'}.

[0190] Correct delete: {'the'} (exists in the reference, should actually be kept, so not a valid delete).

[0191] P = 0 / 1 = 0, R = 0 / 1 = 0, F1 score = 0.

[0192] ADD:

[0193] Added 1-grams: parts in the candidate but not in the source: {'a'}.

[0194] Correct add: {} (added parts not in the reference).

[0195] P = 0 / 1 = 0, R = 0 / 1 = 0, F1-score = 0.

[0196] Take the average of the F1-scores for each editing operation, and the result is used as the SARI score:

[0197] SARI = (0.8 + 0 + 0) / 3 ≈ 0.267.

[0198] The above introduced the calculation of the SARI score using 1-gram as an example. In addition, the value of n in n-gram can also be other values. Or, when the N value includes multiple values, the corresponding SARI score can be calculated for each N value, and finally, the average of the SARI scores under all N values is taken as the final SARI score to more comprehensively measure the quality of text changes, that is, to measure the noise robustness of the speech transcription system.

[0199] The SARI metric quantifies the ability of the speech transcription system to maintain semantic consistency under noise interference by analyzing the rationality of addition, deletion, and retention operations. The higher the SARI metric score, the stronger the noise robustness of the speech transcription system.

[0200] In some embodiments of the present application, the speech transcription evaluation system can also have the ability to evaluate the credibility of the score of the noise robustness metric for machine evaluation (for example, the score of the noise robustness metric calculated using the SARI metric), that is, it can receive the calibration result of the noise robustness metric given by an expert based on the first transcription text, the second transcription text, and the reference text, and then compare the expert calibration result and the machine evaluation result to give a comparison result of the human-machine consistency effect. This comparison result can further verify whether the score of the noise robustness metric evaluated by the machine can directly replace the expert calibration. In a possible application scenario, before the speech transcription evaluation system of the present application is officially applied, the similarity between the score of the noise robustness metric evaluated by the machine and the result marked by the expert can be verified through the above processing process (such as the similarity exceeding a set threshold). If the similarity exceeds the set threshold, it means that the result of the machine evaluation is reliable enough, and the speech transcription evaluation system can be applied to the evaluation stage; if the similarity does not exceed the set threshold, it means that the result of the machine evaluation is not reliable enough, and the noise robustness metric evaluation algorithm can be further optimized until the evaluation result obtained by the machine based on the optimized noise robustness metric evaluation algorithm is reliable enough.

[0201] Refer to Figure 7 As shown, another evaluation process for the human-machine consistency effect of the speech transcription evaluation system is introduced. A possible implementation process may include:

[0202] S1. The test audio is processed through a speech-to-text system (ASR system) to obtain the corresponding first transcribed text, and the reference text corresponding to the test audio is obtained.

[0203] S2. The test audio is processed with noise addition to obtain the loaded test audio, and the second transcribed text is obtained through the ASR system.

[0204] S3. A triple is formed by the reference text, the first transcribed text, and the second transcribed text of the same test audio. For multiple test audios in the test case, a triple set can be constructed.

[0205] S4. For each triple in the triple set, the noise robustness index score is calculated according to the SARI algorithm, and finally a machine scoring vector C in the dimension of the test case is obtained.

[0206] S5. The triple set obtained in S3 is provided to a language expert for manual noise robustness scoring, and finally an artificial scoring vector D in the dimension of the test case is obtained.

[0207] S5. Calculate the similarity between vector C and vector D to obtain the evaluation result of the consistency of the human-machine noise robustness scoring.

[0208] In this step, the similarity between the two scoring vectors can use the Pearson correlation coefficient.

[0209] The method provided in this embodiment obtains the evaluation result of the consistency of the human-machine scoring (noise robustness index score) by comparing the expert scoring result and the machine scoring result. Referring to this evaluation result, the reliability of the noise robustness index evaluation algorithm (such as the SARI algorithm) can be analyzed. So that when it is found that the algorithm is unreliable, the noise robustness index evaluation algorithm can be adjusted in time. After ensuring that the noise robustness index score output by the machine is reliable enough, the speech-to-text evaluation system is applied to the specific evaluation task to ensure the accuracy of the evaluation result of the speech-to-text evaluation system.

[0210] Next, the speech-to-text system evaluation device provided in the embodiment of the present application will be described. The speech-to-text system evaluation device described below can be correspondingly referred to the speech-to-text system evaluation method described above.

[0211] See Figure 8 , Figure 8 which is a schematic structural diagram of a speech-to-text system evaluation device disclosed in the embodiment of the present application.

[0212] As Figure 8 shown, the device may include:

[0213] A test audio and reference text acquisition unit 11, configured to acquire the test audio and its corresponding reference text;

[0214] The first transcription text acquisition unit 12 is configured to acquire a first transcription text of the test audio, where the first transcription text is obtained by transcribing the test audio through a speech transcription system to be evaluated;

[0215] The semantic consistency index calculation unit 13 is configured to determine the semantic consistency between the first transcription text and the reference text, and obtain a semantic consistency index score;

[0216] The evaluation result determination unit 14 is configured to determine an evaluation result of the speech transcription system according to a score of a set evaluation index, where the set evaluation index at least includes the semantic consistency index.

[0217] In a possible implementation, the device of the present application further includes:

[0218] The noise-added audio acquisition unit is configured to acquire a noise-added test audio, where the noise-added test audio is an audio obtained by performing noise addition processing on the test audio;

[0219] The second transcription text acquisition unit is configured to acquire a second transcription text of the noise-added test audio, where the second transcription text is obtained by transcribing the noise-added test audio through the speech transcription system;

[0220] The noise robustness index calculation unit is configured to determine a noise robustness index score based on the first transcription text, the second transcription text, and the reference text, where the noise robustness index is used to measure the stability of the transcription text of the audio before and after noise addition. On this basis, the evaluation index adopted by the evaluation result determination unit further includes the noise robustness index.

[0221] In a possible implementation, the device of the present application further includes:

[0222] The recognition rate index calculation unit is configured to calculate a recognition rate index score according to the reference text and the first transcription text, where the recognition rate index represents the correct rate of character-to-sound transcription. On this basis, the evaluation index adopted by the evaluation result determination unit further includes the recognition rate index.

[0223] In a possible implementation, the process in which the semantic consistency index calculation unit determines the semantic consistency between the first transcription text and the reference text and obtains a semantic consistency index score includes:

[0224] Invoking a large model to instruct the large model to evaluate the semantic consistency between the first transcription text and the reference text, and output a semantic consistency index score.

[0225] In a possible implementation, the process of the noise robustness metric calculation unit determining the noise robustness metric score based on the first transcription text, the second transcription text, and the reference text includes:

[0226] Using the first transcription text as the source text and the second transcription text as the generated text, calculate the SARI metric score based on the source text, the generated text, and the reference text as the noise robustness metric score.

[0227] In a possible implementation, the process of the recognition rate metric calculation unit calculating the recognition rate metric score according to the reference text and the first transcription text includes:

[0228] Perform edit distance alignment on the reference text and the first transcription text;

[0229] Calculate the correct recognition rate COR metric score and the accuracy ACC metric score based on the alignment result;

[0230] Integrate the COR metric score and the ACC metric score to obtain the recognition rate metric score.

[0231] In a possible implementation, the process of the recognition rate metric calculation unit integrating the COR metric score and the ACC metric score to obtain the recognition rate metric score includes:

[0232] Perform weighted addition on the COR metric score and the ACC metric score to obtain the recognition rate metric score;

[0233] When the test audio is from a short audio test set, the first weight corresponding to the COR metric is less than the second weight corresponding to the ACC metric, and the duration of the test audio included in the short audio test set is less than the first set duration threshold;

[0234] When the test audio is from a long audio test set, the first weight corresponding to the COR metric is greater than the second weight corresponding to the ACC metric, and the duration of the test audio included in the long audio test set is greater than the second set duration threshold.

[0235] An embodiment of the present application also provides an electronic device. Refer to Figure 9 As shown, it shows a schematic structural diagram of an electronic device suitable for implementing the electronic device in the embodiment of the present application. The electronic device in the embodiment of the present application may include, but is not limited to, terminals such as mobile phones, tablet computers, translators, servers, and the like. Figure 9 The electronic device shown is only an example and should not bring any limitations to the functions and usage scopes of the embodiments of the present application.

[0236] Such as Figure 9As shown in the figure, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which may perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 into the random access memory (RAM) 603, so as to implement the speech-to-text system evaluation method in the foregoing embodiments of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.

[0237] Generally, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 9 an electronic device with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices may be implemented or had.

[0238] In an embodiment of the present application, there is also provided a computer program product including computer-readable instructions, which, when running on an electronic device, cause the electronic device to implement any one of the speech-to-text system evaluation methods provided in the embodiments of the present application.

[0239] In an embodiment of the present application, there is also provided a computer-readable storage medium carrying one or more computer programs, which, when executed by an electronic device, can cause the electronic device to implement any one of the speech-to-text system evaluation methods provided in the embodiments of the present application.

[0240] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the device embodiments provided in the present application, the connection relationships between the modules indicate that they have communication connections, which may be specifically implemented as one or more communication buses or signal lines.

[0241] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware. Of course, it can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be various, such as analog circuits, digital circuits, or dedicated circuits. However, for the present application, in more cases, software program implementation is a better embodiment. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a readable storage medium, such as a floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disc of a computer, etc., and includes several instructions for causing a computer device (which can be a personal computer, training device, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0242] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0243] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, training device, or data center to another website, computer, training device, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store, or a data storage device such as a training device or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0244] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

Claims

1. A speech transcription system evaluation method, characterized in that: include: Get the test audio and its corresponding reference text; Obtaining a first transcribed text of the test audio, where the first transcribed text is obtained by transcribing the test audio through a speech transcription system to be evaluated; Determining semantic consistency between the first transcribed text and the reference text to obtain a semantic consistency index score; The evaluation result of the speech transcription system is determined according to the score of the set evaluation index, and the set evaluation index at least includes the semantic consistency index.

2. The method according to claim 1, characterized in that Also includes: Acquire a noise-added test audio, where the noise-added test audio is the audio obtained after the test audio is subjected to noise-adding processing; Obtaining a second transcribed text of the noisy test audio, where the second transcribed text is obtained by transcribing the noisy test audio through the speech transcription system; Determining a noise robustness index score based on the first transcribed text, the second transcribed text, and the reference text, wherein the noise robustness index is used to measure the stability of the transcribed text of the audio before and after noise addition; The set evaluation index also includes the noise robustness index.

3. The method according to claim 1 or 2, characterized in that: Also includes: Calculating a recognition rate index score based on the reference text and the first transcribed text, wherein the recognition rate index represents the accuracy of phonetic character transcription; The set evaluation index also includes the recognition rate index.

4. The method according to claim 1, characterized in that: The process of determining the semantic consistency between the first transcribed text and the reference text and obtaining a semantic consistency index score includes: The big model is called to instruct the big model to evaluate the semantic consistency between the first transcribed text and the reference text, and output a semantic consistency index score.

5. The method according to claim 2, characterized in that: The process of determining a noise robustness index score based on the first transcribed text, the second transcribed text, and the reference text includes: The first transcribed text is used as the source text, the second transcribed text is used as the generated text, and a SARI index score is calculated based on the source text, the generated text and the reference text as a noise robustness index score.

6. The method according to claim 3, characterized in that The process of calculating the recognition rate index score according to the reference text and the first transcribed text includes: Performing edit distance alignment on the reference text and the first transcribed text; Calculate the correct recognition rate COR index score and the accuracy rate ACC index score based on the alignment results; The COR index score and the ACC index score are integrated to obtain a recognition rate index score.

7. The method according to claim 6, characterized in that The process of integrating the COR index score and the ACC index score to obtain the recognition rate index score includes: Performing weighted addition on the COR index score and the ACC index score to obtain a recognition rate index score; In a case where the test audio is from a short audio test set, the first weight corresponding to the COR indicator is less than the second weight corresponding to the ACC indicator, and the duration of the test audio included in the short audio test set is less than a first set duration threshold; In the case where the test audio comes from a long audio test set, the first weight corresponding to the COR indicator is greater than the second weight corresponding to the ACC indicator, and the duration of the test audio included in the long audio test set is greater than the second set duration threshold.

8. A speech transcription system evaluation device, characterized in that: include: A test audio and reference text acquisition unit, used to acquire the test audio and its corresponding reference text; A first transcribed text obtaining unit, configured to obtain a first transcribed text of the test audio, wherein the first transcribed text is obtained by transcribing the test audio through a speech transcription system to be evaluated; A semantic consistency index calculation unit, used to determine the semantic consistency between the first transcribed text and the reference text, and obtain a semantic consistency index score; An evaluation result determination unit is used to determine the evaluation result of the speech transcription system according to the score of the set evaluation index, and the set evaluation index at least includes the semantic consistency index.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the speech transcription system evaluation method according to any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the speech transcription system evaluation method as described in any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, each step of the speech transcription system evaluation method as described in any one of claims 1 to 7 is implemented.

Images