Speech transcription system evaluation method, device, related equipment and program product

Abstract

The present application discloses a method, device, related equipment and program product for evaluating a speech transcription system, which relates to the field of speech recognition technology. When evaluating the speech transcription system, the present application introduces a semantic consistency index, which can measure the semantic consistency between the first transcription text of the test audio and the reference text, that is, measure the degree to which the first transcription text and the reference text express the same meaning at the semantic level, and can accurately measure the ability of the speech transcription system to transmit the semantics of the test audio. Furthermore, at least based on the score of the semantic consistency index, the evaluation result of the speech transcription system is determined, which improves the objectivity of the evaluation result. At the same time, the semantic consistency evaluation index is also more in line with the real reading experience of humans, so the obtained evaluation result is also more in line with the real feelings of humans.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and more specifically, to a method, device, related equipment and program product for evaluating a speech transcription system. Background Art

[0002] Speech transcription, as an important case of artificial intelligence technology, is widely used in various fields. It is crucial to accurately evaluate the performance of a speech transcription system.

[0003] Currently, the mainstream speech transcription systems in the industry generally only consider the accuracy of character-for-character transcription of the speech transcription system. If the character-for-character transcription accuracy is high, it is considered that the transcription effect of the speech transcription system is good. However, in actual business scenarios, it is often the case that the character-for-character transcription accuracy is already very high, but there are still problems such as unclear meaning and deviation in understanding when people read the transcribed text. That is to say, the evaluation results of the current speech transcription system evaluation method are not objective enough and inconsistent with human's real feelings. Summary of the Invention

[0004] In view of the above problems, this application is proposed to provide a method, device, related equipment and program product for evaluating a speech transcription system, so as to improve the objectivity of the evaluation results of the speech transcription system and the consistency with human's real feelings. The specific solutions are as follows:

[0005] In a first aspect, a method for evaluating a speech transcription system is provided, including:

[0006] Obtain a test audio and its corresponding reference text;

[0007] Obtain a first transcribed text of the test audio, where the first transcribed text is obtained by transcribing the test audio through a speech transcription system to be evaluated;

[0008] Determine the semantic consistency between the first transcribed text and the reference text to obtain a semantic consistency index score;

[0009] Determine the evaluation result of the speech transcription system according to the score of the set evaluation index, where the set evaluation index at least includes the semantic consistency index.

[0010] In a possible design, in another implementation manner of the first aspect of the embodiments of this application, it further includes:

[0011] Obtain a noise-added test audio, where the noise-added test audio is an audio obtained by adding noise to the test audio;

[0012] Obtain a second transcribed text of the noise-added test audio, where the second transcribed text is obtained by transcribing the noise-added test audio through the speech transcription system;

[0013] Determine a noise robustness index score based on the first transcript text, the second transcript text, and the reference text, where the noise robustness index is used to measure the stability of the transcript text of the audio before and after adding noise;

[0014] The set evaluation metrics further include the noise robustness index.

[0015] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, it further includes:

[0016] Calculate a recognition rate index score according to the reference text and the first transcript text, where the recognition rate index represents the correct rate of character-to-phoneme transcription;

[0017] The set evaluation metrics further include the recognition rate index.

[0018] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the process of determining the semantic consistency between the first transcript text and the reference text to obtain a semantic consistency index score includes:

[0019] Call a large model to instruct the large model to evaluate the semantic consistency between the first transcript text and the reference text, and output a semantic consistency index score.

[0020] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the process of determining a noise robustness index score based on the first transcript text, the second transcript text, and the reference text includes:

[0021] Use the first transcript text as the source text and the second transcript text as the generated text, and calculate the SARI index score based on the source text, the generated text, and the reference text as the noise robustness index score.

[0022] In a possible design, in another implementation manner of the first aspect of the embodiments of the present application, the process of calculating a recognition rate index score according to the reference text and the first transcript text includes:

[0023] Perform edit distance alignment on the reference text and the first transcript text;

[0024] Calculate the correct recognition rate COR index score and the accuracy ACC index score based on the alignment result;

[0025] Integrate the COR index score and the ACC index score to obtain the recognition rate index score.

[0026] In a possible design, in another implementation of the first aspect of the embodiments of the present application, the process of integrating the COR metric score and the ACC metric score to obtain the recognition rate metric score includes:

[0027] Performing weighted addition on the COR metric score and the ACC metric score to obtain the recognition rate metric score;

[0028] When the test audio is from a short audio test set, the first weight corresponding to the COR metric is less than the second weight corresponding to the ACC metric, and the duration of the test audio included in the short audio test set is less than the first set duration threshold;

[0029] When the test audio is from a long audio test set, the first weight corresponding to the COR metric is greater than the second weight corresponding to the ACC metric, and the duration of the test audio included in the long audio test set is greater than the second set duration threshold.

[0030] In a second aspect, there is provided an evaluation device for a speech transcription system, including:

[0031] A test audio and reference text acquisition unit, configured to acquire a test audio and its corresponding reference text;

[0032] A first transcription text acquisition unit, configured to acquire a first transcription text of the test audio, where the first transcription text is obtained by transcribing the test audio through a speech transcription system to be evaluated;

[0033] A semantic consistency metric calculation unit, configured to determine the semantic consistency between the first transcription text and the reference text to obtain a semantic consistency metric score;

[0034] An evaluation result determination unit, configured to determine the evaluation result of the speech transcription system according to the scores of the set evaluation metrics, where the set evaluation metrics at least include the semantic consistency metric.

[0035] In a third aspect, there is provided an electronic device, including: a memory and a processor;

[0036] The memory is configured to store a program;

[0037] The processor is configured to execute the program to implement each step of the speech transcription system evaluation method described in any one of the foregoing first aspects of the present application.

[0038] In a fourth aspect, there is provided a readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, each step of the speech transcription system evaluation method described in any one of the foregoing first aspects of the present application is implemented.

[0039] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the various steps of the speech transcription system evaluation method described in any one of the first aspects of the present application.

[0040] By leveraging the above technical solution, this application introduces a semantic consistency index when evaluating speech transcription systems. This index can measure the semantic consistency between the first transcription of the test audio and the reference text, that is, it measures the degree to which the first transcription and the reference text express the same meaning at the semantic level. It can accurately measure the speech transcription system's ability to convey the semantics of the test audio, and then determine the evaluation results of the speech transcription system based at least on the semantic consistency index score, thereby improving the objectivity of the evaluation results. At the same time, the semantic consistency evaluation index is more consistent with the real human reading experience, so the evaluation results obtained are also more consistent with the real human experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0042] Figure 1 A schematic diagram of the alignment of a reference text with several different transcriptions is provided;

[0043] Figure 2 A processing flow diagram of a speech transcription system integrated with a speech transcription evaluation system is illustrated;

[0044] Figure 3 A flowchart of a speech transcription system evaluation method provided in an embodiment of the present application;

[0045] Figure 4 An example of a recognition rate indicator calculation process diagram is shown;

[0046] Figure 5 A schematic diagram of the edit distance alignment of a reference text and a first transcription text is provided;

[0047] Figure 6 This example shows a flow chart of the human-computer consistency evaluation process for a speech transcription evaluation system.

[0048] Figure 7 This example shows a flowchart of the human-computer consistency evaluation process for another speech transcription evaluation system.

[0049] Figure 8Schematic diagram of the structure of an evaluation device for a speech-to-text transcription system disclosed in an embodiment of the present application;

[0050] Figure 9 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Detailed implementation manners

[0051] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0052] Currently, the mainstream speech-to-text transcription systems in the industry generally use the character conversion accuracy rate of speech-to-character transcription to measure the transcription effect. For example, the COR (Correct Recognition Rate) or ACC (Accuracy) index is used to reflect the quality of the speech-to-text transcription system. The higher the score of the COR index or ACC index, the higher the transcription effect of the speech-to-text transcription system is considered. The prior art does not consider whether the actual semantics of the transcribed text and the audio are consistent, nor does it consider the robustness of the transcription system in various complex noise scenarios. Therefore, there is often a problem that although the COR index and ACC index are already very high, whether reading the transcribed text or using the transcribed text for downstream tasks, there will always be frequent problems such as deviation in information transmission or failure of downstream cascaded tasks due to transcription errors.

[0053] The COR index strictly statistically measures the character consistency between the transcribed text and the reference text, ignoring character conversion errors such as insertions, deletions, and substitutions. This index is calculated by the formula (1):

[0054] COR = H / N × 100% (1)

[0055] Wherein, N represents the total number of characters in the reference text, and H represents the number of correctly transcribed characters.

[0056] The ACC index represents the overall proportion of the correct part in the transcribed text, taking into account the influence of insertions, deletions, and substitution errors. The specific calculation formulas are shown as follows (2) and (3).

[0057] WER = (S + I + D) / N (2)

[0058] ACC = 1 - WER (3)

[0059] Refer to Figure 1, an example is provided. The transcription results of the same audio in three speech transcription systems are REC1, REC2, and REC3 respectively, and the alignment results with the reference text LAB are as Figure 1 shown.

[0060] The reference text of the audio has 15 characters. There are no insertion or deletion errors in the three transcription systems, and the substitution error is 1 for all of them. According to formulas (1), (2), and (3), the COR and ACC indicators of the three systems can be calculated to be 93.33%. According to the existing technology, it can be concluded that there is no difference in the quality of the three systems.

[0061] However, through the analysis of the transcription results of the three systems, it can be found that:

[0062] The error in REC1 is the transcription error of the characters in the entity words, which has the greatest impact on the conveyance of the correct semantics. The error in REC2 is the error of the modal particles, which has basically no loss to the semantic conveyance in the current context. The error in REC3 is the transcription error of the verbs, which has a certain impact on the semantic transmission in the current context, but the degree of impact is less than that of the entity word error in REC1. Therefore, although the transcription accuracy indicators of the three systems are the same, from an objective perspective and combined with people's subjective feelings, the effect of REC2 is the best, REC3 is the second, and REC1 is the worst. Therefore, it is not robust enough to use only the COR and ACC indicators to evaluate the transcription system. It only focuses on the literal errors, including insertion, deletion, and substitution errors, and completely ignores the use of words in a specific context. In some cases, even if the literal errors are few, it may cause inappropriate context due to local errors, resulting in information transmission errors, which is somewhat deviated from people's subjective feelings.

[0063] In addition, the type and intensity of noise have a significant impact on the transcription accuracy of speech. A robust speech transcription system should keep the text transcribed from the same audio stable in various noise scenarios. The current evaluation method of speech transcription systems also does not consider the impact of noise on the speech transcription system.

[0064] In view of this, the present application provides a method for evaluating a speech transcription system, which can at least solve some of the defects existing in the prior art. The method for evaluating a speech transcription system of the present application can be applied to evaluate the transcription effect of speech transcription systems in various industries and scenarios.

[0065] The present application provides an evaluation system for a speech transcription system (hereinafter referred to as the speech transcription evaluation system), which can be deployed on a terminal or a server. Exemplarily, the speech transcription evaluation system can be deployed on the same device as the speech transcription system, or the two can be deployed on different devices.

[0066] Such as Figure 2, which exemplifies the processing flow of a speech transcription system integrated with a speech transcription evaluation system.

[0067] Transcribe the test audio through a speech transcription system (ASR system) to obtain the transcribed text of the test audio (defined as the first transcribed text).

[0068] In addition, obtain the reference text of the test audio, which is the standard text corresponding to the content of the test audio. The test audio and the reference text can be obtained from a publicly available dataset, or an expert can annotate the test audio to obtain the reference text. In addition, it can also be to perform speech synthesis on the reference text and use the synthesized speech as the test audio.

[0069] By sending the reference text and the first transcribed text into the speech transcription evaluation system, execute the specific evaluation process, such as determining the semantic consistency between the first transcribed text and the reference text, obtaining the semantic consistency index score, and obtaining the evaluation result of the speech transcription system based on this semantic consistency index score. Examples are in the form of the evaluation score and evaluation level of the speech transcription system.

[0070] In some possible implementations, as Figure 2 shown in the right branch in, for the test audio, it is also possible to obtain a noise-added test audio through automatic noise addition processing. Further transcribe it through the ASR system to obtain the transcribed text of the noise-added test audio (defined as the second transcribed text). On this basis, the second transcribed text can also be sent into the speech transcription evaluation system for the evaluation system to calculate the noise robustness index score. Then, the evaluation result of the speech transcription system can be determined based on the scores of various set evaluation metrics such as the semantic consistency index score and the noise robustness index score.

[0071] Next, from the perspective of the speech transcription evaluation system, an embodiment of the present application provides a method for evaluating a speech transcription system. Refer to Figure 3 , this speech transcription system evaluation method specifically includes the following steps:

[0072] Step S100, obtain the test audio and its corresponding reference text.

[0073] Specifically, when testing the speech transcription system, the target scenario (domain) to be tested can be specified, that is, this test task is: to test the transcription effect of the speech transcription system in the target scenario.

[0074] The test audio obtained in this step can be from an audio test set related to the target scenario. Exemplarily, if the target scenario is a human-computer interaction scenario, the test audio can be from a short audio test set related to the human-computer interaction scenario. Another example is that if the target scenario is a meeting scenario, the test audio can be from a long audio test set related to meeting transcription in the meeting scenario.

[0075] The reference text corresponding to the test audio is the standard text corresponding to the content of the test audio. The test audio and the reference text can be obtained from a publicly available dataset, or the reference text can be obtained by an expert annotating the test audio. In addition, it can also be that the reference text is synthesized into speech, and the synthesized speech is used as the test audio.

[0076] Step S110: Obtain the first transcription text of the test audio, which is obtained by transcribing the test audio through the speech transcription system to be evaluated.

[0077] Step S120: Determine the semantic consistency between the first transcription text and the reference text to obtain the semantic consistency index score.

[0078] In this embodiment, the semantic consistency index is used as an evaluation index for evaluating the effect of the speech transcription system. The semantic consistency index is used to measure the semantic consistency between the first transcription text and the reference text, that is, to measure the ability of the speech transcription system to transmit the semantics of the test audio.

[0079] The semantic consistency index score represents the degree of semantic consistency between the first transcription text and the reference text. The higher the semantic consistency index score, the higher the degree of semantic consistency between the first transcription text and the reference text, indicating that the transcription effect of the speech transcription system is better.

[0080] Step S130: Determine the evaluation result of the speech transcription system according to the score of the set evaluation index, where the set evaluation index at least includes the semantic consistency index.

[0081] This application can pre-define one or more evaluation indexes for evaluating the speech transcription system, and at least the semantic consistency index is included in these evaluation indexes.

[0082] In this step, according to the score of the set evaluation index, the evaluation effect of the speech transcription system can be calculated. The evaluation effect can be reflected in various forms. For example, the evaluation score, evaluation level, etc. are used to represent the evaluation effect of the speech transcription system.

[0083] When evaluating a speech-to-text system using the evaluation method provided in the embodiments of this application, a semantic consistency metric is introduced. This metric can measure the semantic consistency between the first transcribed text of the test audio and the reference text, that is, it measures the degree to which the first transcribed text and the reference text express the same meaning at the semantic level, and can accurately measure the ability of the speech-to-text system to convey the semantics of the test audio. Furthermore, based at least on the score of the semantic consistency metric, the evaluation result of the speech-to-text system is determined, which improves the objectivity of the evaluation result. At the same time, the semantic consistency evaluation metric is also more in line with the real reading experience of humans, so the obtained evaluation result is also more in line with the real feelings of humans.

[0084] In some possible implementations, the evaluation metric of the speech-to-text system may only include the semantic consistency metric, and then the calculated score of the semantic consistency metric can be directly used as the evaluation score of the speech-to-text system.

[0085] In other possible implementations, the evaluation metric of the speech-to-text system may further include a recognition rate metric. This recognition rate metric represents the correct rate of character-for-sound transcription, that is, the correct rate of transcribing the test audio into characters. On this basis, the method of this application may further include:

[0086] Calculate the score of the recognition rate metric based on the reference text and the first transcribed text.

[0087] Then the evaluation score of the speech-to-text system can be comprehensively obtained based on the score of the recognition rate metric and the score of the semantic consistency metric. For example, the scores of the recognition rate metric and the semantic consistency metric are weighted and added, and the result is used as the evaluation score T of the speech-to-text system:

[0088] T = s1 × wgt1 + s2 × wgt2;

[0089] where s1 represents the score of the recognition rate metric, s2 represents the score of the semantic consistency metric, and wgt1 and wgt2 are two weight values that can be flexibly adjusted according to the user's focus on the recognition rate metric and the semantic consistency metric, and the sum of the two weight values is equal to 1.

[0090] An optional calculation process for the score of the recognition rate metric is introduced in this embodiment.

[0091] Specifically, the score of the recognition rate metric can be determined based on the score of the correct recognition rate COR metric and / or the accuracy rate ACC metric. For example, the score of the correct recognition rate COR metric or the accuracy rate ACC metric can be used as the score of the recognition rate metric. Or, the scores of the COR metric and the ACC metric can be integrated to obtain the score of the recognition rate metric. The following formula exemplifies a calculation method for the score of the recognition rate metric:

[0092] s1 = α × COR + β × ACC;

[0093] Where α and β represent two weight values, and their sum equals 1. COR and ACC represent the scores of two metrics respectively.

[0094] The COR metric and the ACC metric each have their own preferred applicable scenarios.

[0095] The COR metric focuses on measuring the correct proportion at the character level, ignoring character-level phonetic transcription errors such as insertions, deletions, and substitutions, and can reflect local correctness. Therefore, it is more suitable for long audio, such as the transcribed text of meeting records, the generation of long speeches or blog content, tasks that require statistical local accuracy (such as subtitle generation), etc.

[0096] The ACC metric is considered correct only when the transcribed text is exactly the same as the reference text, ensuring absolute accuracy of the text. Short audio usually corresponds to short texts (such as "Turn on the air conditioner", "Navigate to the company", etc.), with short sentences and key semantics. Minor errors may lead to serious consequences. For example, "Turn off the light" being wrongly transcribed as "Turn on the light", even if only one character is replaced, may completely change the intention. Therefore, the ACC metric is more suitable for short audio, such as voice assistant commands (such as smart home control), key information extraction (such as phone numbers, verification codes), short sentence translation, or command execution, etc. The ACC metric better meets the "zero tolerance for errors" requirement of short audio.

[0097] According to the respective preferred application scenarios of the COR metric and the ACC metric, in the method for evaluating the performance of a speech transcription system in this embodiment, different weights can be set for the COR metric and the ACC metric when testing the performance of the speech transcription system in different scenarios. For example:

[0098] When the test audio is from a short audio test set, the first weight α corresponding to the COR metric is less than the second weight β corresponding to the ACC metric;

[0099] When the test audio is from a long audio test set, the first weight α corresponding to the COR metric is greater than the second weight β corresponding to the ACC metric.

[0100] Among them, the duration of the test audio included in the short audio test set is less than the first set duration threshold, and the duration of the test audio included in the long audio test set is greater than the second set duration threshold. The first set duration threshold and the second set duration threshold can be the same or different. When their values are different, the first set duration threshold is less than the second set duration threshold.

[0101] Combined with Figure 4 As shown, a calculation process for the recognition rate metric is introduced, specifically including:

[0102] For the test audio, obtain its reference text and the first transcription text of the test audio obtained by a speech-to-text system (ASR system).

[0103] Furthermore, perform an edit distance alignment on the reference text and the first transcription text.

[0104] Calculate the COR metric score and the ACC metric score based on the alignment result.

[0105] Perform a weighted addition on the COR metric score and the ACC metric score to obtain the recognition rate score.

[0106] Combined Figure 5 , which provides an example of the edit distance alignment between the reference text LAB and the first transcription text REC.

[0107] It is statistically found that there are 16 characters in total in the reference text, i.e., N = 16. Among them, 13 characters are correctly transcribed, i.e., H = 13, the number of inserted characters is 2, i.e., I = 2, the number of substitution errors is 2, i.e., S = 2, and the number of deletion errors is 1, i.e., D = 1. Then, according to the aforementioned formulas (1), (2), and (3), COR can be calculated as 81.25% and ACC as 68.75%. Define α = 0.2 and β = 0.8, then the recognition rate metric score s1 = 71.25 is calculated.

[0108] In some embodiments of the present application, the determination process of the semantic consistency metric score is introduced.

[0109] In one possible implementation, a semantic consistency discriminant model can be pre-trained. This model can adopt a deep neural network model structure. First, obtain the training data, which includes text pairs and the text pairs are labeled with semantic consistency scores. Use this training data to train the semantic consistency discriminant model to obtain the trained semantic consistency discriminant model. Furthermore, the semantic consistency discriminant model can be used to process the first transcription text and the reference text to output the semantic consistency metric score of the two.

[0110] In another possible implementation, this embodiment can call a large model and, by virtue of the natural language understanding and processing capabilities of the large model, output the semantic consistency metric score of the reference text and the first transcription text. That is to say, in this embodiment, a large model can be called to instruct the large model to evaluate the semantic consistency between the first transcription text and the reference text and output the semantic consistency metric score.

[0111] When calling a large model to evaluate the semantic consistency between the first transcription text and the reference text, the first transcription text and the reference text can be concatenated with a pre-set prompt to form a prompt, and this prompt is sent into the large model to obtain the semantic consistency index score output by the large model. In the prompt, the thinking mode of the large model can be specified to guide the large model to more accurately evaluate the semantic consistency index score. The following is an example of some content of the prompt:

[0112] We are performing a speech transcription task and need to evaluate the semantic change of the text after speech transcription relative to the reference text. The output is the degree of semantic change (an integer between 0 and 100, the higher the degree, the larger the value).

[0113] When judging whether the semantics has changed, the following steps can be used for thinking:

[0114] step1: First, judge whether only repeated words, colloquial words, and redundant word expressions in the reference text are lost, added, or corrected after speech recognition, and there is basically no semantic change. For example, changing "It's okay too" to "Sure", then output the marker {"No semantic change": 0} and end the judgment.

[0115] step2: If the conditions of step1 are not met, then try to judge whether there are the following semantic inconsistencies:

[0116] (1) "Sentence pattern change":

[0117] …

[0118] Output format:

[0119] {

[0120] "reason": "<Scoring reason>"

[0121] "score": ""

[0122] }

[0123] Please strictly evaluate the following data according to the above format:

[0124] Reference text: [Reference text]

[0125] Speech transcription text: [First transcription text].

[0126] This embodiment calls the capabilities of a large model, does not require a pre-trained model, and can generate a more accurate semantic consistency index score by leveraging the natural language understanding and processing capabilities of the large model.

[0127] In some embodiments of the present application, the speech-to-text evaluation system can also have the ability to evaluate the credibility of the semantic consistency index scores given by the large model. That is, it can receive the calibration results of experts on the semantic consistency index scores between the reference text and the first transcribed text, and then compare the expert calibration results with the large model output results to give the comparison results of the human-machine consistency effect. This comparison result can further verify whether the large model can directly replace experts in the semantic consistency evaluation work. In a possible application scenario, before the speech-to-text evaluation system of the present application is officially applied, the similarity between the semantic consistency evaluation results given by the large model and the results marked by experts can be verified through the above processing process (such as the similarity exceeding the set threshold). If the similarity exceeds the set threshold, it means that the output result of the large model is reliable enough, and the speech-to-text evaluation system can be applied to the evaluation stage; if the similarity does not exceed the set threshold, it means that the output result of the large model is not reliable enough, and the network parameters and prompt words of the large model can be further optimized until the output result of the large model is reliable enough.

[0128] Referring Figure 6 As shown, an evaluation process for the human-machine consistency effect of a speech-to-text evaluation system is introduced. A possible implementation process may include:

[0129] S1. The test audio is passed through a speech-to-text system (ASR system) to obtain the corresponding first transcribed text, and the reference text corresponding to the test audio is obtained.

[0130] S2. A parallel sentence pair is formed by the reference text and the first transcribed text of the same test audio, and a parallel sentence pair set can be constructed for multiple test audios in the test case.

[0131] S3. The large model is called to score the machine semantic consistency index for each parallel sentence pair in the parallel sentence pair set, and finally a machine scoring vector A at the test case dimension is obtained.

[0132] S4. The parallel sentence pair set obtained in S2 is provided to language experts for manual semantic consistency index scoring, and finally a manual scoring vector B at the test case dimension is obtained.

[0133] S5. Calculate the similarity between vector A and vector B to obtain the evaluation result of the human-machine semantic scoring consistency.

[0134] In this step, the similarity between the two score vectors can use the Pearson correlation coefficient.

[0135] The method provided in this embodiment obtains the evaluation result of the consistency between human and machine scoring (the score measuring the semantic consistency between the reference text and the first transcription text) by comparing the expert scoring result and the machine scoring result. Referring to this evaluation result, the reliability of the large model can be analyzed, so that when the large model is found to be unreliable, the structure of the large model or the prompt words can be adjusted in time. After ensuring that the output result of the large model is reliable enough, the speech transcription evaluation system is applied to specific evaluation tasks to ensure the accuracy of the evaluation result of the speech transcription evaluation system.

[0136] In some embodiments of the present application, another method for evaluating a speech transcription system is introduced. In the method of this embodiment, a noise robustness index can be further added to the evaluation indexes of the speech transcription system to measure the stability of the transcription text of the audio before and after adding noise. Then, the evaluation indexes of the speech transcription system can simultaneously include a semantic consistency index and a noise robustness index. Or, the evaluation indexes of the speech transcription system can simultaneously include a recognition rate index, a semantic consistency index, and a noise robustness index.

[0137] By adding a noise robustness index to the evaluation indexes, the anti-noise ability of the speech transcription system can be evaluated.

[0138] Taking the example that the evaluation indexes simultaneously include a recognition rate index, a semantic consistency index, and a noise robustness index, the evaluation score of the speech transcription system can be obtained comprehensively according to the scores of the recognition rate index, the semantic consistency index, and the noise robustness index. For example, the scores of the recognition rate index, the semantic consistency index, and the noise robustness index are weighted and added, and the result is used as the evaluation score T of the speech transcription system:

[0139] T = s1×wgt1 + s2×wgt2 + s3×wgt3.

[0140] Wherein, s1 represents the score of the recognition rate index, s2 represents the score of the semantic consistency index, s3 represents the score of the noise robustness index, and wgt1, wgt2, and wgt3 are three weight values, which can be flexibly adjusted according to the user's focus on the recognition rate index, the semantic consistency index, and the noise robustness index, and the sum of the three weight values is equal to 1. Generally, the maximum value of a single weight does not exceed 0.5 to avoid a single index having too much influence on the evaluation result.

[0141] In the speech-to-text system evaluation method proposed in this embodiment, two types of indicators, semantic consistency and noise robustness, are added based on the recognition rate indicator, effectively making up for the limitation that the current speech-to-text system evaluation method only focuses on whether the words are correct or wrong. The speech-to-text system evaluation method of this embodiment comprehensively evaluates the speech-to-text system from multiple dimensions of the glyph, semantics, and noise resistance of the transcribed text, and the evaluation reliability is more robust than the existing solutions. Semantic consistency pays more attention to the subjective feelings of users, and noise robustness pays more attention to the special complex scenarios that the speech-to-text system may face. The speech-to-text system evaluation method provided in the embodiments of this application is more reasonable to evaluate the speech-to-text effect from the user's perspective than only looking at the recognition rate.

[0142] In some embodiments of this application, the calculation process of the noise robustness indicator score is introduced:

[0143] S1. Obtain the noise-added test audio, which is the audio obtained by adding noise to the test audio.

[0144] The types and intensities of the noise used during noise addition can be flexibly set. In this way, by adding noise to the test audio, noisy audio in a wider range of scenarios can be obtained, thereby shortening the construction time and cost overhead of the noise-added test audio, enriching the diversity of the test set to a certain extent, and improving the robustness of the speech-to-text system effect evaluation indicators.

[0145] S2. Obtain the second transcribed text of the noise-added test audio, which is obtained by transcribing the noise-added test audio through the speech-to-text system.

[0146] S3. Based on the first transcribed text, the second transcribed text, and the reference text, determine the noise robustness indicator score.

[0147] In some possible implementations, the noise robustness indicator score can be a score obtained based on the statistics of the n-gram changes between the second transcribed text and the first transcribed text and the reference text.

[0148] Exemplarily, the SARI indicator score can be used as the noise robustness indicator score, or other statistical indicators can also be used to calculate the noise robustness indicator score.

[0149] Taking the calculation process of the SARI indicator score as an example, the first transcribed text is used as the source text, the second transcribed text is used as the generated text, and based on the source text, the generated text, and the reference text, the SARI indicator score is calculated.

[0150] SARI (Sentence-level Automatic Relevance Indicator) is a metric used to evaluate text generation tasks such as text simplification and abstract generation. It is mainly used for tasks like text simplification. By comparing the n-gram edit operations between the generated text and the source text and the reference text, it comprehensively calculates the scores for keep, delete, and add operations.

[0151] The core of SARI is to measure the accuracy of the following three types of edit operations:

[0152] KEEP: The generated text correctly retains the important content in the source text.

[0153] DELETE: The generated text correctly deletes the redundant content in the source text.

[0154] ADD: The generated text reasonably adds new content that exists in the reference text.

[0155] The final score is the weighted average of the three, and the formula is: SARI = (Keep score + Delete score + Add score) / 3

[0156] The following provides a calculation process for the SARI metric:

[0157] Step 1: Count n-grams.

[0158] Count the n-gram set of the source text: sgrams;

[0159] Count the n-gram set of the generated text: cgrams;

[0160] Count the n-gram set of the reference texts: rgramsall.

[0161] Step 2: Count word frequencies.

[0162] sgramcounter = Counter(sgrams) (source text word frequency);

[0163] cgramcounter = Counter(cgrams) (generated text word frequency);

[0164] rgramcounter = Counter(rgramsall) (reference text word frequency).

[0165] Step 3: Calculate the scores of three types of editing operations.

[0166] (a) KEEP:

[0167] Kept n-grams: The n-grams shared by the generated text and the source text (intersection), i.e., text{keepgramcounter_rep} = sgram ∩ cgram.

[0168] Correctly kept: The kept n-grams appear in the reference text, i.e.,

[0169] text{keepgramcountergood_rep} = text{keepgramcounter_rep} ∩ rgramsall.

[0170] (b) DELETE:

[0171] N-grams to be deleted: The n-grams that exist in the source text but are not kept in the generated text, i.e., text{delgramcounter_rep}=sgram – cgram.

[0172] Correctly deleted: The n-grams to be deleted do not appear in the reference text, i.e., text{delgramcountergood_rep} = text{delgramcounter_rep} – rgramsall.

[0173] (c) ADD:

[0174] Added n-grams: The n-grams that exist in the generated text but do not appear in the source text, i.e.,

[0175] addgramcounter = cgramcounter - sgramcounter.

[0176] Correctly added: The added n-grams appear in the reference text, i.e.,

[0177] addgramcountergood = addgramcounter ∩ rgramcounter.

[0178] This embodiment further provides an example of calculating the SARI index.

[0179] Source text: ["the", "cat", "sat"], corresponding grams = {1: {'the', 'cat','sat'}}.

[0180] Candidate text: ["a", "cat", "sat"], corresponding cgrams = {1: {'a', 'cat','sat'}}.

[0181] Reference text: ["the", "cat", "sat"], corresponding rgrams = {1: {'the', 'cat','sat'}}.

[0182] Calculation process:

[0183] KEEP:

[0184] 1-grams common to the candidate and source texts: {'cat','sat'}.

[0185] Correctly kept: {'cat','sat'} (all kept are present in the reference).

[0186] Precision P = 2 / 2 = 1.0, Recall R = 2 / 3 ≈ 0.666.

[0187] F1-score = (2 × P × R) / (P + R) ≈ 0.8.

[0188] DELETE:

[0189] 1-grams to be deleted: those in the source text but not in the candidate text: {'the'}.

[0190] Correctly deleted: {'the'} (present in the reference but should be kept, so not a valid deletion).

[0191] P = 0 / 1 = 0, R = 0 / 1 = 0, F1-score = 0.

[0192] ADD:

[0193] New 1-grams: parts in the candidate text that are not in the source text: {'a'}.

[0194] Correctly added: {} (the added parts are not in the reference).

[0195] P = 0 / 1 = 0, R = 0 / 1 = 0, F1 score = 0.

[0196] Take the average of the F1 scores for each editing operation, and the result is used as the SARI score:

[0197] SARI = (0.8 + 0 + 0) / 3 ≈ 0.267.

[0198] The above introduced the calculation of the SARI score using 1-gram as an example. In addition, the value of n in n-gram can also be other values. Or, when the N value includes multiple values, the corresponding SARI score can be calculated for each N value, and finally, the average of the SARI scores for all N values is taken as the final SARI score to more comprehensively measure the quality of text changes, that is, to measure the noise robustness of the speech transcription system.

[0199] The SARI metric quantifies the ability of the speech transcription system to maintain semantic consistency under noise interference by analyzing the rationality of addition, deletion, and retention operations. The higher the SARI metric score, the stronger the noise robustness of the speech transcription system.

[0200] In some embodiments of the present application, the speech transcription evaluation system can also have the ability to evaluate the credibility of the noise robustness metric score obtained by machine evaluation (for example, the noise robustness metric score calculated using the SARI metric), that is, it can receive the calibration result of the noise robustness metric given by an expert based on the first transcription text, the second transcription text, and the reference text, and then compare the expert calibration result and the machine evaluation result to give a comparison result of the human-machine consistency effect. This comparison result can further verify whether the noise robustness metric score obtained by machine evaluation can directly replace the expert calibration. In a possible application scenario, before the speech transcription evaluation system of the present application is officially applied, the similarity between the noise robustness metric score obtained by machine evaluation and the result marked by the expert can be verified through the above processing process (such as the similarity exceeding a set threshold). If the similarity exceeds the set threshold, it means that the result of machine evaluation is reliable enough, and the speech transcription evaluation system can be applied to the evaluation stage; if the similarity does not exceed the set threshold, it means that the result of machine evaluation is not reliable enough, and the noise robustness metric evaluation algorithm can be further optimized until the evaluation result obtained by the machine based on the optimized noise robustness metric evaluation algorithm is reliable enough.

[0201] Refer to Figure 7 As shown, another evaluation process for the human-machine consistency effect of the speech transcription evaluation system is introduced. A possible implementation process may include:

[0202] S1. After the test audio passes through the speech transcription system (ASR system), the corresponding first transcription text is obtained, and the reference text corresponding to the test audio is obtained.

[0203] S2. The test audio is subjected to noise processing to obtain a loaded test audio, and a second transcribed text is obtained through the ASR system.

[0204] S3. A triplet is formed by the reference text, the first transcribed text, and the second transcribed text of the same test audio. A triplet set can be constructed for multiple test audios in a test case.

[0205] S4. Calculate the noise robustness index score for each triple in the triple set according to the SARI algorithm, and finally obtain a machine score vector C of the test case dimension.

[0206] S5. Provide the triplet set obtained in S3 to language experts for artificial noise robustness scoring, and finally obtain a test case dimension artificial scoring vector D.

[0207] S5. Calculate the similarity between vector C and vector D to obtain the evaluation result of the consistency of human-machine noise robustness score.

[0208] In this step, the similarity between two score vectors can be measured using the Pearson correlation coefficient.

[0209] The method provided in this embodiment compares the expert scoring results with the machine scoring results to obtain an evaluation result of the consistency of human and machine scoring (noise robustness index score). With reference to this evaluation result, the reliability of the noise robustness index evaluation algorithm (for example, the SARI algorithm) can be analyzed. When the algorithm is found to be unreliable, the noise robustness index evaluation algorithm can be adjusted in a timely manner to ensure that the noise robustness index score output by the machine is sufficiently reliable. Then, the speech transcription evaluation system can be applied to specific evaluation tasks to ensure the accuracy of the evaluation results of the speech transcription evaluation system.

[0210] The speech transcription system evaluation device provided in an embodiment of the present application is described below. The speech transcription system evaluation device described below and the speech transcription system evaluation method described above can be referenced to each other.

[0211] See also Figure 8 , Figure 8 This is a schematic diagram of the structure of a speech transcription system evaluation device disclosed in an embodiment of the present application.

[0212] like Figure 8 As shown, the device may include:

[0213] A test audio and reference text acquisition unit 11 is used to acquire the test audio and its corresponding reference text;

[0214] The first transcription text acquisition unit 12 is configured to acquire the first transcription text of the test audio, where the first transcription text is obtained by transcribing the test audio through a speech transcription system to be evaluated;

[0215] The semantic consistency index calculation unit 13 is configured to determine the semantic consistency between the first transcription text and the reference text, and obtain a semantic consistency index score;

[0216] The evaluation result determination unit 14 is configured to determine the evaluation result of the speech transcription system according to the score of the set evaluation index, where the set evaluation index at least includes the semantic consistency index.

[0217] In a possible implementation, the device of the present application further includes:

[0218] The noise-added audio acquisition unit is configured to acquire a noise-added test audio, where the noise-added test audio is an audio obtained by performing noise-adding processing on the test audio;

[0219] The second transcription text acquisition unit is configured to acquire the second transcription text of the noise-added test audio, where the second transcription text is obtained by transcribing the noise-added test audio through the speech transcription system;

[0220] The noise robustness index calculation unit is configured to determine a noise robustness index score based on the first transcription text, the second transcription text, and the reference text, where the noise robustness index is used to measure the stability of the transcription text of the audio before and after adding noise. On this basis, the evaluation index used by the evaluation result determination unit further includes the noise robustness index.

[0221] In a possible implementation, the device of the present application further includes:

[0222] The recognition rate index calculation unit is configured to calculate a recognition rate index score according to the reference text and the first transcription text, where the recognition rate index represents the correct rate of phonetic-to-character transcription. On this basis, the evaluation index used by the evaluation result determination unit further includes the recognition rate index.

[0223] In a possible implementation, the process in which the semantic consistency index calculation unit determines the semantic consistency between the first transcription text and the reference text and obtains a semantic consistency index score includes:

[0224] Invoking a large model to instruct the large model to evaluate the semantic consistency between the first transcription text and the reference text, and output a semantic consistency index score.

[0225] In a possible implementation, the process of the noise robustness metric calculation unit determining the noise robustness metric score based on the first transcription text, the second transcription text, and the reference text includes:

[0226] Using the first transcription text as the source text and the second transcription text as the generated text, calculate the SARI metric score based on the source text, the generated text, and the reference text as the noise robustness metric score.

[0227] In a possible implementation, the process of the recognition rate metric calculation unit calculating the recognition rate metric score according to the reference text and the first transcription text includes:

[0228] Perform edit distance alignment on the reference text and the first transcription text;

[0229] Calculate the correct recognition rate COR metric score and the accuracy ACC metric score based on the alignment result;

[0230] Integrate the COR metric score and the ACC metric score to obtain the recognition rate metric score.

[0231] In a possible implementation, the process of the recognition rate metric calculation unit integrating the COR metric score and the ACC metric score to obtain the recognition rate metric score includes:

[0232] Perform weighted addition on the COR metric score and the ACC metric score to obtain the recognition rate metric score;

[0233] When the test audio is from a short audio test set, the first weight corresponding to the COR metric is less than the second weight corresponding to the ACC metric, and the duration of the test audio included in the short audio test set is less than the first set duration threshold;

[0234] When the test audio is from a long audio test set, the first weight corresponding to the COR metric is greater than the second weight corresponding to the ACC metric, and the duration of the test audio included in the long audio test set is greater than the second set duration threshold.

[0235] An embodiment of the present application also provides an electronic device. Refer to Figure 9 As shown, it shows a schematic structural diagram of an electronic device suitable for implementing the electronic device in the embodiment of the present application. The electronic device in the embodiment of the present application may include, but is not limited to, terminals such as mobile phones, tablet computers, translators, servers, and the like. Figure 9 The electronic device shown is only an example and should not impose any limitation on the functions and usage scope of the embodiment of the present application.

[0236] As Figure 9As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which may perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 into the random access memory (RAM) 603, so as to implement the speech-to-text system evaluation method of the foregoing embodiments of the present application. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. The input / output (I / O) interface 605 is also connected to the bus 604.

[0237] Generally, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 9 an electronic device with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices may be implemented or had.

[0238] An embodiment of the present application also provides a computer program product, including computer-readable instructions, which when running on an electronic device, enable the electronic device to implement any speech-to-text system evaluation method provided by the embodiments of the present application.

[0239] An embodiment of the present application also provides a computer-readable storage medium, which carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, can enable the electronic device to implement any speech-to-text system evaluation method provided by the embodiments of the present application.

[0240] In addition, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that they have a communication connection, which can be specifically implemented as one or more communication buses or signal lines.

[0241] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware. Of course, it can also be implemented by dedicated hardware including application-specific integrated circuits, dedicated CPUs, dedicated memories, dedicated components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structures for implementing the same function can also be diverse, such as analog circuits, digital circuits, or dedicated circuits, etc. However, for the present application, in more cases, software program implementation is a better implementation manner. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a readable storage medium, such as a floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disc of a computer, etc., and includes several instructions for causing a computer device (which can be a personal computer, training device, or network device, etc.) to execute the methods described in various embodiments of the present application.

[0242] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0243] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, training device, or data center to another website, computer, training device, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store, or a data storage device such as a training device or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0244] The various embodiments in this specification are described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

Claims

1. A method for evaluating a speech transcription system, characterized in that, Including: Obtain a test audio and its corresponding reference text; Obtain a first transcription text of the test audio, where the first transcription text is obtained by transcribing the test audio through a speech transcription system to be evaluated; Determine the semantic consistency between the first transcription text and the reference text to obtain a semantic consistency index score; Determine the evaluation result of the speech transcription system according to the score of the set evaluation index, where the set evaluation index at least includes the semantic consistency index; Also including: Obtain a noise-added test audio, where the noise-added test audio is an audio obtained by adding noise to the test audio; Obtain a second transcription text of the noise-added test audio, where the second transcription text is obtained by transcribing the noise-added test audio through the speech transcription system; Taking the first transcription text as the source text and the second transcription text as the generated text, calculate the SARI index score based on the source text, the generated text and the reference text as the noise robustness index score, where the noise robustness index is used to measure the stability of the transcription text of the audio before and after adding noise; The set evaluation index also includes the noise robustness index.

2. The method according to claim 1, wherein Also including: Calculate an identification rate index score according to the reference text and the first transcription text, where the identification rate index represents the correct rate of phonetic-character transcription; The set evaluation index also includes the identification rate index.

3. The method according to claim 1, characterized in that, The process of determining the semantic consistency between the first transcription text and the reference text to obtain a semantic consistency index score includes: Call a large model to instruct the large model to evaluate the semantic consistency between the first transcription text and the reference text and output a semantic consistency index score.

4. The method according to claim 2, wherein The process of calculating an identification rate index score according to the reference text and the first transcription text includes: Perform edit distance alignment on the reference text and the first transcription text; Calculate the correct recognition rate COR index score and the accuracy ACC index score based on the alignment result; Integrate the COR index score and the ACC index score to obtain an identification rate index score.

5. The method according to claim 4, characterized in that, The process of integrating the COR index score and the ACC index score to obtain an identification rate index score includes: Perform weighted addition on the COR index score and the ACC index score to obtain an identification rate index score; When the test audio is from a short audio test set, the first weight corresponding to the COR index is less than the second weight corresponding to the ACC index, and the duration of the test audio included in the short audio test set is less than a first set duration threshold; When the test audio is from a long audio test set, the first weight corresponding to the COR index is greater than the second weight corresponding to the ACC index, and the duration of the test audio included in the long audio test set is greater than a second set duration threshold.

6. A speech-to-text system evaluation device, characterized in that, Including: A test audio and reference text acquisition unit for obtaining a test audio and its corresponding reference text; A first transcription text acquisition unit, configured to acquire a first transcription text of the test audio, where the first transcription text is obtained by transcribing the test audio through a speech transcription system to be evaluated; A semantic consistency index calculation unit, configured to determine the semantic consistency between the first transcription text and the reference text, and obtain a semantic consistency index score; An evaluation result determination unit, configured to determine an evaluation result of the speech transcription system according to the scores of the set evaluation metrics, where the set evaluation metrics at least include the semantic consistency index; Obtain a noise-added test audio, where the noise-added test audio is an audio obtained by performing noise addition processing on the test audio; Obtain a second transcription text of the noise-added test audio, where the second transcription text is obtained by transcribing the noise-added test audio through the speech transcription system; Using the first transcription text as the source text and the second transcription text as the generated text, calculate the SARI index score based on the source text, the generated text, and the reference text, and use it as the noise robustness index score.

7. An electronic device, characterized in that, Comprising: A memory and a processor; The memory is configured to store a program; The processor is configured to execute the program to implement each step of the speech transcription system evaluation method according to any one of claims 1 to 5.

8. A readable storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by the processor, each step of the speech transcription system evaluation method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, each step of the speech transcription system evaluation method according to any one of claims 1 to 5 is implemented.

Images