Speech annotation quality evaluation method and device, electronic equipment and storage medium

Abstract

The application provides a speech annotation quality evaluation method and device, electronic equipment and storage medium, the method comprises the steps of: determining the annotation text of the original speech; based on the annotation text, speech synthesis is carried out to obtain synthesized speech; the original speech is recognized to obtain the original speech text representation and the original recognition text; the synthesized speech is recognized to obtain the synthesized speech text representation and the synthesized recognition text; based on the feature similarity between the original speech text representation and the synthesized speech text representation, and / or the text edit distance between the original recognition text and the synthesized recognition text, the annotation quality evaluation result is determined. The speech annotation quality evaluation method, device, electronic equipment and storage medium provided by the application can accurately determine the annotation quality evaluation result, so as to quickly screen out unqualified annotation texts, greatly improve the checking efficiency of the annotation texts, and greatly save the labor and time cost.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and in particular to a method, apparatus, electronic device, and storage medium for evaluating the quality of speech annotation. Background Technology

[0002] In real-world speech recognition applications, models that perform well in general scenarios often experience a significant drop in performance when transferred to specific scenarios. This typically necessitates collecting and annotating corpora from these specific scenarios, and then iterating using models designed for general scenarios to achieve better recognition results. The quality of these annotations directly determines the effectiveness of model training; therefore, verifying the quality of speech annotations is crucial.

[0003] Currently, the quality of speech annotations is checked manually in two stages. The first stage is a full inspection, which requires checking all the annotated data. The second stage is a quality check, which randomly selects a certain percentage of the data from the full inspection for further inspection. Only data that passes both stages can be used for subsequent model training. However, this verification method is extremely labor-intensive and time-consuming, especially the first stage, which requires checking each annotated data item individually, making it particularly time-consuming. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for evaluating the quality of voice annotation, in order to solve the shortcomings of existing technologies that consume manpower and time in the quality verification of annotated data.

[0005] This invention provides a method for evaluating the quality of speech annotation, comprising:

[0006] Determine the annotated text of the original speech;

[0007] Speech synthesis is performed based on the labeled text to obtain synthesized speech;

[0008] The original speech is subjected to speech recognition to obtain the original speech text representation and the original recognized text.

[0009] The synthesized speech is subjected to speech recognition to obtain a synthesized speech text representation and a synthesized recognized text;

[0010] The annotation quality evaluation result is determined based on the feature similarity between the original speech text representation and the synthesized speech text representation, and / or the text edit distance between the original recognized text and the synthesized recognized text.

[0011] According to a speech annotation quality evaluation method provided by the present invention, the step of performing speech recognition on the original speech to obtain the original speech text representation and the original recognized text includes:

[0012] The acoustic features of the original speech are input into the speech recognition model to obtain the original speech text representation and the original recognized text output by the speech recognition model.

[0013] The speech recognition model is trained based on the acoustic features of sample speech and the sample labeled text; the speech recognition model is trained based on the differences between the sample labeled text and the corresponding sample speech text representations, as well as the differences between the sample recognized text and the sample labeled text.

[0014] The process of performing speech recognition on the synthesized speech to obtain a synthesized speech text representation and synthesized recognized text includes:

[0015] The acoustic features of the synthesized speech are input into the speech recognition model to obtain the synthesized speech text representation and the synthesized recognized text output by the speech recognition model.

[0016] According to a speech annotation quality evaluation method provided by the present invention, the training steps of the speech recognition model include:

[0017] The acoustic features of the sample speech are input into the initial model of the speech recognition model to obtain the sample speech text representation and the sample recognition text output by the initial model;

[0018] Based on the differences between the speech text representations of samples with the same labeled text and / or the differences between the speech text representations of samples with different labeled text, as well as the differences between the labeled text and the recognized text, the initial model is iterated to obtain the speech recognition model.

[0019] According to the speech annotation quality evaluation method provided by the present invention, the sample speech includes original sample speech and synthesized sample speech, wherein the synthesized sample speech is obtained by speech synthesis of the sample annotation text;

[0020] The differences between the sample speech text representations corresponding to the different sample labeled texts include the differences between the original speech text representations corresponding to the different sample labeled texts and / or the differences between the synthesized speech text representations corresponding to the different sample labeled texts;

[0021] The differences between the speech text representations of the same labeled text and the corresponding samples include at least one of the following: the differences between the original speech text representations of the same labeled text and the corresponding synthesized speech text representations of the same labeled text and the corresponding original speech text representations of the same labeled text and the corresponding synthesized speech text representations of the same labeled text.

[0022] According to a speech annotation quality evaluation method provided by the present invention, the step of inputting the acoustic features of the original speech into a speech recognition model to obtain the original speech text representation and the original recognized text output by the speech recognition model includes:

[0023] The acoustic features of the original speech are input into the first coding layer of the speech recognition model to obtain the original coding features output by the first coding layer;

[0024] The original encoded features are input into the attention layer of the speech recognition model to obtain the attention features output by the attention layer;

[0025] The attention features are input into the second coding layer of the speech recognition model to obtain the original speech-text representation output by the second coding layer;

[0026] The attention features are input into the decoding layer of the speech recognition model to obtain the original recognized text output by the decoding layer.

[0027] According to a speech annotation quality evaluation method provided by the present invention, the step of determining the text edit distance includes:

[0028] Determine the number of replacement operations, insertion operations, and deletion operations corresponding to the conversion of the original recognized text into the synthesized recognized text;

[0029] The text editing distance is determined based on the number of replacement operations, the number of insertion operations, and the number of deletion operations.

[0030] According to a speech annotation quality evaluation method provided by the present invention, determining the text edit distance based on the number of replacement operations, the number of insertion operations, and the number of deletion operations includes:

[0031] Based on the number of replacement operations, determine the first edit distance;

[0032] The second edit distance is determined based on the number of insertion operations;

[0033] The third edit distance is determined based on the number of deletion operations;

[0034] The average of the first edit distance, the second edit distance, and the third edit distance is taken as the text edit distance.

[0035] The present invention also provides a speech annotation quality evaluation device, comprising:

[0036] The text determination unit is used to determine the annotated text of the original speech.

[0037] A speech synthesis unit is used to perform speech synthesis based on the labeled text to obtain synthesized speech;

[0038] The first recognition unit is used to perform speech recognition on the original speech to obtain the original speech text representation and the original recognized text.

[0039] The second recognition unit is used to perform speech recognition on the synthesized speech to obtain a synthesized speech text representation and a synthesized recognized text.

[0040] The quality evaluation unit is used to determine the annotation quality evaluation result based on the feature similarity between the original speech text representation and the synthesized speech text representation, and / or the text edit distance between the original recognized text and the synthesized recognized text.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the speech annotation quality evaluation method as described above.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech annotation quality evaluation method as described above.

[0043] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the speech annotation quality evaluation method as described above.

[0044] The speech annotation quality evaluation method, apparatus, electronic device, and storage medium provided by this invention, based on the feature similarity between the original speech text representation and the synthesized speech text representation, and / or the text edit distance between the original recognized text and the synthesized recognized text, can accurately determine the annotation quality evaluation result, thereby achieving accurate quality evaluation of the annotated text. This allows for the rapid screening of unqualified annotated text, greatly improving the efficiency of annotated text verification, while also significantly saving manpower and time costs. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the speech annotation quality evaluation method provided by the present invention;

[0047] Figure 2 This is a flowchart illustrating the speech recognition model training method provided by the present invention;

[0048] Figure 3 This is a flowchart illustrating the implementation of step 130 in the speech annotation quality evaluation method provided by the present invention;

[0049] Figure 4 This is a schematic diagram of the structure of the speech recognition model provided by the present invention;

[0050] Figure 5 This is a flowchart illustrating the text editing distance determination method provided by the present invention;

[0051] Figure 6 This is a schematic diagram of the voice annotation quality evaluation device provided by the present invention;

[0052] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] With breakthroughs in deep learning technology in speech recognition, continuous speech recognition technology has been widely applied in various industries such as education, entertainment, healthcare, and transportation, and its effectiveness has been widely recognized by the industry. However, since continuous speech recognition technology is a typical data-driven, supervised learning pattern recognition technology, the quantity and quality of the training data directly affect the system's recognition performance. Due to differences in industry sectors, the data to be recognized for continuous speech recognition tasks in the same language varies significantly, including factors such as channel, text topic, speaker, and environmental noise. These objective differences make it difficult to establish a universal continuous speech recognition system applicable to all industries. In real-world speech recognition applications, models that perform well in certain general scenarios show a significant drop in recognition performance when transferred to specific scenarios. It is usually necessary to collect and annotate corpora for these specific scenarios, and then apply general-scenario models for iterative training. The quality of these annotations directly determines the effectiveness of model training; therefore, verifying the quality of speech annotations is crucial.

[0055] Currently, the quality of voice annotations is checked manually in two stages. The first stage is a full inspection, which requires checking all the annotated data. The second stage is a quality inspection, which randomly selects a certain percentage of the data from the full inspection for further review. Only data that passes both stages is considered acceptable, thus effectively ensuring the quality of the annotated data. However, this verification method is particularly labor-intensive and time-consuming, especially the first stage, which requires verifying each annotated data entry individually, making it extremely time-consuming.

[0056] In response, this invention provides a method for evaluating the quality of speech annotation. Figure 1 This is a flowchart illustrating the speech annotation quality evaluation method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps:

[0057] Step 110: Determine the annotated text of the original speech.

[0058] Here, the original speech can be used as training sample speech data for the speech recognition model. Specifically, it can be pre-collected by a recording device or recorded in real time; this embodiment of the invention does not impose specific limitations on this. After obtaining the original speech, speech annotation processing is performed on the original speech to obtain the annotated text of the original speech.

[0059] Step 120: Perform speech synthesis based on the labeled text to obtain synthesized speech.

[0060] Specifically, synthesized speech refers to speech data obtained by combining speech data from annotated text. When synthesizing speech based on annotated text, it is possible to synthesize only a single speed-incremented speech or multiple speed-incremented speech; this embodiment of the invention does not specifically limit this.

[0061] Step 130: Perform speech recognition on the original speech to obtain the original speech text representation and the original recognized text;

[0062] Step 140: Perform speech recognition on the synthesized speech to obtain the synthesized speech text representation and the synthesized recognized text.

[0063] Specifically, the original speech-text representation can represent the text content information corresponding to the original speech, while the original recognized text refers to the text obtained after speech recognition of the original speech. When performing speech recognition on the original speech, a speech recognition model can be used to perform speech recognition on the original speech, resulting in the original speech-text representation and the original recognized text.

[0064] Similarly, synthesized speech text representation can represent the text content information corresponding to synthesized speech, while synthesized recognized text refers to the text obtained after speech recognition of synthesized speech. When performing speech recognition on synthesized speech, a speech recognition model can be used to perform speech recognition on the synthesized speech, resulting in both synthesized speech text representation and synthesized recognized text.

[0065] Step 150: Determine the annotation quality evaluation result based on the feature similarity between the original speech text representation and the synthesized speech text representation, and / or the text edit distance between the original recognized text and the synthesized recognized text.

[0066] Specifically, the annotation quality evaluation results can characterize the accuracy of the annotated text, while synthesized speech is obtained by synthesizing speech from the annotated text. The higher the content similarity between the original speech and the synthesized speech, the higher the accuracy of the annotated text corresponding to the synthesized speech; the lower the content similarity between the original speech and the synthesized speech, the lower the accuracy of the annotated text corresponding to the synthesized speech.

[0067] Furthermore, feature similarity can characterize the text content relevance between the original speech and the synthesized speech. The greater the text content relevance, the higher the content similarity between the original speech and the synthesized speech, and thus the higher the accuracy of the annotated text.

[0068] Furthermore, text edit distance characterizes the degree of text difference between the original recognized text and the synthesized recognized text. The smaller the text difference, the higher the similarity between the original and synthesized recognized text. In other words, the higher the content similarity between the original speech corresponding to the original recognized text and the synthesized speech corresponding to the synthesized recognized text, the higher the accuracy of text annotation. Text edit distance, also known as Levenshtein distance, is a quantitative measure of the degree of difference between two strings. It measures the minimum number of processing steps required to transform one string into another, including character replacement, deletion, and insertion.

[0069] It is understood that the annotation quality evaluation result can be a specific quality score or the quality level to which the annotated text belongs, and this embodiment of the invention does not impose any specific limitations on this. Furthermore, based on the annotation quality evaluation result, it can be determined whether the annotated text is qualified. If it is qualified, it can be used for subsequent iterative training of the speech recognition model; if it is unqualified, the sample speech can be re-annotated or the annotated text can be corrected.

[0070] The speech annotation quality evaluation method provided in this invention, based on the feature similarity between the original speech text representation and the synthesized speech text representation, and / or the text edit distance between the original recognized text and the synthesized recognized text, can accurately determine the annotation quality evaluation result, thereby achieving accurate quality evaluation of the annotated text. This allows for the rapid screening of unqualified annotated text, greatly improving the efficiency of annotated text verification, while also significantly saving manpower and time costs.

[0071] As an optional implementation, feature similarity can be represented by the cosine similarity between the original speech text representation and the synthesized speech text representation, that is, feature similarity can be determined by the following formula:

[0072]

[0073] Where cos_sim represents feature similarity, test_text_vector1 represents the original speech-text representation, and test_text_vector2 represents the synthesized speech-text representation. The larger cos_sim is, the higher the semantic similarity between the original speech-text representation and the synthesized speech-text representation, that is, the higher the similarity of the pronunciation sequences between the original speech and the synthesized speech, and thus the higher the accuracy of the annotated text.

[0074] The quality score corresponding to the quality evaluation result can be determined using the following formula:

[0075] score=cos_sim-α·Dist_edit_avg

[0076] Where score represents quality score, Dist_edit_avg represents text edit distance, and α represents adjustment factor, which can be set according to actual conditions.

[0077] It is understood that the embodiments of the present invention can set a threshold score_thred. When score > score_thred, it indicates that the accuracy of the labeled text is high, that is, the labeled text is qualified; when score ≤ score_thred, it indicates that the accuracy of the labeled text is low, that is, the labeled text is unqualified, and it can be re-labeled.

[0078] Based on the above embodiments, step 130 includes:

[0079] The acoustic features of the original speech are input into the speech recognition model to obtain the original speech text representation and the original recognized text output by the speech recognition model.

[0080] The speech recognition model is trained based on the acoustic features of the sample speech and the sample labeled text; the speech recognition model is trained based on the differences between the sample labeled text and the corresponding sample speech text representations, as well as the differences between the sample recognized text and the sample labeled text.

[0081] Step 140 includes:

[0082] The acoustic features of the synthesized speech are input into the speech recognition model to obtain the synthesized speech text representation and the synthesized recognized text output by the speech recognition model.

[0083] Specifically, the acoustic features corresponding to the original speech and the synthesized speech can be LPC (Linear Prediction Coefficient), MFCCs (Mel Frequency Cepstral Coefficients), or PLP (Perceptual Linear Predictive), or any combination thereof. This embodiment of the invention does not impose specific limitations on these features.

[0084] After determining the acoustic features corresponding to the original speech and the synthesized speech, a speech recognition model is used to perform speech recognition, resulting in the original recognized text corresponding to the original speech and the original speech text representation representing the semantic information of the original speech, as well as the synthesized recognized text corresponding to the synthesized speech and the synthesized speech text representation representing the semantic information of the synthesized speech.

[0085] The speech recognition model is trained based on the differences between the speech text representations corresponding to the labeled sample text, and the differences between the recognized sample text and the labeled sample text. The differences between the speech text representations corresponding to the labeled sample text can represent the differences between speech text representations corresponding to the same labeled sample text, as well as the differences between speech text representations corresponding to different labeled sample texts. In other words, when training the speech recognition model based on the differences between the speech text representations corresponding to the labeled sample text, it can minimize the distance between speech text representations corresponding to the same labeled sample text and maximize the distance between speech text representations corresponding to different labeled sample texts. This avoids interference from speech rate, intonation, prosody, and gender between different sample speech on the accuracy of extracting speech text representations, enabling the trained speech recognition model to accurately extract the corresponding speech text representations. Simultaneously, training the speech recognition model based on the differences between the recognized sample text and the labeled sample text allows the trained speech recognition model to accurately acquire the corresponding recognized text.

[0086] Therefore, the speech recognition model trained based on the differences between the sample annotation text and the corresponding sample speech text representations, as well as the differences between the sample recognition text and the sample annotation text in the embodiments of the present invention, can not only accurately extract the corresponding speech text representations, but also accurately obtain the corresponding recognition text.

[0087] Based on any of the above embodiments Figure 2 This is a flowchart illustrating the speech recognition model training method provided by the present invention, as shown below. Figure 2 As shown, the training steps for a speech recognition model include:

[0088] Step 210: Input the acoustic features of the sample speech into the initial model of the speech recognition model to obtain the sample speech text representation and sample recognition text output by the initial model;

[0089] Step 220: Based on the differences between the speech text representations of the same sample labeled text and / or the differences between the speech text representations of different sample labeled texts, as well as the differences between the sample labeled text and the sample recognized text, perform parameter iteration on the initial model to obtain the speech recognition model.

[0090] Specifically, there may be differences in speech rate, intonation, prosody, gender, etc., between sample speech corresponding to the same labeled text. In order to avoid the influence of these factors on the final extracted sample speech text representation, the embodiments of the present invention train the speech recognition model based on the differences between the sample speech text representations corresponding to the labeled text. This can minimize the distance between the sample speech text representations corresponding to the same labeled text and maximize the distance between the sample speech text representations corresponding to different labeled texts. This avoids the interference of speech rate, intonation, prosody, and gender between different sample speech on the accuracy of the extracted sample speech text representation. In other words, it enables the trained speech recognition model to accurately extract the corresponding speech text representation.

[0091] Meanwhile, training the speech recognition model based on the difference between the sample recognition text and the sample annotation text enables the trained speech recognition model to accurately obtain the corresponding recognition text.

[0092] Therefore, the speech recognition model trained based on the differences between the sample annotation text and the corresponding sample speech text representations, as well as the differences between the sample recognition text and the sample annotation text in the embodiments of the present invention, can not only accurately extract the corresponding speech text representations, but also accurately obtain the corresponding recognition text.

[0093] Based on any of the above embodiments, the sample speech includes the original sample speech and the synthesized sample speech, which is obtained by synthesizing speech from the labeled sample text.

[0094] The differences between the speech text representations of different sample labeled texts include the differences between the original speech text representations of different sample labeled texts and / or the differences between the synthesized speech text representations of different sample labeled texts;

[0095] The differences between the speech text representations of the same labeled text and the corresponding samples include at least one of the following: the differences between the original speech text representations of the same labeled text and the corresponding synthesized speech text representations of the same labeled text and the corresponding original speech text representations of the same labeled text and the corresponding synthesized speech text representations of the same labeled text.

[0096] Specifically, training a speech recognition model based on the differences between the original speech text representations corresponding to different labeled text samples can maximize the distance between these representations, enabling the model to accurately learn the differences between them. Similarly, training a speech recognition model based on the differences between the synthesized speech text representations corresponding to different labeled text samples can also maximize the distance between these representations, further enabling the model to accurately learn the differences between them.

[0097] Training a speech recognition model based on the differences between the original speech text representations of corresponding samples with the same labeled text minimizes the distance between these representations, enabling the model to accurately learn the shared information within each representation. Similarly, training a speech recognition model based on the differences between the synthesized speech text representations of corresponding samples with the same labeled text minimizes the distance between these representations, allowing the model to accurately learn the shared information within each representation. Finally, training a speech recognition model based on the differences between the original and synthesized speech text representations of corresponding samples with the same labeled text minimizes the distance between these representations, enabling the model to accurately learn the shared information between both.

[0098] Optionally, when the sample speech includes both original sample speech and synthesized sample speech, the loss function Loss corresponding to the speech recognition model is:

[0099] Loss=CTCLoss1+CTCLoss2+TripletLoss+MMDLoss

[0100] Among them, CTCLoss1 is determined based on the difference between the original recognized text and the labeled text of the sample; CTCLoss2 is determined based on the difference between the synthesized recognized text and the labeled text of the sample; TripletLoss is determined based on the difference between the sample speech text representations corresponding to different labeled texts, the difference between the original speech text representations corresponding to the same labeled text, and the difference between the synthesized speech text representations corresponding to the same labeled text; MMDLoss is determined based on the difference between the original speech text representations and the synthesized speech text representations corresponding to the same labeled text.

[0101] Based on any of the above embodiments Figure 3 This is a flowchart illustrating the implementation of step 130 in the speech annotation quality evaluation method provided by the present invention, as shown below. Figure 3 As shown, in step 130, the acoustic features of the original speech are input into the speech recognition model to obtain the original speech text representation and the original recognized text output by the speech recognition model, including:

[0102] Step 131: Input the acoustic features of the original speech into the first coding layer of the speech recognition model to obtain the original coding features output by the first coding layer;

[0103] Step 132: Input the original encoded features into the attention layer of the speech recognition model to obtain the attention features output by the attention layer;

[0104] Step 133: Input the attention features into the second coding layer of the speech recognition model to obtain the original speech-text representation output by the second coding layer;

[0105] Step 134: Input the attention features into the decoding layer of the speech recognition model to obtain the original recognized text output by the decoding layer.

[0106] Specifically, the speech recognition model includes a first encoding layer, an attention layer, a second encoding layer, and a decoding layer. Figure 4 This is a schematic diagram of the speech recognition model provided by the present invention, as shown below. Figure 4 As shown, the first encoding layer encodes the acoustic features of the original speech to obtain the original encoded features. Then, the attention layer performs attention transformation on the original encoded features to obtain attention features. Subsequently, the second encoding layer can encode the attention features to obtain the original speech text representation that can characterize the semantic information of the original speech. Finally, the decoding layer decodes the attention features to obtain the original recognized text.

[0107] Based on any of the above embodiments Figure 5It is a schematic flowchart of the method for determining the text edit distance provided by the present invention. As Figure 5 shown, the steps for determining the text edit distance include:

[0108] Step 510: Determine the number of replacement operations, insertion operations, and deletion operations corresponding to converting the original recognized text into the synthesized recognized text;

[0109] Step 520: Determine the text edit distance based on the number of replacement operations, insertion operations, and deletion operations.

[0110] Specifically, there may be differences between the strings corresponding to the original recognized text and the synthesized recognized text respectively. If the original recognized text is to be converted into the synthesized recognized text, the strings corresponding to the original recognized text and the synthesized recognized text can be made consistent through replacement operations, insertion operations, and deletion operations. For example, if the original recognized text is "It's sunny today" and the synthesized recognized text is "It's really sunny today", then when converting the original recognized text into the synthesized recognized text, the "good" in the original recognized text needs to be replaced with "really", that is, the number of replacement operations is 1, the number of insertion operations is 0, and the number of deletion operations is 0.

[0111] Based on the number of replacement operations, insertion operations, and deletion operations, the edit distance between the original recognized text and the synthesized recognized text can be determined, that is, the text edit distance. The smaller the text edit distance, the smaller the text difference degree between the original recognized text and the synthesized recognized text, that is, the higher the similarity between the original recognized text and the synthesized recognized text. Therefore, the content similarity between the original speech corresponding to the original recognized text and the synthesized speech corresponding to the synthesized recognized text is higher, and thus the accuracy of the labeled text is higher.

[0112] Optionally, the embodiment of the present invention can determine the text edit distance based on the sum of the number of replacement operations, insertion operations, and deletion operations, or can also determine the text edit distance based on the average value of the number of replacement operations, insertion operations, and deletion operations. The embodiment of the present invention does not make specific limitations on this.

[0113] Based on any of the above embodiments, step 520 includes:

[0114] Determine the first edit distance based on the number of replacement operations;

[0115] Determine the second edit distance based on the number of insertion operations;

[0116] Determine the third edit distance based on the number of deletion operations;

[0117] Take the average value of the first edit distance, the second edit distance, and the third edit distance as the text edit distance.

[0118] Specifically, when the character lengths of the original recognized text and the synthesized recognized text are the same, the influence of the character length can be ignored, and the text edit distance is determined based on the sum of the first edit distance, the second edit distance, and the third edit distance. When the character lengths of the original recognized text and the synthesized recognized text are different, the character length will affect the accuracy of the text edit distance. For example, if the original recognized text is "It is sunny today" and the synthesized recognized text is "It is very sunny today啊啊啊啊", then when converting the original recognized text into the synthesized recognized text, 4 "啊" in the original recognized text need to be deleted, that is, the deletion operation is performed 4 times, and "很" needs to be inserted, that is, the insertion operation is performed 1 time. However, in fact, the similarity between the original recognized text and the synthesized recognized text is relatively high at this time. If the text edit distance is determined based on the sum of the first edit distance, the second edit distance, and the third edit distance, it may cause errors and reduce the accuracy of the text edit distance.

[0119] In this regard, the embodiment of the present invention uses the average value of the first edit distance, the second edit distance, and the third edit distance as the text edit distance, so as to avoid the influence of the character length on the accuracy of the text edit distance. Among them, the first edit distance can be the number of substitution operations, the second edit distance can be the number of insertion operations, and the third edit distance can be the number of deletion operations.

[0120] Optionally, the text edit distance can be determined based on the following formula:

[0121] Dist_edit_avg = 2 * Dist_edit / (N1 + N2)

[0122] Where, Dist_edit_avg represents the text edit distance, N1 represents the number of characters in the original recognized text, and N2 represents the number of characters in the synthesized recognized text.

[0123] Based on any of the above embodiments, the voice annotation quality evaluation method provided by the present invention specifically includes two stages: model training and voice annotation quality evaluation. Specifically as follows:

[0124] I. Model training stage:

[0125] S1. Collect open-source continuous sample voice annotation data sets in multiple languages.

[0126] S2. Label the sample texts in S1, and use a synthesis algorithm to synthesize the corresponding sample synthesized voices. Denote the i-th sample original voice corresponding to the labeled text as x_real_i, and its corresponding sample synthesized voice as x_synthesis_i.

[0127] S3. For the sample speech in S1 and S2, filter out invalid sounds and extract acoustic features (Filer Bank, FB features). Let the FB feature corresponding to the labeled text of the i-th sample be fb_i, and the FB feature of the synthesized speech of the corresponding sample be fb_synthesis_i.

[0128] S4. Initialize the parameters of the first speech recognition model (LSA_TEXT_REPRESENTATION_Net1) and the second speech recognition model (LSA_TEXT_REPRESENTATION_Net2). For the FB features in S3, send one batch of fb_i to LSA_TEXT_REPRESENTATION_Net1 at a time, and simultaneously send one batch of fb_synthesis_i to LSA_TEXT_REPRESENTATION_Net2. Jointly train the two models using CTCLOss, TripletLoss, and MMDLoss. The loss function for joint training is shown below:

[0129] Loss=CTCLoss1+CTCLoss2+TripletLoss+MMDLoss

[0130] Specifically, CTCLoss1 operates on LSA_TEXT_REPRESENTATION_Net1, and CTCLoss2 operates on LSA_TEXT_REPRESENTATION_Net2. TripLetLoss ensures that the text representations extracted by the two models are closer when the labeled text content is the same, and farther apart when the labeled text content is different. MMDLoss is used to remove interference from the synthesized speech and the original speech due to differences in speech rate, intonation, prosody, and gender on the text representation.

[0131] The LSA_TEXT_REPRESENTATION_Net1 and LSA_TEXT_REPRESENTATION_Net2 have the same structure, both including a first encoding layer, an attention layer, a second encoding layer, and a decoding layer. The first encoding layer encodes the acoustic features to obtain encoded features; the attention layer performs attention transformation on the encoded features to obtain attention features; the second encoding layer encodes the attention features to obtain the text representation; and the decoding layer decodes the attention features to obtain the recognized text. The second encoding layer can be a CNN (Convolutional Neural Network) structure, used to transform the attention features output by the attention layer into a fixed-dimensional vector.

[0132] S5. Repeat S4 until the loss is stable or the maximum number of iterations N is reached. At this time, the LSA_TEXT_REPRESENTATION_Net1 and LSA_TEXT_REPRESENTATION_Net2 obtained can be used to determine whether the labeled text meets the standard.

[0133] II. Voice annotation quality evaluation stage:

[0134] S6. The original speech to be checked is test_real, and its corresponding labeled text is text_test. A speech synthesis tool is used to synthesize text_test into the synthesized speech test_synthesis.

[0135] S7. Extract FB features from the original speech and synthesized speech in S6, denoted as test_fb_real and test_fb_synthesis, respectively.

[0136] S8. Load LSA_TEXT_REPRESENTATION_Net1 from training phase S5, and extract the original speech text representation test_text_vector1 and the original recognized text test_text_rec1 from test_fb_real in S7.

[0137] S9. Load LSA_TEXT_REPRESENTATION_Net2 from training phase S5, and extract the synthesized speech-text representation test_text_vector2 and the synthesized recognition test_text_rec2 from test_fb_synthesis in S7.

[0138] S10. Calculate the cosine similarity cos_sim between test_text_vector1 and test_text_vector2.

[0139] S11. Calculate the text edit distance between test_text_rec1 and test_text_rec2, denoted as Dist_edit. Generally, with the same number of characters, the closer the recognition results are, the smaller the edit distance. Considering the influence of character length on the edit distance, we calculate an average edit distance, denoted as Dist_edit_avg, as shown in the following formula:

[0140] Dist_edit_avg=2*Dist_edit / (N1+N2)

[0141] Where N1 and N2 represent the number of words in test_text_rec1 and test_text_rec2, respectively.

[0142] S12. Based on cos_sim in S10 and Dist_edit_avg in S11, calculate the annotation consistency score (this score is used to characterize the annotation quality evaluation result), as shown in the following formula:

[0143] score=cos_sim-α·Dist_edit_avg

[0144] In the formula, α is an adjustment factor, which is generally small, so that Dist_edit_avg serves as a supplement to cos_sim. Its value can be determined according to the actual situation.

[0145] S13. Define a threshold value, score_thred. Determine whether the final annotation is qualified based on the threshold value. That is, if the score is greater than score_thred, the annotation is considered qualified; otherwise, the annotation is considered unqualified and needs to be sent for inspection and re-annotated.

[0146] Through the above steps, the quality evaluation of the annotated text is completed. It can quickly screen out unqualified continuous speech annotation data from a batch of original annotation data, correct the unqualified annotation data, and the qualified annotation data does not need to be modified. This greatly speeds up the inspection process of annotation data and saves a lot of manpower and time costs.

[0147] The speech annotation quality evaluation device provided by the present invention is described below. The speech annotation quality evaluation device described below can be referred to in correspondence with the speech annotation quality evaluation method described above.

[0148] Based on any of the above embodiments Figure 6 This is a schematic diagram of the speech annotation quality evaluation device provided by the present invention, as shown below. Figure 6 As shown, the device includes:

[0149] The text determination unit 610 is used to determine the annotated text of the original speech;

[0150] The speech synthesis unit 620 is used to perform speech synthesis based on the labeled text to obtain synthesized speech;

[0151] The first recognition unit 630 is used to perform speech recognition on the original speech to obtain the original speech text representation and the original recognized text.

[0152] The second recognition unit 640 is used to perform speech recognition on the synthesized speech to obtain a synthesized speech text representation and a synthesized recognized text.

[0153] The quality evaluation unit 650 is used to determine the annotation quality evaluation result based on the feature similarity between the original speech text representation and the synthesized speech text representation, and / or the text edit distance between the original recognized text and the synthesized recognized text.

[0154] Based on any of the above embodiments, the step of performing speech recognition on the original speech to obtain the original speech-text representation and the original recognized text includes:

[0155] The acoustic features of the original speech are input into the speech recognition model to obtain the original speech text representation and the original recognized text output by the speech recognition model.

[0156] The speech recognition model is trained based on the acoustic features of sample speech and the sample labeled text; the speech recognition model is trained based on the differences between the sample labeled text and the corresponding sample speech text representations, as well as the differences between the sample recognized text and the sample labeled text.

[0157] The process of performing speech recognition on the synthesized speech to obtain a synthesized speech text representation and synthesized recognized text includes:

[0158] The acoustic features of the synthesized speech are input into the speech recognition model to obtain the synthesized speech text representation and the synthesized recognized text output by the speech recognition model.

[0159] Based on any of the above embodiments, the training steps of the speech recognition model include:

[0160] The acoustic features of the sample speech are input into the initial model of the speech recognition model to obtain the sample speech text representation and the sample recognition text output by the initial model;

[0161] Based on the differences between the speech text representations of samples with the same labeled text and / or the differences between the speech text representations of samples with different labeled text, as well as the differences between the labeled text and the recognized text, the initial model is iterated to obtain the speech recognition model.

[0162] Based on any of the above embodiments, the sample speech includes the original sample speech and the synthesized sample speech, wherein the synthesized sample speech is obtained by synthesizing the sample labeled text.

[0163] The differences between the sample speech text representations corresponding to the different sample labeled texts include the differences between the original speech text representations corresponding to the different sample labeled texts and / or the differences between the synthesized speech text representations corresponding to the different sample labeled texts;

[0164] The differences between the speech text representations of the same labeled text and the corresponding samples include at least one of the following: the differences between the original speech text representations of the same labeled text and the corresponding synthesized speech text representations of the same labeled text and the corresponding original speech text representations of the same labeled text and the corresponding synthesized speech text representations of the same labeled text.

[0165] Based on any of the above embodiments, the step of inputting the acoustic features of the original speech into a speech recognition model to obtain the original speech text representation and the original recognized text output by the speech recognition model includes:

[0166] The acoustic features of the original speech are input into the first coding layer of the speech recognition model to obtain the original coding features output by the first coding layer;

[0167] The original encoded features are input into the attention layer of the speech recognition model to obtain the attention features output by the attention layer;

[0168] The attention features are input into the second coding layer of the speech recognition model to obtain the original speech-text representation output by the second coding layer;

[0169] The attention features are input into the decoding layer of the speech recognition model to obtain the original recognized text output by the decoding layer.

[0170] Based on any of the above embodiments, the step of determining the text editing distance includes:

[0171] Determine the number of replacement operations, insertion operations, and deletion operations corresponding to the conversion of the original recognized text into the synthesized recognized text;

[0172] The text editing distance is determined based on the number of replacement operations, the number of insertion operations, and the number of deletion operations.

[0173] Based on any of the above embodiments, determining the text editing distance based on the number of replacement operations, the number of insertion operations, and the number of deletion operations includes:

[0174] Based on the number of replacement operations, determine the first edit distance;

[0175] The second edit distance is determined based on the number of insertion operations;

[0176] The third edit distance is determined based on the number of deletion operations;

[0177] The average of the first edit distance, the second edit distance, and the third edit distance is taken as the text edit distance.

[0178] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7 As shown, the electronic device may include a processor 710, a memory 720, a communications interface 730, and a communications bus 740, wherein the processor 710, the memory 720, and the communications interface 730 communicate with each other via the communications bus 740. The processor 710 can call logical instructions in the memory 720 to execute a speech annotation quality evaluation method. This method includes: determining the annotated text of the original speech; performing speech synthesis based on the annotated text to obtain synthesized speech; performing speech recognition on the original speech to obtain an original speech text representation and an original recognized text; performing speech recognition on the synthesized speech to obtain a synthesized speech text representation and a synthesized recognized text; and determining the annotation quality evaluation result based on the feature similarity between the original speech text representation and the synthesized speech text representation, and / or the text edit distance between the original recognized text and the synthesized recognized text.

[0179] Furthermore, the logical instructions in the aforementioned memory 720 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0180] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the speech annotation quality evaluation method provided by the above methods, the method comprising: determining the annotation text of the original speech; performing speech synthesis based on the annotation text to obtain synthesized speech; performing speech recognition on the original speech to obtain an original speech text representation and an original recognized text; performing speech recognition on the synthesized speech to obtain a synthesized speech text representation and a synthesized recognized text; and determining the annotation quality evaluation result based on the feature similarity between the original speech text representation and the synthesized speech text representation, and / or the text edit distance between the original recognized text and the synthesized recognized text.

[0181] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned speech annotation quality evaluation methods. The method includes: determining the annotation text of the original speech; performing speech synthesis based on the annotation text to obtain synthesized speech; performing speech recognition on the original speech to obtain an original speech text representation and an original recognized text; performing speech recognition on the synthesized speech to obtain a synthesized speech text representation and a synthesized recognized text; and determining an annotation quality evaluation result based on the feature similarity between the original speech text representation and the synthesized speech text representation, and / or the text edit distance between the original recognized text and the synthesized recognized text.

[0182] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for evaluating the quality of speech annotation, characterized in that, include: Determine the annotated text of the original speech; Speech synthesis is performed based on the labeled text to obtain synthesized speech; The original speech is subjected to speech recognition to obtain the original speech text representation and the original recognized text. The synthesized speech is subjected to speech recognition to obtain a synthesized speech text representation and a synthesized recognized text; The annotation quality evaluation result is determined based on the feature similarity between the original speech text representation and the synthesized speech text representation, and / or the text edit distance between the original recognized text and the synthesized recognized text; The speech recognition is performed by a speech recognition model, which is trained based on the differences between the sample labeled text and the corresponding sample speech text representations, as well as the differences between the sample recognized text and the sample labeled text. The sample speech includes the original sample speech and the sample synthesized speech.

2. The speech annotation quality evaluation method according to claim 1, characterized in that, The process of performing speech recognition on the original speech to obtain the original speech text representation and the original recognized text includes: The acoustic features of the original speech are input into the speech recognition model to obtain the original speech text representation and the original recognized text output by the speech recognition model. The speech recognition model is trained based on the acoustic features of sample speech and sample labeled text; the speech recognition model is trained based on the differences between the sample labeled text and the corresponding sample speech text representations, as well as the differences between the sample recognized text and the sample labeled text. The synthesized speech is subjected to speech recognition to obtain a synthesized speech text representation and a synthesized recognized text, including: The acoustic features of the synthesized speech are input into the speech recognition model to obtain the synthesized speech text representation and the synthesized recognized text output by the speech recognition model.

3. The speech annotation quality evaluation method according to claim 2, characterized in that, The training steps of the speech recognition model include: The acoustic features of the sample speech are input into the initial model of the speech recognition model to obtain the sample speech text representation and the sample recognition text output by the initial model; Based on the differences between the speech text representations of samples with the same labeled text and / or the differences between the speech text representations of samples with different labeled text, as well as the differences between the labeled text and the recognized text, the initial model is iterated to obtain the speech recognition model.

4. The speech annotation quality evaluation method according to claim 3, characterized in that, The sample speech includes the original sample speech and the synthesized sample speech, wherein the synthesized sample speech is obtained by synthesizing the speech from the labeled sample text. The differences between the sample speech text representations corresponding to the different sample labeled texts include the differences between the original speech text representations corresponding to the different sample labeled texts and / or the differences between the synthesized speech text representations corresponding to the different sample labeled texts; The differences between the speech text representations of the same labeled text and the corresponding samples include at least one of the following: the differences between the original speech text representations of the same labeled text and the corresponding synthesized speech text representations of the same labeled text and the corresponding original speech text representations of the same labeled text and the corresponding synthesized speech text representations of the same labeled text.

5. The speech annotation quality evaluation method according to claim 2, characterized in that, The step of inputting the acoustic features of the original speech into a speech recognition model to obtain the original speech text representation and the original recognized text output by the speech recognition model includes: The acoustic features of the original speech are input into the first coding layer of the speech recognition model to obtain the original coding features output by the first coding layer; The original encoded features are input into the attention layer of the speech recognition model to obtain the attention features output by the attention layer; The attention features are input into the second coding layer of the speech recognition model to obtain the original speech-text representation output by the second coding layer; The attention features are input into the decoding layer of the speech recognition model to obtain the original recognized text output by the decoding layer.

6. The speech annotation quality evaluation method according to any one of claims 1 to 5, characterized in that, The steps for determining the text editing distance include: Determine the number of replacement operations, insertion operations, and deletion operations corresponding to the conversion of the original recognized text into the synthesized recognized text; The text editing distance is determined based on the number of replacement operations, the number of insertion operations, and the number of deletion operations.

7. The speech annotation quality evaluation method according to claim 6, characterized in that, Determining the text editing distance based on the number of replacement operations, the number of insertion operations, and the number of deletion operations includes: The first edit distance is determined based on the number of replacement operations; The second edit distance is determined based on the number of insertion operations; The third edit distance is determined based on the number of deletion operations; The average of the first edit distance, the second edit distance, and the third edit distance is taken as the text edit distance.

8. A voice annotation quality evaluation device, characterized in that, include: The text determination unit is used to determine the annotated text of the original speech. A speech synthesis unit is used to perform speech synthesis based on the labeled text to obtain synthesized speech; The first recognition unit is used to perform speech recognition on the original speech to obtain the original speech text representation and the original recognized text. The second recognition unit is used to perform speech recognition on the synthesized speech to obtain a synthesized speech text representation and a synthesized recognized text. The quality evaluation unit is used to determine the annotation quality evaluation result based on the feature similarity between the original speech text representation and the synthesized speech text representation, and / or the text edit distance between the original recognized text and the synthesized recognized text; the speech recognition is performed by a speech recognition model, which is trained based on the difference between the sample annotation text and the sample speech text representation, as well as the difference between the sample recognized text and the sample annotation text, and the sample speech includes the original sample speech and the sample synthesized speech.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the speech annotation quality evaluation method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech annotation quality evaluation method as described in any one of claims 1 to 7.

Images