Method and apparatus for generating training data to be applied to a speech recognition model

Abstract

A method and device for generating training data to be applied to a speech recognition model, the method for generating training data to be applied to a speech recognition model includes the steps of acquiring speech data, recognizing the speech data using a plurality of speech recognition models, respectively, and outputting a plurality of text data, acquiring target text data based on the plurality of text data, acquiring an evaluation result of the speech evaluation model (13) for the target text data and the speech data, and judging the evaluation result and, if the evaluation result is as expected, combining the target text and the speech data as training data.

Description

[Technical Field]

[0001] The present invention relates to the field of speech analysis and synthesis, and in particular to a method and apparatus for generating training data to be applied to a speech recognition model. [Background technology]

[0002] Speech recognition technology has evolved from the previous GMM-HMM to a hybrid model using HMM topology and neural networks, and has now reached the current CTC / RNNT / LAS end-to-end modeling method using transformer / conformer. Modeling capabilities are becoming increasingly stronger, but the demand for training data is also growing exponentially. The demand for end-to-end data has changed from data sparse to date hungry. Currently, the amount of data marked by speech recognition manufacturers has reached the level of nearly 100,000 hours. Here, data is<audio、transcription> , i.e., refers to audio and text labels, so both the human and economic costs for data marking are high. Summary of the Invention [Problem to be solved by the invention]

[0003] Currently, the general-purpose speech recognition performance of some speech recognition manufacturers has reached an excellent level, with a word error rate (WER) below 3%, surpassing the level of manual transcription and reaching commercial standards. However, for various reasons, many companies do not want to directly access the interfaces of speech recognition service providers and instead prefer to create their own speech recognition models. Possible considerations include data security, cost, and non-ideal effects during actual service. In this case, companies typically build commercial speech recognition systems by collecting speech samples, marking the data, and then training a model, but this processing method has very low cost-performance. [Means for solving the problem]

[0004] In view of this, the present application provides a method for generating training data to be applied to a speech recognition model, acquiring audio data; recognizing the speech data using a plurality of speech recognition models, respectively, and outputting a plurality of text data; acquiring target text data based on the plurality of text data; obtaining an evaluation result of a speech evaluation model for the target text data and the speech data; determining the evaluation result, and if the evaluation result is as expected, combining the target text data and the speech data as training data.

[0005] Optionally, the step of obtaining target text data based on the plurality of text data comprises: obtaining text graph data based on the plurality of text data, and arranging similar parts of each of the text data in a juxtaposed relationship with each other; and evaluating the graph data and the voice data of the text using a voice evaluation model, and obtaining a target text based on a path in the graph data that corresponds to an optimal evaluation result.

[0006] Optionally, when obtaining text graph data based on the plurality of text data, determining whether the pronunciation of the parallel portions is similar; If the pronunciation is similar, the method further includes filtering the aligned portions based on a predetermined lexicon to eliminate vocabulary irrelevant to the application scenario.

[0007] The present invention provides another method for generating training data to be applied to a speech recognition model, comprising: obtaining original audio data; converting the frequency of the original audio data to obtain at least one converted audio data; recognizing the original voice data using a plurality of voice recognition models, respectively, to output a plurality of original text data; obtaining first target text data based on the plurality of original text data, and obtaining a first evaluation result of a voice evaluation model for the first target text data and the original voice data; recognizing the converted voice data using a plurality of the voice recognition models, respectively, to output a plurality of converted text data; obtaining second target text data based on the plurality of converted text data, and obtaining a second evaluation result of a voice evaluation model for the second target text data and the converted voice data; comparing the first evaluation result with the second evaluation result, and if the second evaluation result is superior to the first evaluation result, combining the original voice data and the second target text data as training data.

[0008] Optionally, the step of obtaining first target text data based on the original text data specifically includes: obtaining original text graph data based on the plurality of original text data, and arranging the original text data so that similar parts are kept and different parts are in parallel relationship; and evaluating the graph data of the original text and the original voice data using a voice evaluation model, and obtaining a first target text based on a path corresponding to the best evaluation result in the graph data of the original text.

[0009] Optionally, the step of obtaining second target text data based on the plurality of converted text data specifically includes: obtaining graph data of the converted text based on the plurality of converted text data, and arranging the similar parts of each converted text data so that the different parts are in a parallel relationship; and evaluating the graph data of the converted text and the converted voice data using a voice evaluation model, and obtaining a second target text based on the path having the best evaluation result in the graph data of the converted text.

[0010] Optionally, the converted voice data is multiple, and the second target text data and the second evaluation results correspond to each other, and in the step of comparing the first evaluation result with the second evaluation result, an optimal one of the multiple second evaluation results is selected and compared with the first evaluation result.

[0011] Optionally, converting the frequency of said original audio data specifically comprises increasing and / or decreasing the fundamental frequency of the audio signal.

[0012] Optionally, said converted audio data is reduced by at least one semitone from said original audio data.

[0013] Accordingly, the present invention provides an apparatus for generating training data to be applied to a speech recognition model, comprising at least one processor and a memory connected to communicate with the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the above-mentioned training data generation method. [Effects of the Invention]

[0014] According to the training data generation method and device provided by the present invention, a speech recognition model provided by another service provider is used to recognize unmarked speech, an initial recognized text is obtained, and then a speech evaluation model is used to evaluate the text and speech to obtain the matching degree between the pronunciation and the text. Finally, the sample quality is determined based on the evaluation result, and high-quality speech samples and their text labels are used as training data, thereby realizing automatic marking of speech and obtaining high-quality training data, improving the model training efficiency and achieving good training results.

[0015] According to the training data generation method and device provided by the present invention, voice data is processed to obtain frequency-converted voice data, and then the original voice and the converted voice are respectively recognized using a voice recognition model provided by another service provider to obtain the corresponding recognized text, and then a voice evaluation model is used to evaluate the text and voice to obtain the matching degree between the original pronunciation and the corresponding text, and between the converted voice and the corresponding text, and finally, the two are compared, and if the evaluation result of the converted voice is better, the frequency-converted voice is matched with the corresponding text, and the text corresponding to the original voice and the converted voice are used as training data, and automatic marking of the voice is realized to obtain high-quality training data, improve the model training efficiency, and achieve good training results. [Brief explanation of the drawings]

[0016] In order to more clearly explain the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings necessary for describing the specific embodiments or the prior art. Obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings on the premise that they do not make efforts that amount to inventive step. [Figure 1] FIG. 2 is a schematic diagram of a method for generating training data in one embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram of textual graph data according to an embodiment of the present invention; [Figure 3] FIG. 2 is a schematic diagram of the data flow of one training data generation method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] The following clearly and completely describes the technical solution of the present invention in conjunction with the drawings, and the described embodiments are not all embodiments but only some of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without any inventive effort are also within the scope of protection of the present invention.

[0018] Furthermore, the technical features of the different embodiments of the present invention described below may be combined with each other if they do not conflict with each other.

[0019] In the first application scenario of the present invention, speech samples from a specific field are obtained as training data, and the specific field in this application refers to highly specialized fields such as chemical engineering and medicine, while the general field refers to general daily life fields. Due to the large difference between the speech content related to the specific field and the content of general-purpose speech data, a speech recognition model trained with general-purpose speech data is difficult to apply to speech recognition tasks in the specific field, and there are few speech samples in the specific field.

[0020] For such application scenarios, in order to obtain usable training data, this embodiment provides a method for generating training data to be applied to a speech recognition model, which is executed by an electronic device such as a computer or a server, and includes the following operations:

[0021] S1A: Acquire audio data. The audio data in this embodiment can be any specific or general audio, such as a single word or a single phrase, with unknown text. This solution does not limit the language, and can be any language, such as Chinese or English.

[0022] S2A: Recognizes the voice data using multiple speech recognition models, respectively, and outputs multiple text data. The speech recognition models are trained neural network models using deep learning algorithms, such as models provided by other service providers, and although the models have certain text recognition capabilities, they may not be suitable for specific fields.

[0023] As shown in Figure 1, taking two speech recognition models as an example, a first speech recognition model 11 and a second speech recognition model 12 each perform speech-to-text recognition on the speech data audio, and the recognition result output from the first speech recognition model 11 is transcription-1, and the recognition result output from the second speech recognition model 12 is transcription-2.

[0024] In this step, a speech recognition model with a hotword recognition function is preferably used, which acquires a number of vocabulary words provided by the user in advance and uses these vocabulary words as reference when recognizing speech data, thereby making the recognition result more accurate. For example, if the actual text contains the vocabulary "amino acid", the general model JPEG0007719891000001.jpg4170 is likely to be pronounced similarly, and if the model has acquired the vocabulary in advance, it will output accurate recognition results.

[0025] S3A: Obtain target text data based on multiple text data. Because various factors such as the performance of each speech recognition model, the training algorithm used, and the training samples may differ, these recognition results may have many inconsistencies, but they may also be completely consistent. Therefore, there are various situations in this step. For example, if multiple recognition results (text data) are completely consistent, the recognition result is the target text data.

[0026] For situations where there is a mismatch in the multiple recognition results, there are various preferred embodiments for obtaining target text data. JPEG0007719891000002.jpg32170

[0027] When faced with such a situation, a preferred processing method is to pre-establish a glossary based on an application scenario, in which all vocabulary that can appear in the application scenario is stored; if the output results of multiple models are inconsistent, the glossary is searched for these inconsistent words, and words that exist in the glossary or that appear frequently are retained.

[0028] JPEG0007719891000003.jpg21170

[0029] Another suitable processing method is to introduce a speech evaluation model that searches for the text that best matches the speech data from multiple recognition results. Specifically, text graph data (Graph) is obtained based on multiple text data, and similar parts in each text data are retained and different parts are arranged in parallel. JPEG0007719891000004.jpg21170 indicates a relationship between the two nodes, and in the graph these are called parallel nodes or branching nodes. Therefore, the graph data in FIG. 2 has four paths, and each path is made up of a vocabulary sequence.

[0030] The graph data is used as the recognition object of the speech evaluation model. In this embodiment, a model is used that can analyze the matching degree between the pronunciation situation of the speech and the text, and the model outputs a score out of 100 points. The higher the score, the more the pronunciation situation matches the text or the better the pronunciation quality. The speech evaluation model can be realized in various ways, for example, see the speech evaluation method and related device disclosed in Chinese Patent Document CN110797049A. However, this solution is not limited to this evaluation solution, and other evaluation models can be used to output other types of evaluation results. Any algorithm that can evaluate graph data and corresponding speech data is possible. The form of the evaluation result can be, for example, a classification result, which can classify the matching condition between the pronunciation and the path into "excellent," "average," "poor," etc. that can be used for comparison.

[0031] The speech evaluation model of this embodiment searches the graph data, and the model matches all paths in the graph data with the speech data, that is, evaluates the matching degree or quality of each path of the speech data, and determines the optimal path as the target text.

[0032] The above two methods may also be combined. For example, when using more speech recognition models to obtain more recognition results, directly using the third method to search for the optimal path would result in the following problems: (1) excessively large numbers of paths in the graph data would increase the amount of calculation required for the model, reducing search efficiency; and (2) the speech evaluation model may not have the ability to perform semantic analysis, so that the evaluation results of the paths located there may be identical for words that are identical in pronunciation or words that differ only in intonation. In a preferred embodiment, the first method is used to first perform initial selection of branch nodes in the graph data (Graph) to eliminate vocabulary that is irrelevant to the application scenario, thereby reducing the number of branch nodes and further reducing the number of paths, thereby avoiding the above problems.

[0033] S4A: Obtain the evaluation result of the speech evaluation model for the target text data and speech data. Referring back to Figure 1, the speech evaluation model 13 evaluates the target text data (transcription and audio) to obtain the evaluation result (score). If the speech evaluation model is used to search for the optimal path in step S3A, the speech evaluation model calculates a score or other form of evaluation result using the text of the optimal path as a reference in this step. Of course, if the target text is not obtained using the speech evaluation model in step S3A, but is generated in another manner as described above, the speech evaluation model is used for evaluation in this step for the first time.

[0034] S5A: Evaluate the evaluation result, and if the evaluation result is as expected, combine the target text and the speech data as training data. For example, a score threshold is set in advance. If the evaluation result for the target text data and the speech data is higher than the threshold, the matching degree between the speech and the target text is high enough, and the data pair is suitable for training a speech recognition model and is a high-quality sample. Conversely, the matching degree is poor, and the data pair is not suitable for training a model.

[0035] According to the training data generation method provided by an embodiment of the present invention, a speech recognition model provided by another service provider is used to recognize unmarked speech, an initial recognized text is obtained, and then a speech evaluation model is used to evaluate the text and speech to obtain the matching degree between the pronunciation and the text. Finally, the sample quality is determined based on the evaluation result, and high-quality speech samples and their text labels are used as training data, thereby realizing automatic marking of speech, obtaining high-quality training data, improving the model training efficiency, and achieving good training results.

[0036] In the second application scenario, the speech of a specific group is obtained as training data. Take the service scenario for young children as an example. Children's vocal organs are significantly different from those of adults, so children's speech sounds are significantly different from adult speech sounds. In reality, there are fewer children's speech samples than ordinary adult speech samples. If a speech recognition model is directly trained using adult speech and then used to recognize children's speech, the recognition results obtained will be poor. Therefore, children's speech recognition has always been a difficult problem in speech recognition.

[0037] Currently, the conventional method is to enhance the data, for example, by using digital signals to achieve an overall upward shift in the spectrum of adult speech, making the sound sound sharper and more similar to a child's speech. Although this method brings certain benefits to speech recognition training tools, its efficiency is very low and the final result is not ideal enough. The fundamental reasons are that the method itself is not targeted and the acquired data is not realistic enough.

[0038] For such an application scenario, another embodiment of the present application provides a method for generating training data to be applied to a speech recognition model, the method being executed by an electronic device such as a computer or a server, the method including the following operations: S1B: Acquire original voice data. For example, the original voice is a child's voice.

[0039] S2B: Convert the frequency of the original voice to obtain at least one converted voice data. Specific conversion methods include, but are not limited to, changing the frequency of the voice signal. For children's voices, this embodiment adopts a method of reducing the fundamental frequency of the voice signal, so that the converted voice data is reduced by at least one semitone from the sound of the original voice data.

[0040] In other similar application scenarios, for example, the original data is a male voice with a very deep throat, or a female voice with a very sharp throat, then the corresponding method is used to process or convert the frequency.

[0041] 3, the original audio data is denoted as audio-0. In order to obtain better training data, in this step, the original audio data is subjected to multiple conversion processes, each time reducing the audio quality by one semitone, to obtain more converted audio data. For example, the converted audio data reduced by one semitone is denoted as audio-1, the converted audio data reduced by two semitones is denoted as audio-2, and so on, and the converted audio data reduced by n semitones is denoted as audio-n.

[0042] As can be seen from the verification of the actual embodiment, in this step, if the converted speech data is obtained that is reduced by 3 to 5 semitones in order, good recognition results can be obtained and high calculation efficiency can be achieved.

[0043] S3B: Recognize the original voice data using multiple voice recognition models, respectively, and output multiple original text data. Specifically, see S2A in the above embodiment, this step recognizes audio-0 and obtains multiple original text data.

[0044] S4B: Obtain first target text data based on the plurality of original text data, and obtain a first evaluation result of the audio evaluation model for the first target text data and the original audio data. Specifically, refer to step S3A in the above embodiment. For ease of description, the first target text data will be referred to as "transcription-0." In this embodiment, the third embodiment provided in the above embodiment is preferably adopted, that is, the audio evaluation model searches for "transcription0" based on the graph data of audio-0 and the original text data.

[0045] Furthermore, the audio evaluation model evaluates audio-0 and transcription-0 to obtain the first evaluation result score-0.

[0046] S5B: Use multiple speech recognition models to recognize the converted speech data, respectively, and output multiple converted text data. See S2A in the above embodiment for similarity. Given multiple converted speech data, such as audio-1...audio-n, multiple speech recognition models recognize audio-1 to obtain the corresponding multiple converted text data, multiple speech recognition models recognize audio-2 to obtain the corresponding multiple converted text data, and multiple speech recognition models recognize audio-n to obtain the corresponding multiple converted text data.

[0047] S6B: Obtain second target text data based on the plurality of converted text data, and obtain a second evaluation result of the speech evaluation model for the second target text data and the converted speech data. Similarly, see S3A in the above embodiment.

[0048] Here, if there are multiple converted audio data and multiple corresponding converted text data, the second target text data transcription-1 corresponding to audio-1, the second target text data transcription-2 corresponding to audio-2, ..., the second target text data transcription-n corresponding to audio-n are obtained.

[0049] Furthermore, the audio evaluation model evaluates audio-1 and transcription-1 to obtain a second evaluation result score-1, the audio evaluation model evaluates audio-2 and transcription-2 to obtain a second evaluation result score-2, and so on, and the audio evaluation model evaluates audio-n and transcription-n to obtain a second evaluation result score-n.

[0050] Furthermore, the above steps S3B and S4B and steps S5B and S6B may be executed in parallel, and the present solution does not limit the order in which the steps are executed.

[0051] S7B: Compare the first evaluation result with the second evaluation result. If the second evaluation result is superior to the first evaluation result, combine the original audio data and the second target text as training data. In step S2B, if multiple converted audio data are obtained, first determine the most suitable one from the multiple second evaluation results. For example, if the highest score among score-0...score-n in the above example is score-x, compare score-x with score-0 in this step. If score-x is superior to score-0, combine the original audio audio-0 and the target text transcription-x corresponding to score-x as training data. The matching degree between the original audio and the target text is sufficiently high, making the data set suitable for training a speech recognition model and a high-quality sample. Conversely, the matching degree is poor and unsuitable for model training.

[0052] In a preferred embodiment, the comparison operation in step S7B is configured as follows: if the second evaluation result is superior to the first evaluation result, further determine whether the superiority of the second evaluation result over the first evaluation result is as expected. Only if the superiority is sufficiently large (as expected), combine the original audio data and the second target text as training data. Taking the above score-type evaluation result as an example, if score-x is greater than score-0, further determine whether the excess value is greater than a predetermined threshold. Only if it is greater than the predetermined threshold, combine audio-0 and transcription-x as training data. Similar determinations can be made for other types of evaluation results, such as classification results. In this way, the acquired training data is more targeted, and the effect of training the model is more obvious.

[0053] According to the training data generation method provided by an embodiment of the present invention, speech data is processed to obtain frequency-converted speech data, and then the original speech and the converted speech are respectively recognized using a speech recognition model provided by another service provider to obtain the corresponding recognized text. Then, the text and speech are evaluated using a speech evaluation model to obtain the matching degree between the original pronunciation and the text, and the matching degree between the converted speech and the text. Finally, the two are compared, and if the evaluation result of the converted speech is better, the original speech is more closely matched to the text corresponding to the converted speech. Therefore, the text corresponding to the original speech and the converted speech are used as training data, thereby realizing automatic marking of speech and obtaining high-quality training data, improving the training efficiency of the model and achieving better training results.

[0054] As will be appreciated by those skilled in the art, embodiments of the present invention may be provided as a method, a system, or a computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied in one or more computer-usable storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0055] The present invention will be described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. Each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, may be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, where the instructions, executed by the processor of the computer or other programmable data processing device, generate an apparatus that implements the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0056] These computer program instructions are further stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture that includes a command device, which implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0057] These computer program instructions, when loaded into a computer or other programmable data processing device, cause the computer or other programmable device to perform a series of operational steps to generate a computer-implemented process, and the instructions executed on the computer or other programmable device provide steps to implement the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0058] Obviously, the above examples do not limit the embodiments, but are merely for the purpose of clearly illustrating the examples. Those skilled in the art may make other different modifications or improvements based on the above description. It is not possible and not necessary to cover all embodiments here. Any obvious modifications or improvements obtained therefrom still fall within the scope of protection of the invention.

Claims

1. 1. A method for generating training data to be applied to a speech recognition model, comprising: acquiring audio data; recognizing the speech data using a plurality of speech recognition models, respectively, and outputting a plurality of text data; acquiring target text data based on the plurality of text data; obtaining an evaluation result of a speech evaluation model for the target text data and the speech data; determining the evaluation result, and if the evaluation result is higher than a preset threshold, combining the target text data and the speech data as training data; 1. A method comprising: The speech evaluation model is a speech evaluation model capable of analyzing the matching degree between the pronunciation situation of the speech and the text, The step of acquiring target text data based on the plurality of text data includes: obtaining text graph data based on the plurality of text data, and arranging similar parts of each of the text data in a juxtaposed relationship with each other; evaluating the graph data of the text and the voice data using a voice evaluation model, and obtaining a target text based on a path in the graph data corresponding to an optimal evaluation result.

2. When graph data of text is obtained based on the plurality of text data, determining whether the pronunciation of the parallel portions is similar; 2. The method of claim 1, further comprising: if the pronunciation is similar, filtering the aligned portions based on a predetermined lexicon to eliminate vocabulary irrelevant to the application scenario.

3. 1. A method for generating training data to be applied to a speech recognition model, comprising: obtaining original audio data; converting the frequency of the original audio data to obtain at least one converted audio data; recognizing the original voice data using a plurality of voice recognition models, respectively, to output a plurality of original text data; obtaining first target text data based on the plurality of original text data, and obtaining a first evaluation result of a voice evaluation model for the first target text data and the original voice data; recognizing the converted voice data using a plurality of the voice recognition models, respectively, to output a plurality of converted text data; obtaining second target text data based on the plurality of converted text data, and obtaining a second evaluation result of a voice evaluation model for the second target text data and the converted voice data; comparing the first evaluation result with the second evaluation result, and if the second evaluation result is superior to the first evaluation result, combining the original speech data and the second target text data as training data, The speech evaluation model is a speech evaluation model capable of analyzing the matching degree between the pronunciation situation of the speech and the text, The step of obtaining first target text data based on the original text data includes: obtaining original text graph data based on the plurality of original text data, and arranging the original text data so that similar parts are kept and different parts are in parallel relationship; and evaluating the graph data of the original text and the original voice data using a voice evaluation model, and obtaining a first target text based on a path corresponding to an optimal evaluation result in the graph data of the original text; The step of obtaining second target text data based on the plurality of converted text data includes: obtaining graph data of the converted text based on the plurality of converted text data, and arranging the similar parts of each converted text data so that the different parts are in a parallel relationship; evaluating the graph data of the converted text and the converted speech data using a speech evaluation model, and obtaining a second target text based on a path having an optimal evaluation result in the graph data of the converted text.

4. The method of claim 3, wherein the converted voice data is a plurality of data, the second target text data and the second evaluation result correspond to each other, and in the step of comparing the first evaluation result with the second evaluation result, an optimal one of the plurality of second evaluation results is selected and compared with the first evaluation result.

5. 4. The method of claim 3, wherein the step of converting the frequency of the original audio data specifically includes the steps of increasing and / or decreasing the fundamental frequency of the audio signal.

6. 6. The method of claim 5, wherein the converted audio data is reduced by at least one semitone from the original audio data.

7. 10. An apparatus for generating training data to be applied to a speech recognition model, comprising: at least one processor; and a memory connected to communicate with said at least one processor, said memory storing instructions executable by said one processor, said instructions being executed by said at least one processor to cause said at least one processor to perform the method for generating training data according to any one of claims 1 to 6.

Images