Methods for generating training corpora and methods for training speech recognition models

Abstract

This invention provides a method for generating training corpora. From the text data in the corpus, a special word (such as a custom noun or a character with a broken pronunciation) is identified. At least one sentence is generated based on the special word, and text data for training purposes is added. At least one audio data corresponding to the at least one sentence is synthesized. The sentence and its corresponding audio data are appropriately processed to form the training corpus.

Description

Technical Field

[0001] The present invention relates to a method for generating training corpus and a method for training a speech recognition model, in particular to a method for expanding training corpus, and training a speech recognition model by using the expanded training corpus. Prior Art

[0002] "TTS (text to speech)" generally refers to the technology of converting text into speech, which is a speech synthesis technology that can convert plain text into natural speech. Conventional TTS still has challenges in processing polyphonic characters. For example, for the Chinese single character "中", the text-to-speech module must clarify whether it should be pronounced as "中 [ㄓㄨㄥ]" in "高中 (high school)" or "中 [ㄓㄨㄥˋ]" in "命中 (hit)". Alternatively, for special proper nouns, for example, a specific brand or trademark is the English word "MOMO", the text-to-speech module must clarify whether it should be pronounced as "MOMO [mɔ mɔ]" in English phonetics or "MOMO [ㄇㄡˇ ㄇㄡˇ]" in a common pronunciation manner. For another example, for the self-created compound noun "MyVideo", the text-to-speech module must clarify whether it should be pronounced as "MyVideo [maɪ vɪdɪo]" in English phonetics or "MyVideo [ㄇㄞˇ vɪdɪo]" in a mixed Chinese-English pronunciation. Most conventional TTS processes by calculating the probability of these two pronunciations, but this processing method still has a considerable probability of leading to pronunciation errors.

[0003] In practice, synthetic speech data generated by TTS can be used to train a speech recognition model. Since TTS converts original text data into speech data, the accuracy of the speech data after TTS conversion will affect the subsequent training results.

[0004] The first figure illustrates a known training process, particularly the process of collecting and processing corpus before entering training.

[0005] First, the first step is to collect corpus, such as, but not limited to, video / audio data without background noise or containing various speaking tones, which usually has audio files and corresponding text files. In addition, specific corpus can be collected for specific purposes. For example, to train a speech recognition model for a specific service, the collected corpus can be derived from materials of the service provider, such as meeting recordings, document contents, etc. For example, for the speech recognition service of the e-commerce platform "MOMO", the original content of the corpus can be derived from the speech data and corresponding text data collected within the "MOMO" company organization.

[0006] Next, the collected data will be preprocessed, for example, but not limited to, removing data from unknown or illegal sources, removing data where the audio and text do not correspond, converting simplified Chinese to traditional Chinese, changing the file name to ensure consistency between audio and text data, and converting video data into audio files.

[0007] Post-processing of the corpus includes, but is not limited to, segmenting the speech data (e.g., into 10-second segments) and similarly segmenting the corresponding text data. These speech and text data are then categorized according to their subsequent purpose. For example, which data will be used as training data and which will be used as testing data.

[0008] Feature training, which can be achieved using known methods such as cepstral mean-variance normalization (CMVN), involves processing the corpus. The purpose of this processing is to identify human speech sounds within the speech data of the corpus, discarding any unnecessary noise.

[0009] For textual data in the corpus, known methods (such as the word segmentation tool Jieba) can be used to segment the text into words, and a dictionary can be built based on this segmentation. Then, these word segmentation symbols are tokenized based on the dictionary. In this way, the collected corpus has been converted into numerical and vector forms that can be understood by computers and fed into neural networks for training.

[0010] As the foregoing technical description shows, corpus processing is crucial for subsequent training, especially since the corpus must include characters with broken pronunciations or certain invented and proper nouns. However, collecting a complete and comprehensive corpus in practice is quite challenging. Therefore, how to effectively expand the quantity of training corpora is a pressing issue that needs to be addressed by those skilled in the art. Summary of the Invention

[0011] The present invention aims to provide a method for generating training corpus. A special word is received as input via an input interface, and the special word has corresponding speech data. A generation module generates at least one sentence containing the special word based on the special word received at the input interface. A speech synthesis module synthesizes at least one speech data corresponding to the at least one sentence, based at least on the special word, the speech data corresponding to the special word, and the at least one sentence containing the special word. A corpus processing module preprocesses the at least one sentence and the at least one speech data corresponding to the at least one sentence to convert them into training corpus that can be understood by a neural network.

[0012] In one specific embodiment, the special word is composed of Chinese, English, or a combination thereof.

[0013] In one specific embodiment, the method further includes receiving input of the special word, wherein the input interface receives the speech data corresponding to the special word.

[0014] In one specific embodiment, the input interface includes a player configured to play the audio data corresponding to the special word. The audio data for the special word can be provided by the user. For example, the user can directly record the pronunciation of the special word using a recording module. The audio data for the special word can also originate from a TTS model or corpus provided after audio-visual processing.

[0015] In one specific embodiment, the at least one sentence is generated based on the response of a large language model or a web crawling method.

[0016] In one specific embodiment, the method further includes, by the corpus processing module, at least based on the special word and the speech data corresponding to the special word, establishing an index value in a dictionary, the index value being associated with the special word and the speech data corresponding to the special word.

[0017] In one specific embodiment, the method further includes, by the corpus processing module, establishing an index value in a dictionary based at least on the special word and the corresponding English pinyin or Zhuyin, the index value being associated with the special word and the corresponding speech data.

[0018] In one specific embodiment, the training corpus is suitable for training a speech recognition model.

[0019] In one specific embodiment, the method further includes expanding, via a copy-and-reproduce module, at least one speech data corresponding to at least one sentence into multiple derived speech data, wherein the derived speech data correspond to the same text data but have different intonation features.

[0020] Another objective of this invention is to provide a method for training a speech recognition model, which trains a speech recognition model through a neural network based on the aforementioned training corpus. Simple Explanation of the Diagram

[0021] The invention can be further understood by referring to the following figures and descriptions. Non-limiting and non-exhaustive examples are described with reference to the following figures. The components in the figures are not necessarily actual dimensions; the focus is on illustrating the structure and principle.

[0022] The first illustration shows the training process of a known speech recognition model.

[0023] The second figure shows that the method for generating training corpora according to the present invention includes multiple steps.

[0024] The third figure illustrates an input interface proposed in this invention. Implementation

[0025] The invention will now be described in more detail with reference to the accompanying drawings, and specific exemplary embodiments will be shown by way of illustration. However, the subject matter of this invention can be embodied in many different forms, and therefore the construction of the subject matter covered or claimed is not limited to any of the exemplary embodiments disclosed in this specification; the exemplary embodiments are merely illustrative. Similarly, the invention aims to provide a reasonably broad scope for the claimed or covered subject matter. Furthermore, the drawings and illustrations in this invention are generally not drawn to scale and are not intended to correspond to actual relative dimensions.

[0026] For the purposes of consistency and ease of understanding, the same features are indicated by reference numerals in the exemplary drawings (although this is not the case in some examples). However, features in different embodiments may differ in other respects, and therefore should not be narrowly limited to the features shown in the drawings. The terms "first" and "second," etc., in the specification of this invention and the above-described drawings are used to distinguish different objects and not to describe a particular order.

[0027] In addition to the learned method demonstrated in the first figure, this invention proposes a method to supplement the deficiencies of existing training corpora. By establishing a more complete dictionary index, the ability to distinguish between Chinese characters with broken pronunciations or English words with multiple pronunciations is improved.

[0028] The second figure shows that the method for generating training corpora according to the present invention includes multiple steps.

[0029] First, in step S201, original speech data and its corresponding text data are collected as elements for establishing a training corpus. This step is not substantially different from the corpus collection described with reference to the first figure. A person skilled in the art can collect relevant videos and audio files through various possible means, and distinguish speech data and text data from each video and audio file. The speech data can be played alone via a player to generate an audio signal, particularly human speech. The speech data can be processed by known means to make the human voice part clearer while other sounds are suppressed. The text data can be displayed by a reader or an editor. The text data may comprise a plurality of time markers, which indicate the display time of the corresponding text in a video or audio file, such as subtitles and lyrics.

[0030] In step S204, when some special words appear in the collected text data, they can be screened out and processed by the method of the present invention. Said special words include, but are not limited to, custom nouns and heterophonic characters. Custom words include, for example, the trademark noun "MOMO" or the service name "MyVideo". A heterophonic character generally refers to a character that has different pronunciations. For example, the character "中" can be pronounced as "ㄓㄨㄥ" or "ㄓㄨㄥˋ".

[0031] The third figure illustrates an input interface proposed by the present invention, which is provided for a person skilled in the art to input a screened special word. The input interface can be generated by an application program executed by a terminal device (such as a desktop computer, a laptop computer, a tablet computer) or a server, and presented on a display screen. A user can input text into the input interface through the terminal device. The input interface comprises a text interface 30A and a player 30B.

[0032] The text interface 30A comprises a "dictionary word" field, a "dictionary word TTS pronunciation" field, and a "dictionary word description" field. The "dictionary word" field is configured to receive the aforementioned screened special word, such as the custom noun "MyVideo" exemplified in the figure. The player 30B comprises a "generate dictionary word pronunciation" button, which, in response to a click, converts the word input in the "dictionary word" field into corresponding speech data through a TTS module, and can be played for audition by the player 30B, but the present invention is not limited thereto to generate said speech data. In an original corpus, special words originally have corresponding speech data, which can be loaded into the player 30B by known means. The "dictionary word TTS pronunciation" field is configured to display the pronunciation basis of the TTS module. As shown in the figure, although "MyVideo" is input in the "dictionary word" field, for the TTS module, the pronunciation generated is based on the text data "買 Video". Therefore, it is necessary to expand relevant corpora for such special words like "MyVideo".

[0033] In step S206, at least one sentence is generated based on the special word received at the input interface, serving as part of the text data in the training corpus. The player 30B also includes a "Generate Dictionary Word Description" button, which, in response to the special word entered in the "Dictionary Word" field, calls a generation module based on a large language model. This generation module generates at least one sentence based on the special word, and the at least one sentence contains the special word. As shown in the figure, the generated sentence displayed in the "Dictionary Word Description" field is "MyVideo provides OTT services for on-demand streaming videos. MyVideo provides you with a variety of the latest and most exciting blockbuster movies…". In one embodiment, the generation module generates the sentence based on a large language model or a web crawling method. Accordingly, the user can obtain text data expanded and generated based on the special word, and this text data may be text data not collected in step S201 (due to limitations in collection methods and time costs).

[0034] In another embodiment, the generation module can generate more sentences in the "Dictionary Word Description" field based on a specific word already entered in the "Dictionary Word" field and a sentence already entered in the "Dictionary Word Description" field. The entered sentence can be a sentence already generated and displayed in the "Dictionary Word Description" field, which can be entered or modified by the user. Accordingly, the generation module can generate other sentences that meet the desired description based on fine-tuning information.

[0035] In step S208, each sentence generated by the above generation model or in the "dictionary entry description" field is processed by a phoneme conversion model (such as the known g2pW model), and each character or word in the sentence is converted into phoneme data. The phoneme conversion model is configured to be able to understand the pronunciation of a word or a character according to the context of the word or the character in the sentence. For example, the word "MyVideo" is converted into phoneme data described as "[maɪvɪdɪo] ", the word "MoMo" is converted into phoneme data described as "[mɔ mɔ] ", and the character "中" is converted into phoneme data described as "[ㄓㄨㄥ] " or "[ㄓㄨㄥˋ] ". Accordingly, each sentence is converted into a combination of multiple pieces of phoneme data. For example, ['銀行', '行動'] will be converted into correct polyphonic character phonetic notation: [['ㄧㄣ2', 'ㄏㄤ2'], ['ㄒㄧㄥ2', 'ㄉㄨㄥ4']]. For example, ['上校請技術人員校正FN儀器'] will be converted into correct polyphonic character phonetic notation: [['ㄕㄤ4', 'ㄒㄧㄠ4', 'ㄑㄧㄥ3', 'ㄐㄧ4', 'ㄕㄨ4', 'ㄖㄣ2', 'ㄩㄢ2', 'ㄐㄧㄠ4', 'ㄓㄥ4', None, None, 'ㄧ2', 'ㄑㄧ4']].

[0036] In step S210, through a speech synthesis module, speech data corresponding to each sentence is synthesized based on the plurality of phoneme data constituting each sentence. In one embodiment, 600MB of data per batch can be stored in a network attached storage (NAS).

[0037] In step S212, the at least one sentence generated above (i.e., text data) and the synthesized speech data above (i.e., speech data) are processed by a corpus processing module to obtain a data format understandable by a neural training network, and a training corpus is completed. For example, the text data can be subjected to chunking and tokenization processing to obtain matrix vector data. The speech data can also be processed according to the same logic. Said chunking and tokenization processing includes establishing index values in a dictionary. For example, the index value of the term "MyVideo[maɪvɪdɪo]" in the dictionary is the numerical value "1", the index value of the character "中[ㄓㄨㄥ]" in the dictionary is the numerical value "2", and the index value of the character "中[ㄓㄨㄥˋ]" in the dictionary is the numerical value "3". In this way, the words in the sentence can be chunked and given index values, forming a data format like a matrix or a vector. The specific processing details are well known to those skilled in the art, and will not be repeated here. The training corpus can be combined with the corpus collected and processed from step S201, and used for a specific neural training network to train an ASR model.

[0038] In one embodiment, the above steps may further include a cloning method. For example, a regeneration and replication module can use the phoneme combination obtained in step S208 or the speech data corresponding to each sentence synthesized in step S210 to reproduce and generate multiple speech data by replicating and regenerating the speech data with different character intonations (such as male, female, child, adult, etc.). In this way, the same sentence can correspond to multiple speech data, thereby further expanding the amount of training data. In one embodiment, replication and reproduction can be performed in batches of 120GB and then transmitted to the user.

[0039] Since the expanded training corpus of this invention covers more textual and audio data about special words and broken sounds, when combined with the training corpus obtained in known ways, it can deepen the understanding of special words and broken sounds of the trained ASR model and improve the accuracy of speech recognition.

[0040] However, it should be understood that the various specific embodiments of the present invention are merely illustrative, and various changes can be made without departing from the scope and spirit of the present invention, and all such changes should be included within the scope of the present invention. Therefore, the specific embodiments described in this specification are not intended to limit the present invention, and the true scope and spirit of the present invention are disclosed in the following claims.

[0041] S201: Steps S204: Steps S206: Steps S208: Steps S210: Steps S212: Steps 30A: Text Interface 30B: Playback Interface

Claims

1. A method for generating training corpus, comprising: The input of a special word is received through an input interface, and the special word has a corresponding voice data. A generation module enables a large language model to generate at least one sentence containing the special word received at the input interface; a speech synthesis module synthesizes at least one piece of speech data corresponding to the at least one sentence based on the special word, the speech data corresponding to the special word, and the at least one sentence containing the special word; and a corpus processing module converts the at least one sentence and the at least one piece of speech data corresponding to the at least one sentence into a training corpus that can be understood by a neural network; wherein the corpus processing module further establishes an index value in a dictionary based on the special word and the speech data corresponding to the special word, the index value being associated with the special word and its corresponding speech data.

2. The method for generating training corpus as described in claim 1, wherein, This special word is composed of Chinese, English, or a combination thereof.

3. The method for generating training corpus as described in claim 1 further includes: receiving input of the special word, wherein the input interface receives the speech data corresponding to the special word.

4. The method for generating training corpus as described in claim 3, wherein, The input interface includes a player configured to play the audio data corresponding to the special word.

5. The method for generating training corpus as described in claim 1, wherein, At least one sentence is generated based on the response of a large language model or a web crawling technique.

6. The method for generating training corpus as described in claim 1 further comprises: establishing an index value in a dictionary by the corpus processing module, at least based on the special word and an English pinyin or a phonetic transcription corresponding to the special word, the index value being associated with the special word and the corresponding speech data.

7. The method for generating training corpus as described in claim 1, wherein, This training corpus is suitable for training a speech recognition model.

8. The method for generating training corpus as described in claim 1 further comprises: expanding, via a copy-and-reproduce module, at least one speech data corresponding to at least one sentence into multiple derived speech data, wherein the derived speech data correspond to the same text data but have different intonation features.

9. A method for training a speech recognition model, comprising: training a speech recognition model through a neural network based on the training corpus as described in any one of claims 1 to 8.

Images