Construction method and device of multilingual phoneme representation model, equipment and medium

Abstract

The invention discloses a multilingual phoneme representation model construction method and device, equipment and a medium, and the method comprises the steps: obtaining text data of multiple languages, and carrying out the preprocessing of the text data; randomly dividing the preprocessed text data into corresponding parts according to the language environment type, and converting each part of text data into phonemes of the corresponding language environment; the phonemes are segmented to form a phoneme sequence separated by spaces, word boundaries in the text data are marked, a multilingual phoneme representation model is generated, the multilingual phoneme representation model can be directly used as an input phoneme encoder of a speech synthesis model, and multilingual text-to-speech phoneme representation is realized. Therefore, the performance of the high-strength neural TTS model in the aspects of naturalness and rhythm is also beneficial for generating voice with quite high quality under limited training data.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for constructing a multilingual phoneme representation model. Background Art

[0002] Large-scale pre-trained language models, such as BERT (Bidirectional Encoder Representations from Transformers), RoBERTa (Robustly Optimized BERT Pretraining Approach), and ALBERT (ALite BER, ALBERT and ALBERT are both pre-trained language models based on the BERT architecture), can consider the contextual information before and after, improving the performance of natural language processing research and application tasks.

[0003] However, the success of these pre-trained language models is limited to English, and it is worth exploring pre-trained models of phonemic representations beyond English from social, linguistic, cultural, machine learning, and cognitive perspectives. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a method, apparatus, device and medium for constructing a multilingual phoneme representation model that can be applied to financial technology or other related fields, aiming to address the problem that existing speech models are limited to English.

[0005] The technical solution of the present invention is as follows:

[0006] The present invention provides a method for constructing a multilingual phoneme representation model, which comprises:

[0007] Acquiring text data in multiple languages ​​and preprocessing the text data;

[0008] The preprocessed text data is randomly divided into corresponding numbers according to the language environment type, and each piece of text data is converted into the phoneme of the corresponding language environment;

[0009] The phonemes are segmented to form phoneme sequences separated by spaces, and word boundaries in the text data are marked to generate a multilingual phoneme representation model.

[0010] In one embodiment, the step of acquiring text data in multiple languages ​​and preprocessing the text data includes:

[0011] Get text data in multiple languages ​​from pre-training corpora;

[0012] Convert all words and sentences of text data into plain text, and filter out single words and repeated sentences to form text to be processed;

[0013] The text to be processed is segmented into words and sentences.

[0014] In one embodiment, acquiring text data in multiple languages ​​from a pre-training corpus includes:

[0015] Download the wiki40b dataset from the Hugging Face dataset repository and obtain text documents belonging to the Wikipedia language and region;

[0016] Use the Wikipedia dataset to supplement text in other languages.

[0017] In one embodiment, after the text to be processed is segmented into words and sentences, the method further includes:

[0018] Obtain the text to be processed that needs to be normalized;

[0019] The text to be processed is converted from written form to spoken form using a text-to-speech tool.

[0020] In one embodiment, the step of randomly dividing the preprocessed text data into corresponding numbers of copies according to the language environment type and converting each copy of the text data into phonemes of the corresponding language environment includes:

[0021] The preprocessed text data is randomly divided into corresponding numbers according to the language environment type, and a portion of the text data to be converted is allocated to each language;

[0022] Converting the text data to be converted into phonemes of a corresponding language environment through a multilingual conversion model;

[0023] identifying whether the phoneme is in a pronunciation dictionary of a multilingual conversion model;

[0024] When the recognized phoneme is in the pronunciation dictionary of the multilingual conversion model, the phoneme description of the text data is generated using the pronunciation dictionary; when the recognized phoneme is not in the pronunciation dictionary of the multilingual conversion model, the phoneme description of the text data is generated using the multilingual conversion model.

[0025] In one embodiment, the step of segmenting the phonemes to form phoneme sequences separated by spaces, marking word boundaries in the text data, and generating a multilingual phoneme representation model includes:

[0026] Segment the continuous phoneme sequence output by the multilingual conversion model;

[0027] Inserting spaces as phoneme boundary indicators in the segmented phoneme sequence;

[0028] Use metasymbols to mark word boundaries to distinguish the phonemes of different words.

[0029] In one embodiment, after segmenting the phonemes to form phoneme sequences separated by spaces and marking word boundaries in the text data to generate a multilingual phoneme representation model, the method further includes:

[0030] Set the parameters of the multilingual phoneme representation model, including the number of Transformer blocks, hidden size, and number of self-attention heads;

[0031] The multilingual phoneme representation model is pre-trained using a dynamic masking strategy of the RoBERTa pre-training method to optimize the multilingual phoneme representation model.

[0032] A device for constructing a multilingual phoneme representation model, comprising:

[0033] A preprocessing module, used to obtain text data in multiple languages ​​and preprocess the text data;

[0034] A phoneme conversion module, used to randomly divide the preprocessed text data into corresponding numbers according to the language environment type, and convert each text data into the phoneme of the corresponding language environment;

[0035] The phoneme processing module is used to segment the phonemes to form phoneme sequences separated by spaces, mark word boundaries in the text data, and generate a multilingual phoneme representation model.

[0036] A device for constructing a multilingual phoneme representation model, the device comprising at least one processor; and

[0037] a memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned method for constructing a multilingual phoneme representation model.

[0039] A non-volatile computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the one or more processors can execute the above-mentioned method for constructing a multilingual phoneme representation model.

[0040] Beneficial effects: The present invention discloses a method, device, equipment and medium for constructing a multilingual phoneme representation model. Compared with the prior art, the present invention preprocesses data in multiple languages, and randomly divides the preprocessed text data into corresponding numbers according to the types of languages, and converts each portion of text data into phonemes of the corresponding language environment, and then segments the phonemes and marks word boundaries to form a multilingual phoneme representation model. The multilingual phoneme representation model can be directly used as an input phoneme encoder of a speech synthesis model, realizing the phoneme representation of multilingual text to speech, thereby improving the performance of the neural TTS model in terms of naturalness and rhythm, and is also conducive to producing fairly high-quality speech with limited training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0042] Figure 1 A flow chart of a method for constructing a multilingual phoneme representation model provided by an embodiment of the present invention;

[0043] Figure 2 A flowchart of step S100 in the method for constructing a multilingual phoneme representation model provided by an embodiment of the present invention;

[0044] Figure 3 A flowchart of step S200 in the method for constructing a multilingual phoneme representation model provided by an embodiment of the present invention;

[0045] Figure 4 A flowchart of step S300 in the method for constructing a multilingual phoneme representation model provided by an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the functional modules of the device for automatically generating test data provided by an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of the hardware structure of a device for automatically generating test data provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments of the present invention are described below in conjunction with the accompanying drawings.

[0049] The multilingual phoneme representation model constructed by the present invention can be directly used as an input encoder of a language synthesis model [such as TTS (Text-to-Speech Model), which can realize speech synthesis of multilingual text data, help improve the performance of VITS (Vision Transformer, a model based on Transformer architecture, used for image recognition and processing tasks) in the TTS model, obtain a more natural rhythm than the original VITS (Voice-based Text-to-Speech Model) model without pre-training, and generate fairly high-quality synthesized speech with limited training data.

[0050] For TTS models, some speech synthesis works embed context words generated by pre-trained BERT into their standard encoders. During training, the input phoneme sequence is fed into the standard TTS encoder to produce phoneme representations, and its corresponding original text is fed into BERT to obtain context word embeddings.

[0051] To construct the input vector for the TTS decoder, the resulting representation of the input phoneme is concatenated with the BERT-based contextual embedding of the corresponding word to which the phoneme belongs. Thus, BERT helps improve the quality of the output synthesized speech. At this point, the pre-trained BERT is used to indirectly provide additional contextual information to the phoneme representation. Therefore, it may be better if the contextualized phoneme representation is directly learned from a pre-trained BERT type model that is learned from unlabeled phoneme-level data.

[0052] In speech synthesis, it was found that pre-trained phoneme representation models, including PnG BERT (Phoneme and Grapheme, PnG BERT is a variant of the BERT architecture for processing phoneme and grapheme data), hybrid phoneme BERT, and phoneme-level BERT, help improve advanced TTS equipment.

[0053] PnG BERT and Hybrid Phoneme BERT are trained based on the BERT pre-training method, where PnG BERT takes phonemes and glyphs (i.e., word tags) as input, while Hybrid Phoneme BERT takes phoneme and subphoneme tags as input. Phoneme-level BERT is trained based on the ALBERT pre-training method and only takes phonemes as input. In addition to the standard masked token prediction task used in PnG BERT and Hybrid Phoneme BERT, Phoneme-level BERT also proposes an additional auxiliary task, which predicts the corresponding glyph for each phoneme. In neural TTS devices, PnG BERT, Hybrid Phoneme BERT, and Phoneme-level BERT can be directly used as input encoders, making them widely used in counterpart robots and intelligent translation scenarios in various languages.

[0054] In order to solve the above problems, the present invention proposes a method for constructing a multilingual phoneme representation model. Figure 1 , Figure 1 The present invention provides a flowchart of an embodiment of a method for constructing a multilingual phoneme representation model.

[0055] The method for constructing a multilingual phoneme representation model provided in this embodiment is applied to mobile devices, fixed terminal devices, etc. based on artificial intelligence, and its terminal operating system may include a handheld device operating system (iPhone operating system, iOS system), Android system or other operating systems, including but not limited to smart phones, tablet computers, portable computers and desktop servers, etc. Figure 1 As shown, the method specifically comprises the following steps:

[0056] S100: Acquire text data in multiple languages, and pre-process the text data.

[0057] In this embodiment, it is necessary to first collect text data that can be used in a multi-language environment phoneme conversion tool, and then perform deduplication, word or sentence segmentation and normalization on the text data so that it can be applied to phoneme conversion in English and other languages ​​besides English.

[0058] In one embodiment, Figure 2 As shown, step S100 includes:

[0059] S101. Obtain text data in multiple languages ​​from a pre-training corpus.

[0060] Optionally, the pre-training corpus contains 330 million phoneme-described sentences (i.e., text data) in nearly 100 languages ​​and regions. Language data in nearly 100 different languages ​​and regions enables the speech synthesis model to understand and generate speech in multiple languages. The sentences in the pre-training corpus are described in the form of phoneme sequences. Phonemes are symbols that constitute the smallest speech unit in a language, forming the input form of the TTS model. At the same time, 330 million sentences provide rich data for training deep learning models, so that the multilingual phoneme representation model constructed by the present invention can be used for speech synthesis training of large-scale multilingual models, and output multilingual natural speech.

[0061] In one embodiment, obtaining text data in multiple languages ​​from a pre-training corpus includes:

[0062] S1011. Download the wiki40b dataset from the Hugging Face dataset library and obtain text documents belonging to the Wikipedia language and region.

[0063] S1012. Use the Wikipedia dataset to supplement texts in other languages.

[0064] The Hugging Face dataset library is an open source machine learning model library that provides a large number of pre-trained models and datasets. The multilingual datasets iki40b and wikipedia used in the present invention can be directly downloaded from the Hugging Face dataset library. In the specific implementation process, the wiki40b dataset consisting of text documents in 41 Wikipedia languages ​​and regions is first downloaded; then, wikipedia is used to extract text in other languages ​​​​except the languages ​​belonging to wiki40b from the Wikipedia dump.

[0065] Download the wiki40b dataset from the Hugging Face dataset library and use the wikipedia dataset to supplement text data in other languages. Do not include:

[0066] Access the Hugging Face dataset repository by going to the Hugging Face official website and navigating to the dataset repository section.

[0067] Then search for the wiki40b dataset by searching for "wiki40b" in the dataset library to find the dataset.

[0068] Then, according to the information and guidance provided by Hugging Face, download the wiki40b dataset. This dataset contains text documents from 41 Wikipedia languages ​​and regions.

[0069] Then check the content of the wiki40b dataset. After the corpus is downloaded, check the content of the dataset to ensure that it contains the required languages ​​and regions.

[0070] Then determine which languages ​​need to be supplemented, and determine which languages ​​need to be supplemented in addition to the languages ​​included in the wiki40b dataset.

[0071] Afterwards, we supplement it with the Wikipedia dataset, which can be extracted from the Hugging Face dataset library or directly from Wikipedia dumps for language texts that are not part of Wiki40b.

[0072] Then, text extraction is performed, and for each selected language, text content is extracted from Wikipedia pages using appropriate tools or scripts.

[0073] Then, text preprocessing is performed by performing preprocessing on the supplementary language text, including removing useless tags, formatting and cleaning the text.

[0074] Then the data is integrated, and the text data obtained from Wiki40b and Wikipedia are integrated to form a unified multilingual dataset.

[0075] After that, data cleaning and validation are performed, by cleaning the data to remove duplicates, errors and other inconsistencies, and verifying the integrity and quality of the data.

[0076] Finally, the dataset is used to perform subsequent steps such as text-to-phoneme conversion and phoneme segmentation by using the generated dataset.

[0077] The present invention ensures that the text datasets obtained from the Hugging Face dataset library cover a wide range of languages ​​and regions to increase the diversity of multilingual model training, and ensures the consistency of texts in different languages ​​through a unified preprocessing step. The cleaning and verification steps help improve data quality and reduce the impact of noise on model training. The generated dataset can be directly used to train a multilingual phoneme representation model to support downstream tasks such as text to speech.

[0078] S102: Convert all words and sentences of the text data into plain text, and filter out single words and repeated sentences to form a text to be processed.

[0079] In one embodiment, the text data may be text in a language supported by the CharsiuG2P tool (CharsiuG2P is a Transformer-based tool for grapheme-to-phoneme conversion in 100 languages), and all words and sentences of the text data are converted into plain text to improve the compatibility of phoneme learning during training.

[0080] In text data, sentences include words, and words include single words, phrases, idioms, proper nouns, terms, etc. Among them, when converting words and sentences of text data into plain text, it includes: changing uppercase to lowercase, setting default font, adjusting text color and font size, removing formatting, images and other non-text elements, etc.

[0081] Taking text data in languages ​​with uppercase and lowercase grammar such as English, German, French, Italian, Portuguese, and Dutch as an example, when converting words and sentences in the text data into plain text, the uppercase in the text data needs to be converted to lowercase to avoid confusion between uppercase and lowercase for phoneme learning.

[0082] After conversion to plain text, the plain text needs to be cleaned up, mainly removing single words and repeated sentences to form the text to be processed, and the plain text data is saved as a new file. The extension of the plain text file is .txt. If the original data is important, back it up before and after the conversion in case the original corpus is lost.

[0083] In addition to the CharsiuG2P toolkit, there are several tools and methods available for phoneme conversion, including:

[0084] Espeak and Festival: Espeak is an open source software speech synthesizer that can convert text to speech, including at the phoneme level. Festival is a general-purpose speech synthesis system that supports text-to-speech conversion in many languages.

[0085] Phonemizer: Phonemizer is a multilingual text-to-phoneme conversion tool that supports conversion in multiple languages ​​and allows users to customize phoneme mapping.

[0086] Mozilla TTS: Mozilla's TTS project provides a deep learning-based text-to-speech service that can handle phoneme conversion.

[0087] Kaldi: Kaldi is an open source speech recognition research toolkit that contains various tools for speech processing, including phoneme conversion.

[0088] CMU Sphinx: CMU Sphinx is a speech recognition system that provides rule-based phoneme conversion tools.

[0089] Google Cloud Text-to-Speech: The Cloud Text-to-Speech service provided by Google supports phoneme conversion in multiple languages.

[0090] DeepPhonemizer: A deep learning-based phoneme conversion tool that can provide high-quality phoneme sequences for multiple languages.

[0091] Hugging Face datasets library: Although mainly used for loading and processing datasets, Hugging Face's datasets library can also be used to obtain phoneme information for certain languages.

[0092] Homemade rules or dictionaries: Phoneme conversion is performed according to language-specific rules or dictionaries, which usually requires linguistic knowledge and a deep understanding of the specific language.

[0093] Self-built phoneme conversion model: Use machine learning methods, such as neural networks, to train a model specifically for phoneme conversion.

[0094] Each tool and method has its advantages and limitations. The present invention uses the CharsiuG2P toolkit to achieve the best phoneme conversion effect.

[0095] S103, segmenting the text to be processed into words and sentences.

[0096] When segmenting, the spaCy toolkit can be used to perform word and sentence segmentation on all text documents in each language.

[0097] For example, when segmenting the Chinese sentence: "I love Tiananmen Gate in Beijing", it can be segmented into: "I love Tiananmen Gate in Beijing".

[0098] Word and sentence segmentation may include the following steps:

[0099] Determine segmentation rules: Determine the segmentation rules for words and sentences based on the language characteristics of the text. For example, spaces are usually used to separate words in French, German, English, etc., and sentences end with a period (.), question mark (?), or exclamation mark (!).

[0100] Text cleaning: remove noise from text, such as extra spaces, special characters, HTML tags, etc.

[0101] Word segmentation: For languages ​​that are segmented based on spaces, such as French, German, and English, you can directly segment the text into words based on spaces.

[0102] For languages ​​without obvious word boundaries, such as Chinese and Japanese, specific word segmentation algorithms are needed to identify words.

[0103] Sentence segmentation: Identify sentence-ending symbols such as full stops, question marks, exclamation marks, etc., and split the text into sentences. For languages without obvious sentence boundaries, statistical or machine learning-based methods may be required to identify sentence boundaries.

[0104] Stop word removal (optional): In some applications, it may be necessary to remove common stop words (such as "de", "he", "shi", etc.) to reduce data redundancy.

[0105] Lemmatization or stemming (optional): Reduce words to their base form or extract the stem to unify the representation of words in different forms.

[0106] Part-of-speech tagging (optional): Tag the part of speech (noun, verb, adjective, etc.) for each word, which helps to understand the grammatical structure of the text.

[0107] Checking and verification: Check whether the segmentation results meet the expectations and make manual corrections if necessary.

[0108] Data storage: Store the segmented words and sentences in an appropriate data structure for subsequent processing and analysis.

[0109] Step by step, after step 103, it further includes:

[0110] S1031. Obtain the text to be processed that needs to be normalized;

[0111] S1032. Use a text-to-speech tool to convert the text to be processed from written form to oral form.

[0112] For example, the present invention can use the NVIDIA NeMo toolkit to convert text from written form to oral form that is only applicable to English, German, Spanish, Vietnamese and Chinese, which can make the naturalness of the subsequent speech synthesis model higher.

[0113] This application determines and extracts the text data that needs to be text-normalized. Text normalization refers to converting written text into a form that is more suitable for oral expression, which is crucial for improving the naturalness of text-to-speech (TTS) systems. And use a text-to-speech tool to convert the text to be processed from written form to oral form, and the written text is converted into a more colloquial expression. Usually, it has the following effects:

[0114] Improve naturalness: The converted text is closer to the way people speak in daily life, making the final speech output more natural and fluent.

[0115] Increase comprehensibility: The normalization process helps to improve the comprehensibility of the speech output, especially for texts containing complex sentence structures or technical terms.

[0116] Reducing ambiguity: Normalization can reduce ambiguity in the text and ensure accuracy during speech synthesis.

[0117] Adaptability: Text normalization tools can usually adapt to different contexts and language styles, providing speech output that better meets the expectations of the target audience.

[0118] Through step S100, the noisy text data mainly includes:

[0119] 1) Remove non-standard characters, special symbols, and punctuation marks from the text, which may include emojis, copyright symbols, registered trademark symbols, etc.

[0120] 2) If the text data comes from a web page, remove HTML or XML tags.

[0121] Numbers and URLs: Sometimes the numbers and URLs in the text may not be the targets of preprocessing, so they may be removed or replaced.

[0122] 3) Extra spaces, including consecutive spaces, spaces at the beginning or end of a paragraph, etc.

[0123] 4) Remove unnecessary line breaks and tab characters to unify the text format.

[0124] 5) Stop words: Remove commonly used words that do not have much significance for text analysis, such as "de", "he", "shi", etc.

[0125] 6) Correct spelling mistakes or remove unrecognizable spellings.

[0126] 7) Remove formatting information such as font styles and color codes from the text.

[0127] 8) Remove repeatedly occurring sentences or phrases from the text.

[0128] 9) Remove non-target language text. If the preprocessing module focuses on a specific language, text in other languages needs to be removed.

[0129] 10) Remove metadata from the text, such as author names, timestamps, etc. These information usually do not help in text content analysis.

[0130] 11) Remove irregular abbreviations or shorthands and convert them to full written forms or unified abbreviated forms.

[0131] 12) Remove noise or interference from the text, such as image noise generated during scanning of documents or character recognition errors.

[0132] By removing this noise, the quality of the text data can be improved, making it more suitable for subsequent analysis and processing, such as phoneme conversion and speech synthesis tasks.

[0133] S200 , randomly dividing the preprocessed text data into corresponding numbers of copies according to the language environment type, and converting each copy of the text data into phonemes of the corresponding language environment.

[0134] The language environment can be understood as the language and dialect characteristics of the text data, such as Chinese, English, French, etc., and their respective dialects or regional variants. In order to train and evaluate the model, this application divides the dataset into different parts, such as training set, validation set and test set. The division of these datasets is random to ensure the generalization ability of the model.

[0135] Phonemes are the smallest units of speech that can distinguish meaning in a language. For example, in English, "cat" and "cut" are similar in spelling but pronounced differently because they contain different phonemes. Therefore, the words in the text are converted into corresponding phoneme sequences for subsequent speech synthesis or speech recognition processing.

[0136] After converting text data into phoneme sequences, these phoneme sequences can be used to drive the speech synthesis engine to generate natural-sounding and fluent speech output. When training the model, it can recognize phonemes in different language environments to improve the model's recognition accuracy for specific languages ​​or dialects.

[0137] By randomly dividing the data set, when the text data converted into phoneme sequences is used for speech synthesis, it can more naturally reflect the pronunciation characteristics of the original text, ensuring that the model will not overfit to a specific data subset, thereby improving the generalization ability of the model on unseen data, making it easier for the model to learn during training. Phoneme-level processing can better capture the rhythmic characteristics of the language and improve the rhythmic performance of the TTS system.

[0138] The randomly split dataset provides the flexibility to construct datasets of different types and sizes to meet different research and application needs, and can effectively convert multilingual text data into phoneme sequences, providing high-quality input for text-to-speech and other speech processing tasks.

[0139] In one embodiment, Figure 3 As shown, step S200 includes:

[0140] S201. Randomly divide the preprocessed text data into corresponding numbers according to the language environment type, and allocate one copy of the text data to be converted to each language.

[0141] When performing phoneme conversion, for languages ​​whose locales do not have their own Wikipedia data, the Wikipedia data of the language can be randomly divided into equal parts, and the number of sentences in each part is the same, and each part corresponds to a locale. If English is a language that does not have its own Wikipedia data, when performing phoneme conversion, 67 million English sentences can be divided into two equal parts, and then converted into phoneme descriptions of British English (eng-uk) and American English (eng-us), respectively, so as to eliminate the influence of British English and American English on phoneme representation learning.

[0142] S202: Convert the text data to be converted into phonemes of a corresponding language environment through a multilingual conversion model.

[0143] When converting a sentence to its phonemic description, the grapheme-to-phoneme conversion toolkit CharsiuG2P can be used. CharsiuG2P is a Transformer-based multilingual tool for converting graphemes to phonemes, supporting graph-to-phoneme conversion in about 100 languages. The pre-trained CharsiuG2P is a powerful multilingual transformer-based model that generates the pronunciation of a word given its orthographic form and an ISO 639-3 language code pair.

[0144] For example, taking "Today's weather is really good" as an example, the general steps of converting it to phonemes may include: using the word segmentation tool to segment the sentence into "today", "weather" and "really good", and then converting each word into pinyin, such as "today" into "jīntiān", and then further converting the pinyin into phonemes, such as "jīn" can be converted into a phoneme sequence

[0145] For example, when using the CharsiuG2P tool to convert the English word "model" into a phoneme sequence, the phoneme sequence is

[0146] S203: Identify whether the phoneme is in the pronunciation dictionary of the multilingual conversion model.

[0147] After the phoneme conversion, it is first identified whether the converted phoneme is in the pronunciation dictionary of the toolkit, so as to facilitate processing according to the specific phoneme.

[0148] S204: When the recognized phoneme is in the pronunciation dictionary of the multilingual conversion model, generate a phoneme description of the text data using the pronunciation dictionary; when the recognized phoneme is not in the pronunciation dictionary of the multilingual conversion model, generate a phoneme description of the text data using the multilingual conversion model.

[0149] The pronunciation dictionary is used to generate the phonetic description of the word in the target language / region of the CharsiuG2P toolkit, otherwise, if the word is not in the vocabulary, we use the pre-trained CharsiuG2P model to generate its phonemic description.

[0150] Assuming that the phoneme description of the English word "ruminate" is to be converted, if the phonemes of the word are already in the pronunciation dictionary, the present invention directly queries the dictionary to obtain its phoneme sequence. For example, the phoneme description of "ruminate" can be found as / ′ru:mIneIt / in the pronunciation dictionary of the CharsiuG2P toolkit.

[0151] When the phoneme is not in the pronunciation dictionary of the multilingual conversion model: You can try to convert a non-common word or a proper noun, such as "BigCiDian" (a multilingual pronunciation dictionary project name), which may not appear in the standard pronunciation dictionary. In this case, the CharsiuG2P toolkit is used to generate the phoneme description.

[0152] S300: Segment the phonemes to form phoneme sequences separated by spaces, mark word boundaries in the text data, and generate a multilingual phoneme representation model.

[0153] The CharsiuG2P toolkit converts each input word into a continuous sequence of phonemes without phoneme boundary indicators (such as blanks). In order to better map phonemes and speech, phoneme segmentation must be performed on the output of the CharsiuG2P toolkit. Next, phoneme segmentation is performed using the segments toolbox (the segments" toolbox is a multilingual text-to-phoneme conversion tool). Therefore, the input word is now converted into a sequence of phonemes separated by spaces. Since blanks are used to separate phonemes, in order to distinguish phonemes belonging to different words or word tokens, this application uses metasymbols to mark word boundaries.

[0154] like Figure 4 As shown, step S300 includes:

[0155] S301. Segment the continuous phoneme sequence output by the multilingual conversion model.

[0156] S302: inserting a space into the segmented phoneme sequence as a phoneme boundary indicator.

[0157] S303: Use metasymbols to mark word boundaries to distinguish phonemes of different words.

[0158] When performing phoneme segmentation, it is first necessary to identify the natural boundaries in the continuous phoneme sequence, which can be achieved through the acoustic characteristics of the phonemes or language rules.

[0159] Then use the segments toolbox to segment the phonemes. In some cases, you may need to customize the segmentation rules according to the specific language or application scenario. When segmenting, the phonemes in some languages ​​may be concurrent, that is, multiple phonemes are pronounced at the same time. This requires special processing methods to correctly segment.

[0160] For those complex phoneme segmentation tasks, you can leverage deep learning-based models such as DeepPhonemizer, a multilingual grapheme-to-phoneme modeling library that uses the latest deep learning techniques and is optimized for use in production systems.

[0161] The segmented phoneme sequence is then evaluated to ensure its accuracy. If necessary, the model or segmentation rules are fine-tuned to improve the results. The phoneme sequence obtained by segmentation can be directly used in text-to-speech synthesis, speech recognition or other speech processing tasks, or can be further processed to improve the naturalness and accuracy of subsequent speech contract and speech recognition.

[0162] Since the segmented phonemes are connected phoneme sequences, the present application adopts inserting spaces in the phoneme sequence as phoneme boundary indicators. The present invention adopts spaces as indicators of phoneme boundaries, making the phoneme sequence easier to read and understand, especially in a polyphone environment. Moreover, when performing subsequent speech synthesis or speech recognition, the spaces can be used as processing units to help the algorithm identify the boundaries between phonemes, thereby performing speech processing more accurately.

[0163] In addition, when synthesizing speech, the alignment between the phoneme sequence and the audio signal is key, and using spaces as separators can make it easier to synchronize text and audio. In addition, clear phoneme boundaries can reduce the difficulty of the model learning the phoneme segmentation rules and improve training efficiency. Clear phoneme boundary indications can help the model better generalize to new or unseen words.

[0164] Support for multi-language processing: For languages ​​that use different phoneme sets, using spaces as common separators helps unify the processing flow.

[0165] The steps of inserting spaces into the segmented phoneme sequence include:

[0166] Use a phoneme segmentation tool or algorithm to segment a continuous phoneme sequence into individual phonemes;

[0167] Recognize and identify the natural boundaries between phonemes, which are where spaces are inserted;

[0168] Insert spaces at the boundary of each phoneme to separate the phonemes;

[0169] Check whether the phoneme sequence after inserting spaces meets the expected segmentation effect and make adjustments if necessary.

[0170] For example, for the phoneme sequence of the English word "hello" After splitting and inserting spaces, it can be expressed as Each phoneme is separated by spaces, clearly indicating the boundaries of the phonemes.

[0171] The word boundaries are marked with a metasymbol (U+2581) in the segmented phoneme sequence to distinguish the phonemes of different words, which usually involves the following steps:

[0172] In step S302, the continuous phoneme sequence is segmented according to the natural boundaries of the phonemes, and the meta-symbols used as markers are determined in step S303. These symbols should be visually distinguishable and not confused with any actual phonemes in the phoneme sequence. Common meta-symbols include spaces, underscores or other special characters. The present invention uses u+2581, i.e., underscore, of the Unicode symbol library as a phoneme boundary indicator.

[0173] In the segmented phoneme sequence, the selected meta-symbol is inserted at the appropriate position according to the word boundary. For example, in English, the phoneme sequence (hello) can be changed to insert an underscore between phonemes to distinguish the individual phonemes.

[0174] Check whether the sequence after inserting the meta-symbol correctly represents the phoneme boundaries of the word, and make necessary adjustments to ensure the accuracy and readability of the phoneme sequence.

[0175] Application to speech processing systems: The phoneme sequences marked with metasymbols are applied to speech processing systems, such as text-to-speech (TTS) or automatic speech recognition (ASR) systems, to improve the processing efficiency and accuracy of the systems.

[0176] The benefits of using metasymbols to mark word boundaries include:

[0177] Improved readability: Clear word boundaries help understand the structure of phoneme sequences.

[0178] Ease of processing: In speech synthesis or recognition systems, clear word boundaries help the algorithm process speech data more accurately.

[0179] Improved alignment: In TTS systems, the marking of word boundaries helps achieve precise synchronization between text and audio.

[0180] Simplify model training: In machine learning model training, clear phoneme boundaries can simplify the model learning process and improve training efficiency.

[0181] For example, in the BERT model, special tokens such as<cls>and <sep>To mark the beginning of the sequence and the boundary of the segment 175. This method can help the model better understand the structure of the input sequence, thereby improving the performance of the model on various natural language processing tasks. In the labeling of phoneme sequences, a similar method can be used to use specific meta-symbols to mark the boundaries of words.

[0182] After the multilingual phoneme representation model is generated, the multilingual phoneme representation model can also be optimized to increase the naturalness of the subsequent speech synthesis. Therefore, after step S300, the following steps can also be included:

[0183] Set the parameters of the multilingual phoneme representation model, including the number of Transformer blocks, hidden size, and number of self-attention heads;

[0184] The multilingual phoneme representation model is pre-trained using a dynamic masking strategy of the RoBERTa pre-training method to optimize the multilingual phoneme representation model.

[0185] The multilingual phoneme representation model of the present invention has the same model architecture as the BERT foundation, i.e., a multi-layer bidirectional Transformer encoder, wherein the number of Transformer blocks, hidden size, and number of self-attention heads are 12,768, and 12, respectively. For pre-training, the text data of the masked language modeling input is used, and the RoBERTa pre-training method is followed to robustly optimize BERT for better performance, i.e., using a dynamic masking strategy without using the next sentence prediction target. Given that BERT and RoBERTa are prior art, their architectures are not described in detail here.

[0186] This application uses a blank tagger that produces a vocabulary of 1960 phoneme types. Therefore, there are 876M parameters in total. To train on a multilingual pre-trained corpus, the present invention uses the RoBERTa implementation in the fairseq library, sets the maximum sequence length to 512, uses the Adam optimization model (Adam Optimizer is an optimization algorithm widely used in deep learning), and uses a batch size of 1024 sequence blocks and a peak learning rate of 0.0001 in 8 A100 GPUs (40GB each).

[0187] Furthermore, the multilingual phoneme representation model can be repeatedly trained and learned. During training, the multilingual phoneme representation model is based on the RoBERTa pre-training method and is trained using the BERT basic model configuration. After training, it can be used as the input phoneme encoder of the strong model VITS, which helps to improve the performance of VITS, obtain a more natural rhythm than the original VITS without pre-training, and produce fairly high-quality synthesized speech with limited training data.

[0188] The specific technical advantages of the present invention in terms of multi-language support include:

[0189] Large-scale multilingual dataset: A pre-trained corpus of 330 million phonemes describing sentences, covering nearly 100 languages ​​and regions, provides a rich data foundation for multilingual support.

[0190] BERT-based model architecture: It uses the same multi-layer bidirectional Transformer encoder architecture as BERT, which has been proven to be effective on multiple languages ​​and can capture complex relationships between languages.

[0191] RoBERTa pre-training method: Using RoBERTa’s pre-training method, the model performance is optimized through a dynamic masking strategy instead of using a fixed mask ratio, which helps to better adapt to the characteristics of different languages.

[0192] Multilingual text-to-phoneme conversion: Using the CharsiuG2P toolkit, it supports text-to-phoneme conversion in about 100 languages ​​and regions, ensuring that texts in different languages ​​can be accurately converted into phoneme sequences.

[0193] Phoneme segmentation technology: Use the segments toolbox to perform phoneme segmentation, separate phonemes with spaces, and use special metasymbols to mark word boundaries, which improves the readability and alignment accuracy of phoneme sequences.

[0194] Multilingual adaptability: The model is exposed to multiple languages ​​during the pre-training phase, which enables the model to adapt to the characteristics of different languages ​​and improves the model's generalization ability in a multilingual environment.

[0195] No language-specific text normalization tools are required: For languages ​​such as English, German, Spanish, Vietnamese, and Chinese, text normalization is applied to convert written text into spoken form, although no normalization tool is found for all languages, this practice reflects the consideration of specific language characteristics.

[0196] Improved TTS performance: The model acts as an input phoneme encoder, which can significantly improve the performance of neural TTS models in terms of naturalness and prosody, especially under limited training data conditions.

[0197] Optimized hyperparameter settings: The Adam optimizer and specific learning rate schedule are used to optimize large-scale multilingual datasets, which improves the efficiency and effectiveness of model training.

[0198] Cross-lingual phoneme representation learning: Through pre-training, the model learns cross-lingual phoneme representations, which helps improve the performance of advanced TTS systems, especially when dealing with non-English languages.

[0199] Another embodiment of the present invention provides a device for constructing a multilingual phoneme representation model, such as Figure 5 As shown, the device 1 comprises:

[0200] A preprocessing module 11 is used to obtain text data in multiple languages ​​and preprocess the text data to ensure that the text data is suitable for subsequent processing, including cleaning data, removing irrelevant characters, and normalizing text;

[0201] The phoneme conversion module 12 is used to randomly divide the preprocessed text data into corresponding parts according to the language environment type, convert each part of the text data into the phoneme of the corresponding language environment, and convert the text data into a phoneme sequence to provide basic data for applications such as speech synthesis and speech recognition;

[0202] The phoneme processing module 13 is used to segment the phonemes to form phoneme sequences separated by spaces, mark word boundaries in the text data, generate a multilingual phoneme representation model, and generate clear and accurate multilingual phoneme representations, which can be directly used to train speech-related machine learning models.

[0203] "The modules described in the present invention refer to a series of computer program instruction segments that can perform specific functions. These modules are more suitable for describing the execution of the test data automatic generation process. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0204] In one embodiment, the preprocessing module 11 includes:

[0205] Download unit, used to obtain text data in multiple languages ​​from pre-trained corpora;

[0206] A preprocessing unit, used to convert all words and sentences of text data into plain text, and filter out single words and repeated sentences to form a text to be processed;

[0207] The text segmentation unit is used to segment the text to be processed into words and sentences. In one embodiment, the phoneme conversion module 12 includes:

[0208] A random allocation unit, used for randomly dividing the preprocessed text data into corresponding numbers according to the language environment type, and allocating one copy of the text data to be converted to each language;

[0209] A phoneme conversion unit, used for converting the text data to be converted into phonemes of a corresponding language environment through a multilingual conversion model;

[0210] A phoneme recognition unit is used to identify whether the phoneme is in the pronunciation dictionary of the multilingual conversion model; when the phoneme is identified in the pronunciation dictionary of the multilingual conversion model, the pronunciation dictionary is used to generate a phoneme description of the text data; when the identified phoneme is not in the pronunciation dictionary of the multilingual conversion model, the multilingual conversion model is used to generate a phoneme description of the text data.

[0211] In one embodiment, the phoneme processing module 13 includes:

[0212] A phoneme segmentation unit, used to segment the continuous phoneme sequence output by the multilingual conversion model;

[0213] A separator unit, used for inserting a space as a phoneme boundary indicator in the segmented phoneme sequence;

[0214] The boundary marking unit is used to mark word boundaries using meta-symbols to distinguish the phonemes of different words.

[0215] Another embodiment of the present invention provides a device for constructing a multilingual phoneme representation model, such as Figure 6 As shown, the device 10 includes:

[0216] One or more processors 110 and memory 120, Figure 6 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0217] The processor 110 is used to complete various control logics of the device 10, and it can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISCMachine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP and / or any other such configuration.

[0218] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the method for constructing a multilingual phoneme representation model in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the method for constructing a multilingual phoneme representation model in the above method embodiment.

[0219] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created according to the use of the device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0220] One or more units are stored in the memory 120, and when executed by one or more processors 110, the following steps are implemented:

[0221] Acquiring text data in multiple languages ​​and preprocessing the text data;

[0222] The preprocessed text data is randomly divided into corresponding numbers according to the language environment type, and each piece of text data is converted into the phoneme of the corresponding language environment;

[0223] The phonemes are segmented to form phoneme sequences separated by spaces, and word boundaries in the text data are marked to generate a multilingual phoneme representation model.

[0224] An embodiment of the present invention provides a non-volatile computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by one or more processors, the above steps S100 to S300 are implemented.

[0225] As an example, non-volatile storage media can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) as external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0226] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the processes of the above-mentioned method embodiments when executed. The storage medium can be a memory, a disk, a floppy disk, a flash memory, an optical storage device, etc.

[0227] Compared with the prior art, the present invention supports a wider range of languages ​​and broadens the scope of application of the TTS system. Unlike BERT, which indirectly provides context information, this solution directly learns phoneme representations from unlabeled phoneme data. The pre-training method of RoBERTa is adopted, and the dynamic masking strategy is used to optimize the model performance.

[0228] The present invention improves the naturalness and rhythm of synthesized speech by using a pre-trained multilingual phoneme representation model as the input encoder of the TTS model, and can generate high-quality speech output even when the training data is limited. It provides new possibilities for pre-training models of phoneme representation for non-English languages ​​and promotes the development of multilingual TTS technology.

[0229] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.< / sep> < / cls>

Claims

1. A method for constructing a multilingual phoneme representation model, characterized in that: include: Acquiring text data in multiple languages ​​and preprocessing the text data; The preprocessed text data is randomly divided into corresponding numbers according to the language environment type, and each piece of text data is converted into the phoneme of the corresponding language environment; The phonemes are segmented to form phoneme sequences separated by spaces, and word boundaries in the text data are marked to generate a multilingual phoneme representation model.

2. The method for constructing a multilingual phoneme representation model according to claim 1, characterized in that: The step of acquiring text data in multiple languages ​​and preprocessing the text data includes: Get text data in multiple languages ​​from pre-training corpora; Convert all words and sentences of text data into plain text, and filter out single words and repeated sentences to form text to be processed; The text to be processed is segmented into words and sentences.

3. The method for constructing a multilingual phoneme representation model according to claim 2, characterized in that: The step of acquiring text data in multiple languages ​​from a pre-training corpus includes: Download the wiki40b dataset from the Hugging Face dataset repository and obtain text documents belonging to the Wikipedia language and region; Use the Wikipedia dataset to supplement text in other languages.

4. The method for constructing a multilingual phoneme representation model according to claim 2, characterized in that: After the text to be processed is segmented into words and sentences, the method further includes: Get the text to be processed that needs to be normalized; The text to be processed is converted from written form to spoken form using a text-to-speech tool.

5. The method for constructing a multilingual phoneme representation model according to claim 1, characterized in that: The step of randomly dividing the preprocessed text data into corresponding numbers of copies according to the language environment type and converting each copy of the text data into phonemes of the corresponding language environment comprises: The preprocessed text data is randomly divided into corresponding numbers according to the language environment type, and a portion of the text data to be converted is allocated to each language; Converting the text data to be converted into phonemes of a corresponding language environment through a multilingual conversion model; identifying whether the phoneme is in a pronunciation dictionary of a multilingual conversion model; When the recognized phoneme is in the pronunciation dictionary of the multilingual conversion model, the phoneme description of the text data is generated using the pronunciation dictionary; when the recognized phoneme is not in the pronunciation dictionary of the multilingual conversion model, the phoneme description of the text data is generated using the multilingual conversion model.

6. The method for constructing a multilingual phoneme representation model according to claim 1, characterized in that: The step of segmenting the phonemes to form phoneme sequences separated by spaces, marking word boundaries in the text data, and generating a multilingual phoneme representation model comprises: Segment the continuous phoneme sequence output by the multilingual conversion model; Inserting spaces as phoneme boundary indicators in the segmented phoneme sequence; Use metasymbols to mark word boundaries to distinguish the phonemes of different words.

7. The method for constructing a multilingual phoneme representation model according to claim 1, characterized in that: After segmenting the phonemes to form phoneme sequences separated by spaces and marking word boundaries in the text data to generate a multilingual phoneme representation model, the method further includes: Set the parameters of the multilingual phoneme representation model, including the number of Transformer blocks, hidden size, and number of self-attention heads; The multilingual phoneme representation model is pre-trained using a dynamic masking strategy of the RoBERTa pre-training method to optimize the multilingual phoneme representation model.

8. A device for constructing a multilingual phoneme representation model, characterized in that: include: A preprocessing module, used to obtain text data in multiple languages ​​and preprocess the text data; A phoneme conversion module, used to randomly divide the preprocessed text data into corresponding numbers according to the language environment type, and convert each text data into the phoneme of the corresponding language environment; The phoneme processing module is used to segment the phonemes to form phoneme sequences separated by spaces, mark word boundaries in the text data, and generate a multilingual phoneme representation model.

9. A device for constructing a multilingual phoneme representation model, characterized in that: The device comprises at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for constructing a multilingual phoneme representation model according to any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium, characterized in that: The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by one or more processors, enable the one or more processors to execute the method for constructing a multilingual phoneme representation model as described in any one of claims 1 to 7.

Images