Multi-speaker multi-lingual speech synthesis system based on self-learning text representation

Abstract

The application discloses a multi-speaker multi-language speech synthesis system based on self-learning text representation, self-learning multi-language text representation, and is embodied in two modules, namely a text-to-SMTR prediction module and an SMTR-to-multi-language acoustic spectrum prediction module. Specifically, the application comprises the following steps: constructing an SMTR extraction method based on a self-learning system; constructing a multi-language text-to-SMTR prediction method; constructing an SMTR-to-multi-language acoustic spectrum prediction method; and constructing an end-to-end multi-language speech synthesis method based on SMTR fusion. The application can improve the accuracy of multi-language speech synthesis.

Description

Technical Field

[0001] This invention belongs to the field of speech synthesis and relates to a self-learning multilingual text representation method.

[0002] Multilingual Text Representation (SMTR) End-to-End Multilingual Speech Synthesis

[0003] The Multilingual Speech Synthesis system specifically utilizes a self-learning multilingual text representation (SMTR) to enable a target speaker who can only speak one language to speak multiple languages ​​through speech synthesis. Background Technology

[0004] Text-to-speech (TTS), also known as text-to-speech conversion, aims to convert text into highly intelligible, natural, fluent, and expressive human-like speech using computers. Research by the Japan Human Information Science Laboratory found that as early as the 18th century, "talking machines" using rubber tubes to create a laryngeal model to mimic human voices were invented. The "speech-to-image playback machine" developed by the Haskins Laboratory in the United States in 1948 is widely recognized as the earliest form of speech synthesis. However, systematic research into text-to-speech conversion only began to emerge in the late 1970s, with the most famous example being the mechanical speech synthesizer invented by von Kempelen, which mimicked the entire articulation process of human speech. In recent decades, with the explosion of technology, speech synthesis technology has rapidly developed, progressing from parametric synthesis methods to waveform concatenation-based synthesis methods, statistical parameter-based synthesis methods, deep neural network-based speech synthesis, and end-to-end speech synthesis technology.

[0005] Parametric synthesis is a method that first analyzes and processes the speech signal, and then represents the speech signal by controlling a finite number of parameters while considering storage capacity and speech quality. Parametric synthesis methods can be divided into linear predictive synthesis and formant synthesis. Both of these methods utilize a source-filter model for speech synthesis.

[0006] The waveform concatenation-based synthesis method relies on a pronunciation corpus, which consists of pre-recorded pronunciation units, ranging from phonemes to entire sentences. During synthesis, given the text input to be synthesized, pronunciation units that match the text are selected from the corpus, and then these pronunciation units are concatenated together to obtain the speech corresponding to the text to be synthesized.

[0007] Among statistical parameter-based synthesis methods, the Hidden Markov Model (HMM)-based parametric speech synthesis model is the most representative. In the model training phase, acoustic feature parameters extracted from the speech signal and contextual information are first used to train the spectral parameter model, fundamental frequency parameter model, and duration model. Then, decision trees are used to process these models to obtain the final prediction model. In the model testing phase, the text to be synthesized is input into the trained model, which predicts the corresponding acoustic feature parameters and duration parameters. Finally, the speech is synthesized by a speech synthesizer to produce the corresponding speech.

[0008] The main structure of a neural network-based speech synthesis model is very similar to that of a Hidden Markov Model (HMM)-based statistical parametric speech synthesis model, consisting of four main parts: front-end text processing, a duration prediction model, an acoustic model, and a vocoder. The front-end text processing mainly performs text normalization, polyphonic character disambiguation, prosodic prediction, and glyph-to-phonetic conversion on the input text. The duration prediction model takes the phoneme sequence obtained after front-end processing as input and predicts the duration corresponding to each phoneme. The acoustic model takes the linguistic features processed by the duration prediction model as input and outputs the acoustic features. Finally, the acoustic features predicted by the acoustic model are processed by the vocoder to obtain the speech corresponding to the input text.

[0009] Compared to traditional speech synthesis methods, end-to-end speech synthesis models require less expert knowledge and human intervention, directly inputting text and outputting speech waveforms. End-to-end speech synthesis models reduce the demands on linguistic knowledge, making it easy to synthesize speech in different languages. Structurally, an end-to-end speech synthesis model consists of three parts: a front-end processing model, an acoustic model, and a vocoder. The acoustic model of the end-to-end speech synthesis model takes a phoneme sequence as input and directly predicts the acoustic features needed for audio synthesis. The output acoustic features, using a vocoder built on a neural network, can synthesize speech waveforms that closely resemble human speech, exhibiting high naturalness and intelligibility.

[0010] With the rapid development of end-to-end speech synthesis, monolingual speech synthesis is very popular in both academia and industry, and multilingual speech synthesis is also gradually developing. There are various technologies used for multilingual speech synthesis, including:

[0011] (1) In order to enable multiple languages ​​to be input in a unified format, the International Phonetic Alphabet (IPA) was selected as the unified input set, and then the IPA was converted into speech features containing a lot of language information.

[0012] (2) In order to overcome the problem of different languages ​​in the multilingual speech synthesis task, a language marker sequence is added to the encoder of the end-to-end speech synthesis model. This sequence has a one-to-one correspondence with the phoneme sequence. The two are concatenated together and input into the text encoder.

[0013] (3) Set up a separate encoder for each language and add a parameter generator to process the language type information corresponding to the phonemes and then concatenate it to each layer of the encoder. Finally, input the output of the encoder into the decoder and speaker classifier.

[0014] (4) First, a cross-language language model is used to output the input text as word vectors, which contain contextual information. Then, the word vectors are concatenated with the encoder output and phonemes to make the content input to the encoder contain more linguistic information.

[0015] These technologies improve the performance of multilingual speech synthesis from the perspectives of the smallest basic unit representation of language, speaker classification, and language classification. However, the characteristics of multilingual speech synthesis tasks mean that it is affected by the datasets of multiple languages ​​and the dependence of each language on the speaker's personality. As a result, existing multilingual speech synthesis technologies have entered a dilemma where it is difficult to achieve both language naturalness and speaker similarity. Summary of the Invention

[0016] To address the issue of inconsistent symbols across language units, firstly, a self-learning multilingual text representation (SMTR) is proposed to solve the cross-lingual challenges in multilingual speech synthesis. Secondly, the multilingual speech synthesis task is divided into two modules for training: text-to-SMTR and real-SMTR-to-audio. In addition, an adversarial speaker classifier is added to the second module, which effectively improves the accuracy and speaker similarity of multilingual speech synthesis.

[0017] End-to-end speech synthesis models, such as Tacotron, Tacotron2, and FastSpeech, have significantly simplified the complexity of traditional speech synthesis pipelines and can already synthesize speech that is highly similar to human voices. However, these achievements are limited to monolingual speech synthesis, and there is still much room for improvement in synthesizing speech from a monolingual speaker speaking multiple languages. The purpose of this invention is to demystify the dependency between language content and speaker personality features in multilingual speech synthesis tasks. This is achieved through stepwise training of the proposed SMTR features and the addition of a speaker classifier to the model.

[0018] The technical solution of this invention is a multi-speaker, multilingual speech synthesis system based on self-learning text representation, which specifically includes the following four steps:

[0019] (1) Constructing an SMTR extraction method based on a self-learning system

[0020] SMTR is an extraction of a self-learning system, which is built on a self-supervised learning framework for speech recognition.

[0021] Self-supervised training is performed using unlabeled data, while labeled data guides the model to predict specific words or phonemes.

[0022] Ultimately, the optimal audio representation is learned.

[0023] We use this self-learning system to learn discrete, latent speech unit characteristics from real audio, and then extract the SMTR containing information for each specific time frame of the speech at the output.

[0024] The challenge of multilingual speech synthesis technology lies in its cross-speaker and cross-language nature. We utilize the unsupervised trained fully convolutional neural network Wav2Vec 2.0 to find a reliable feature that can represent multiple language symbols while maintaining the intelligibility and naturalness of the synthesized speech. It can be divided into three parts: a speech feature encoding network, an SMTR representation network, and a product quantization module.

[0025] (2) Constructing a prediction method for multilingual text to SMTR

[0026] SMTR is a time-to-class matrix containing linguistic information features of cross-lingual utterances at each specific time frame. Only by constructing a method for predicting SMTR from multilingual text can multilingual speech synthesis based on multiple monolingual corpora of different languages ​​be achieved. We use a method similar to Tacotron2, based on a neural network for feature prediction from multilingual text to SMTR. This method includes an encoder and a feature prediction network with an attention mechanism, enabling the prediction of SMTR from input multilingual character sequences, thus achieving the prediction of cross-lingual features, i.e., SMTR, from text in different languages.

[0027] (3) Constructing a SMTR-to-multilingual spectrogram prediction method

[0028] On the one hand, from the perspective of cross-lingual tasks in multilingual speech synthesis, the language-independent nature of SMTR gives it an advantage in multilingual speech synthesis. However, on the other hand, from the perspective of cross-speaker tasks in multilingual speech synthesis, SMTR still contains a small amount of speaker information, which hinders the multilingual speech synthesis task. The goal of the SMTR-to-multilingual spectrogram prediction method is to transform SMTR features into acoustic features called Mel spectrograms, including an adversarial speaker classifier and a decoder that concatenates speaker vectors.

[0029] (4) An end-to-end multilingual speech synthesis method integrating SMTR

[0030] The end-to-end multilingual speech synthesis method first predicts SMTR features from a specified text, then uses the predicted SMTR for adversarial training to maximize the linguistic information contained in the SMTR and minimize the speaker information. Finally, the calculated speaker embedding is fused into the decoder, thus achieving speech synthesis of a target speaker speaking multiple languages ​​using multiple monolingual corpora.

[0031] The beneficial effects not only significantly improve the naturalness of multilingual speech synthesis but also greatly enhance speaker similarity. This invention proposes a self-learning multilingual text representation and introduces a speaker adversarial mechanism. The model is trained in two segments, opening up a new avenue for improving the accuracy of multilingual speech synthesis. The addition of adversarial training and pre-extracted average speaker vectors contributes to advancing existing research on the quality and speaker similarity of multilingual speech synthesis. Attached Figure Description

[0032] Figure 1 SMTR Extraction Method Based on Self-Learning System

[0033] Figure 2 Constructing a method for predicting multilingual text to SMTR

[0034] Figure 3 Constructing SMTR to a multilingual spectrogram prediction method

[0035] Figure 4 An end-to-end multilingual speech synthesis method incorporating SMTR.

[0036] Figure 5 Speech naturalness MOS rating (95% confidence interval).

[0037] Figure 6 Speaker similarity assessment. Detailed Implementation

[0038] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0039] (1) Constructing an SMTR extraction method based on a self-learning system

[0040] like Figure 1 As shown, our self-learning system utilizes an unsupervised trained fully convolutional neural network, Wav2Vec 2.0, to acquire SMTR. This extraction model is divided into three parts: a Speech Feature Encoding network, an SMTR Representations network, and a Quantization Module. The Speech Feature Encoding network outputs the raw audio to the Latent Speech Feature Representation, consisting of seven convolutional layers with kernel sizes of 10, 3, 3, 3, 3, 2, 2, and strides of 5, 2, 2, 2, 2, 2, 2. Using audio with a 16kHz sampling rate, the output feature encoding has a frame length of 20ms and a frame shift of 10ms, equivalent to generating a 512-dimensional feature vector every 20ms. The SMTR Representations network takes the continuous latent speech feature representations obtained in the previous step as input to obtain context-based deep speech features. The SMTR feature representation network does not use absolute position representation, but uses 16 layers of one-dimensional convolution with a kernel size of 128 to encode relative position. The product quantization module converts the latent speech representation output by the feature encoder into a discrete representation, replaces the original vector with a fixed codebook, and selects the corresponding entries from the codebook for parameterization---using the GumbelSoftmax operation (see formula (1)).

[0041] For multiple sets of x, x codebooks are used. The specific calculation process is to directly find the entry in the codebook corresponding to the maximum value during forward propagation, which is equivalent to a discrete operation. However, this step is not differentiable and cannot be backpropagated. To solve this problem, the gumbelsoftmax operation is used to concatenate the representations obtained from different codebooks and then perform a linear transformation to obtain the final SMTR vector representation.

[0042]

[0043] The above formula represents the calculated probability of the m-th input in the x-th codebook, where d is a (2, 320)-dimensional vector;

[0044] n = -log(-log(u)), where u is a uniform sample from 0 to 1; the parameter τ controls the distribution of the sampling structure, initialized to (2, 0.5, 0.999995), and gradually decreases as the number of unit updates (upate_num) increases. When τ is small enough, the value of gumbelsoftmax will be very close to the one-hot vector, thus achieving the essence of discrete sampling, namely, obtaining a one-hot vector.

[0045] (2) Constructing a prediction method for multilingual text to SMTR

[0046] like Figure 2 As shown, the multilingual text-to-SMTR prediction method is a sequence-to-sequence model based on neural networks, consisting of an encoder, a hybrid attention mechanism, and a decoder. It completes the mapping of different languages ​​to the same feature, namely SMTR, to assist in the next step of SMTR-to-speech synthesis.

[0047] The encoder's specific process is as follows: text is fed into the encoder in batches, and each character is mapped to a 512-dimensional vector that passes through 3 convolutional layers. Each convolutional layer contains 512 5×1 convolutional kernels, meaning each kernel spans 5 characters. The convolutional layers perform large-span context modeling on the input character sequence. After the convolutional layers, batch normalization is performed, and the ReLU nonlinear function is used for activation. The output of the last convolutional layer is fed into a bidirectional LSTM layer to generate encoded features. This LSTM contains 512 units.

[0048] The hybrid attention mechanism incorporates positional features into the alignment process. This allows the accumulated attention weights from previous decoding to function as an additional feature, enabling the model to maintain consistency as it moves forward along the input sequence and reducing potential subsequence repetition or omission during decoding. The positional features are derived using 32 one-dimensional convolutional kernels of length 31. The input sequence and positional features are then projected onto a 128-dimensional hidden layer representation to calculate the attention weights.

[0049] The decoder is an autoregressive recurrent neural network that predicts SMTR features from the encoded input sequence. In the decoder, the predicted SMTR features are first fed into an information bottleneck layer consisting of two fully connected layers with 256 hidden ReLU units per layer, concatenated with the attention context vector, and then passed to a two-layer stacked unidirectional LSTM with 1024 units. The output of the LSTM is again concatenated with the attention context vector and then passed through a linear projection to predict the target SMTR.

[0050] (3) Constructing a SMTR-to-multilingual spectrogram prediction method

[0051] like Figure 3 As shown, the goal of the SMTR-to-multilingual spectrogram prediction method is to remove speaker information while preserving the linguistic information in the SMTR, thus achieving SMTR-to-multilingual spectrogram synthesis. This part mainly consists of an adversarial speaker classifier and a decoder that concatenates speaker vectors.

[0052] We map the 272-dimensional SMTR features extracted from Wav2Vec 2.0 to 512 dimensions, pass them through a gradient inversion layer, and then feed them into a speaker classification network with two linear layers containing 256 hidden units for speaker classification training. The decoder is an autoregressive recurrent neural network. In the decoder, the true SMTR is first fed into a bottleneck layer consisting of two fully connected layers with 256 hidden ReLU units per layer, concatenated with the attention context vector, and then passed to a two-layer stacked unidirectional LSTM, each layer consisting of 1024 units. The output of the LSTM is again concatenated with the attention context vector, and then passed through a linear projection to predict the target spectral frame. Finally, the target spectral frame is passed through a 5-layer convolutional network to predict a residual that is superimposed on the spectral frame before convolution to improve the entire spectral reconstruction process. Each of the five convolutional layers consists of 512 5×1 convolutional kernels, followed by batch normalization layers. Except for the last convolutional layer, each batch normalization layer is activated by the tanh function and is performed in parallel with the prediction of the spectral frames. The output of the decoder LSTM is concatenated with the attention context vector, projected into a scalar, and then passed to the sigmoid activation function to predict the probability of whether the output sequence has been completed.

[0053] (4) An end-to-end multilingual speech synthesis method integrating SMTR

[0054] like Figure 4 As shown, the end-to-end speech synthesis method based on SMTR fusion is obtained by connecting the prediction part of multilingual text to SMTR and the spectrogram prediction part of SMTR to multilingual text.

[0055] In the inference process, firstly, we feed the multilingual text into the prediction part of the SMTR (Speech Spectrum Transformer) to obtain the predicted SMTR features. Then, we use this feature to infer the Mel spectra for each language using the trained SMTR to the multilingual spectrogram prediction part. Finally, we use the obtained spectra to perform speech synthesis using a trained Waveglow vocoder, ultimately obtaining the multilingual speech synthesis result from the same speaker. It is worth mentioning that, to reduce time complexity and ensure alignment between the output and the Mel spectra, the waveform is grouped into sets of 8 points, which yields the best synthesis results.

[0056] (5) Comparison of results between the end-to-end multilingual speech synthesis method integrating SMTR and the baseline model

[0057] For the synthesis results of this invention, we use two methods for comparison: the Mean Opinion Score (MOS) test and the speaker similarity test.

[0058] The MOS (Speech Naturalness) test evaluates the naturalness and quality of synthesized speech. Speech naturalness refers to the degree of similarity between the synthesized speech and real human voice in terms of rhythm, pauses, etc., while speech quality refers to the energy stability and noise content of the synthesized speech. The MOS scoring system is from 1 to 5 points, in increments of 0.5 points, with 1 representing the worst and 5 representing the best. Higher scores indicate better naturalness and quality of speech. Results are as follows... Figure 5 As shown.

[0059] Speaker similarity refers to the process of specifying a speaker, synthesizing speech using a baseline model and an end-to-end multilingual speech synthesis method fused with SMTR, and then having a listener judge which speech has a more similar timbre to the specified speaker. If it is difficult to judge, an option can be selected that it cannot be distinguished. The result is as follows: Figure 6 As shown.

[0060] Therefore, this invention not only significantly improves the naturalness of multilingual speech synthesis, but also greatly enhances the speaker similarity.

Claims

1. A multi-speaker, multilingual speech synthesis system based on self-learning text representation, characterized in that, Specifically, it includes the following four steps: (1) Constructing an SMTR extraction method based on a self-learning system SMTR refers to Self-Learning Multilingual Text Representation; (2) Constructing a prediction method for multilingual text to SMTR It includes an encoder and a decoding feature prediction network that incorporates an attention mechanism to predict SMTR from the input multilingual character sequence, thus achieving the prediction of cross-lingual features, namely SMTR, from text in different languages. The specific strategy of step (2) is as follows: the encoder module includes a character embedding layer, a 3-layer convolutional network, and a single-layer bidirectional long short-term memory network layer; the attention mechanism is based on the position-sensitive attention mechanism, the position features are obtained by convolution with 32 1-dimensional convolutional kernels of length 31, and then the input sequence and position features are projected onto the 128-dimensional hidden layer representation to calculate the attention weights. The decoder is an autoregressive recurrent neural network that consists of two stacked unidirectional long short-term memory networks with 1024 units each and a five-layer convolutional network with 512 5×1 convolutional kernels per layer. It predicts the output SMTR from the encoded input sequence. (3) Constructing a SMTR-to-multilingual spectrogram prediction method The SMTR features are converted into acoustic features Mel spectrum, including an adversarial-trained speaker classifier and a decoder that concatenates speaker vectors. The specific strategy of step (3) is to decouple the input SMTR from the fixed speaker, and combine the adversarial training method to introduce a gradient backpropagation layer to remove language-dependent information in the SMTR. The decoder is an autoregressive structure that consists of two unidirectional LSTM networks and a five-layer convolutional network. It predicts the Mel spectrum from the input sequence encoded by the preceding neural network, three frames at a time. (4) An end-to-end multilingual speech synthesis method integrating SMTR First, SMTR features are predicted from the specified text. Then, the predicted SMTR is used for adversarial training. The speaker vectors extracted by a pre-trained model are fused into the decoder. Finally, speech synthesis of the target speaker speaking multiple languages ​​is achieved on monolingual corpora of multiple languages.

2. The multi-speaker, multilingual speech synthesis system based on self-learning text representation according to claim 1, characterized in that, The specific strategy of step (1) is: using the cross-language method of Wav2Vec 2.0, learn the speech units common to several languages, assuming that the common speech units are built on a certain self-learning multilingual text representation feature, namely SMTR; use SMTR as a multilingual text representation and ensure that it provides sufficient linguistic information in the multilingual speech synthesis task; Specifically: The SMTR is obtained by using an unsupervised trained fully convolutional neural network Wav2Vec 2.

0. The extraction model is divided into three parts: speech feature encoding network, SMTR representation network, and product quantization module. The speech feature encoder network outputs the raw audio to the latent speech feature representation. It consists of seven convolutional layers with kernel sizes of 10, 3, 3, 3, 3, 2, 2 and strides of 5, 2, 2, 2, 2, 2. It uses audio with a sampling rate of 16 kHz, and the output feature encoding has a frame length of 20 ms and a frame shift of 10 ms, which is equivalent to generating a 512-dimensional feature vector every 20 ms. SMTR Representation Network: It takes the continuous latent speech feature representation obtained in the previous step as input to obtain context-based deep speech features; The SMTR feature representation network does not use absolute position representation, but instead uses 16 layers of one-dimensional convolutions with a kernel size of 128 to encode relative position. The product quantization module transforms the latent speech representation output by the feature encoder into a discrete representation, replaces the original vector with a fixed codebook, selects corresponding entries from the codebook for parameterization, and uses the GumbelSoftmax operation. The final SMTR vector representation is obtained by concatenating the representations obtained from different codebooks and then performing a linear transformation. GumbelSoftmax: Official (1) The above formula represents the calculated probability of the m-th input in the x-th codebook, where, It is a (2, 320) dimensional vector; It is a uniform sampling from 0 to 1; The parameter τ controls the distribution of the sampling structure. It is initialized to (2, 0.5, 0.999995) and gradually decreases as the number of cell updates increases.

3. The multi-speaker, multilingual speech synthesis system based on self-learning text representation according to claim 1, characterized in that, The specific strategy of step (4) is as follows: First, predict SMTR features from texts in multiple languages. Second, specify a specific speaker and feed the predicted features into the trained SMTR to generate speech feature Mel spectrum in the multilingual phonogram prediction module. Finally, use the trained vocoder to synthesize speech and obtain the multilingual speech synthesis result of the same speaker.

Images