Mongolian speech synthesis method based on ghost and iLPCnet

Abstract

The present invention discloses a Mongolian speech synthesis method based on ghost and iLPCnet, which aligns Mongolian phoneme information sequences based on the Bang pre-training model; generates acoustic features according to the phoneme sequence based on the acoustic model of ghost; and converts the acoustic features to speech waveforms using the iLPCnet model as a vocoder. The present invention uses an Encoder‑Decoder model to convert Mongolian text into phonemes, and then uses a ghost-based acoustic model to directly generate mel spectra from phonemes. The iLPCnet vocoder directly converts the mel spectra into speech waveforms, which can be seamlessly integrated into an end-to-end TTS system, reducing the demand for parameters and improving the speed of speech synthesis, and is suitable for speech synthesis of small languages.

Description

Technical Field

[0001] The invention belongs to the technical field of artificial intelligence, relates to speech synthesis, and particularly relates to a Mongolian speech synthesis method based on ghost and iLPCnet. Background Art

[0002] Speech synthesis technology refers to the technology that converts any text into fluent speech through computer analysis. Speech synthesis research at home and abroad has gone through the development process of speech synthesis based on formant synthesis, waveform splicing, speech synthesis method based on statistical parameters, and speech synthesis method based on deep neural network.

[0003] Existing research on speech synthesis is mostly focused on mainstream languages, while speech synthesis for obscure languages ​​such as Mongolian is still in its early stages due to problems such as scarcity of corpus. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the object of the present invention is to provide a Mongolian speech synthesis method based on ghost and iLPCnet to improve the speed and quality of Mongolian speech synthesis.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A Mongolian speech synthesis method based on ghost and iLPCnet, characterized in that it comprises the following steps:

[0007] Step 1, extracting Mongolian text features and converting the text feature encoding into a phoneme sequence, and inputting the phoneme sequence into the Bang pre-training model;

[0008] Step 2: Use the Bang pre-trained model to train Mongolian phoneme information and align the phoneme information;

[0009] Step 3: Generate acoustic features through phoneme information based on the deepvoice3 acoustic model with a lightweight ghost module;

[0010] Step 4: Using the iLPCnet model that incorporates Mongolian speech features as a vocoder, the acoustic features are converted into speech waveforms, thereby achieving Mongolian speech synthesis.

[0011] In one embodiment, in step 1, the encoder is a ghost module encoder.

[0012] In one embodiment, the Bang pre-trained model consists of a multi-layer stacked Transformer encoder and a multi-layer stacked Transformer decoder; the Transformer encoder uses a self-attention mechanism, and the Transformer decoder uses a cross-stream visible multi-stream self-attention mechanism;

[0013] The cross-stream visible multi-stream self-attention mechanism, in the first prediction stream, each input Mongolian phoneme information is dependent, and each phoneme is predicted by autoregression; in the remaining prediction streams, the first phoneme is predicted by non-autoregression, and the other phonemes are predicted in a way between autoregression and non-autoregression; the target sequence length |Y|=n, then the Bang pre-trained model sets n prediction streams, at this time, the first i information of each phoneme (i refers to any length) is replaced by a mask, and can be predicted in parallel in the same time step.

[0014] In one embodiment, the deepvoice3 acoustic model uses a fully convolutional encoder and decoder and is composed in a serial manner, that is, acoustic features are generated in an autoregressive manner.

[0015] In one embodiment, in step 3, in both the encoder and decoder of the deepvoice3 acoustic model, a multi-layer stacked ghost module is used to replace the full convolution module, and the ghost module includes a convolution part and a linear transformation part. The convolution part uses one-dimensional convolution to reduce the number of output channels, thereby reducing the number of parameters and the amount of calculation; the linear transformation part uses multiple linear mappings to convert the few-channel features into multi-channel features, that is, by dynamically adjusting the number of channels, the output is ensured to be consistent with the acoustic feature dimension.

[0016] In one embodiment, the vocoder is composed of an upsampling network and a speech generation network. The upsampling network matches the time resolution of the input Mongolian acoustic features with the sampling rate of the speech signal in the original Mongolian speech library. The speech generation network is used to automatically regress and generate Mongolian speech based on the acoustic features.

[0017] In one embodiment, the upsampling network uses two 1*3 CNN convolutional layers to extract the local context of acoustic features, construct a context vector, and connect the input acoustic features to make the context vector more dominant over the current frame information (the ongoing conversion from acoustic features to acoustic features of speech waveform); the speech generation network uses a fully connected layer to map the dimension of the context vector constructed by the upsampling network to the input dimension of the context, and then filters the noise through a filter. Finally, the output dimension of the context is determined by the result in the upsampling network (speech waveform spectrogram), and Mongolian speech is output.

[0018] In one embodiment, the acoustic feature is a mel spectrum.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] First, a Bang-based pre-trained model is used to align Mongolian phoneme information. Then, an acoustic model based on the ghost module is used to directly generate mel spectra from phonemes. Finally, the iLPCnet vocoder directly converts the mel spectra into speech waveforms, greatly improving the speed and naturalness of Mongolian speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the process of the present invention.

[0022] Figure 2 It is the cross-stream visible multi-stream attention map in the Bang pre-trained model.

[0023] Figure 3 This is the decoding process of the Encoder-Decoder model of the present invention.

[0024] Figure 4 It is a schematic diagram of the ghost model of the present invention.

[0025] Figure 5 It is a schematic diagram of the iLPCnet model of the present invention. DETAILED DESCRIPTION

[0026] The embodiments of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0027] The Mongolian speech synthesis method based on ghost and iLPCnet mainly includes three steps: front-end processing, predicting mel spectrum model and mel spectrum conversion speech. Figure 1 As shown, the specific steps are as follows:

[0028] Step 1: Extract Mongolian text features, convert the text feature encoding into a phoneme sequence, and input the phoneme sequence into the Bang pre-trained model.

[0029] For example, in the present invention, the Mongolian text form has a variety of characteristics, which can be extracted by using an encoder, especially a ghost module encoder. The phoneme sequence is composed of phonemes. For example, the Mongolian " "Extract text features, expressed as: "cqhihv"

[0030] The encoding is converted into phonemes, expressed as: "q oe h ws h"

[0031] Step 2: Use the Bang pre-trained model to train Mongolian phoneme information and align the phoneme information.

[0032] The mechanism of the Bang pre-training model can provide phoneme alignment information for the subsequent steps of the present invention using a non-autoregressive method, and ultimately achieve parallel output of acoustic features.

[0033] The Bang pre-training model consists of a multi-layer stack of Transformer encoders using a self-attention mechanism and a multi-layer stack of Transformer decoders using a cross-stream visible multi-stream self-attention mechanism. In order to optimize the non-autoregressive Mongolian phoneme sequence and efficiently parallelize the calculation, the Transformer decoder of the present invention uses a cross-stream visible multi-stream self-attention mechanism.

[0034] Under this mechanism, in the first prediction stream, each input Mongolian phoneme information is dependent (i.e. known), and each phoneme is predicted or calculated by autoregression; in the remaining prediction streams, the first phoneme is predicted by non-autoregression, and the other phonemes are predicted in a way between autoregression and non-autoregression. Assuming that the target sequence length |Y|=n, the Bang pre-trained model sets n prediction streams. At this time, the first i information of each phoneme (i refers to any length) is replaced by [MASK] (mask), and they are predicted in parallel in the same time step. This pre-training method alleviates the problem that each phoneme in the non-autoregressive Mongolian phoneme sequence has no dependency. In order to optimize the memory usage and calculation amount, the Bang pre-trained model adopts a block calculation scheme. Because each position only sees the prediction stream information before it, the Bang pre-trained model calculates from the first prediction stream to the last prediction stream.

[0035] Specifically, Figure 2 As shown, the training process of the Bang pre-training model of the present invention for phonemes is as follows:

[0036] The first line predicts the Mongolian phoneme y 4 : shows the main stream (the sentence where each input phoneme has dependencies) and the first predicted stream (the sentence where there are no dependencies between the input phonemes). The phoneme y in the main stream is masked using [M] (i.e., mask). 1 ,y 2 ,y 3 Cover to calculate attention, that is, phoneme y 4 The phoneme y 1 ,y 2 ,y 3 All the phoneme information in the first prediction stream is predicted by autoregression with complete previous context information.

[0037] The second line predicts the phoneme y4 :The previous information of each predicted phoneme is covered by one character, phoneme y 4 Yes 1 ,y 2 To support, but y 3 is covered by [M] (i.e., mask). The first predicted stream phoneme y 3 [M] and the phoneme y in the second prediction stream 4 Then the conditional probability P(y 3 ,y 4 |y 1 ,y 2 ). That is, as the attention flow increases, the previous context information is covered, and the generation method also moves from autoregressive to non-autoregressive.

[0038] The last row shows the phoneme y 4 In the fourth prediction stream, the prediction is finally performed in a non-autoregressive manner. At this time, the predicted phoneme y in the fourth prediction stream 4 The [M] (i.e., mask) is composed of the first predicted phone y 1 [M], the phoneme y in the second prediction stream 2 [M] and the phoneme y in the third prediction stream 3 In this way, the gap between autoregressive and non-autoregressive is bridged.

[0039] Step 3, based on the deepvoice3 acoustic model with a lightweight ghost module, such as Figure 3 As shown in the figure, acoustic features are generated through phoneme information. Compared with traditional TTS, the rhythm and expressiveness of the generated speech will be better.

[0040] End-to-end speech synthesis first needs to generate acoustic features by the acoustic model, such as mel spectrum, linear spectrum, etc. The present invention uses mel spectrum, and mel frequency spectrogram is related to linear frequency spectrogram (amplitude of short-time Fourier transform). Inspired by the response test of the human auditory system, the mel spectrum is obtained by applying a nonlinear transformation to the frequency axis of the short-time Fourier transform, and compressing the frequency range with fewer dimensions. It is smoother than the waveform sample, and because each frame is phase-invariant, it is easier to use such a representation to train with mean square error (MSE) loss. Compared with the linguistic and acoustic features used in traditional vocoders, the mel spectrogram is a lower-level acoustic representation of the audio signal, so it is more direct to train speech synthesis on the mel spectrogram. The present invention is trained on a pre-normalized phoneme sequence, and the boundaries of words and syllables, and punctuation marks are also included as special marks. After obtaining the aligned phoneme sequence, parallel training is performed, and there is no recursive connection between the frames, which can improve the performance of Mongolian speech synthesis, speed up the training speed, and make the audio process smoother.

[0041] The deepvoice3 acoustic model includes an encoder and a decoder. The traditional deepvoice3 acoustic model uses a fully convolutional encoder and decoder and is composed in a serial manner, that is, the mel spectrum is generated in an autoregressive manner. Its disadvantages are a large number of parameters and a slow speed in synthesizing speech.

[0042] like Figure 3 As shown, in the encoder and decoder of the existing deepvoice3 acoustic model, the present invention uses a multi-layer stacked ghost module to replace the full convolution module. The ghost module divides the ordinary convolution layer into two parts, such as Figure 4 As shown in the figure, there are convolution part and linear transformation part. The convolution part uses one-dimensional convolution to reduce the number of output channels, thereby reducing the number of parameters and calculations. The linear transformation part uses multiple linear mappings to transform the low-channel features into multi-channel features, that is, by dynamically adjusting the number of channels, the output is guaranteed to be consistent with the mel spectrum dimension.

[0043] In the acoustic model obtained by the present invention, the encoder and the decoder are performed in a feedforward manner, and the encoder and the decoder are connected through phoneme alignment information.

[0044] The model inputs text, and after encoding by the encoder containing the ghost module, it provides text representation. Through Bang pre-training, it obtains the aligned phoneme information to achieve hard alignment of the encoding step and the decoding step. After decoding by the decoder containing the ghost module, the output mel spectrum map is predicted in parallel.

[0045] Step 4: Using the iLPCnet model that incorporates Mongolian speech features as a vocoder, the acoustic features are converted into speech waveforms, thereby achieving Mongolian speech synthesis.

[0046] The acoustic features are input into the vocoder as conditions to construct a neural network vocoder based on iLPCnet.

[0047] This vocoder is a vocoder that cleverly combines digital signal processing (DSP) and neural network (NN) for Mongolian speech synthesis. This vocoder can synthesize high-quality speech in real time on an ordinary CPU. Figure 5 As shown in the figure, it consists of two sub-networks: an upsampling network, which matches the temporal resolution of the Mongolian acoustic features in the input acoustic model with the sampling rate of the speech signal in the original Mongolian speech library; and a waveform generation network, which automatically regresses and generates Mongolian speech. In the upsampling network, two 1*3 CNN convolutional layers are used to extract the local context of the acoustic features, construct the context vector, and connect the input acoustic features. The purpose is to make the context vector more dominant over the current frame information (the ongoing conversion from acoustic features to acoustic features of the speech waveform). In the speech generation network, a fully connected layer is used to map the dimension of the context vector (constructed by the upsampling network) to the input dimension of the context, and then filter the noise through a filter. Finally, the output dimension of the context is determined by the result in the upsampling network (speech waveform spectrogram), and the Mongolian speech is output.

Claims

1. A Mongolian speech synthesis method based on ghost and iLPCnet, characterized in that: The steps include: Step 1, extracting Mongolian text features, and converting the text feature encoding into a phoneme sequence, and inputting the phoneme sequence into a Bang pre-training model; the Bang pre-training model consists of a multi-layer stacked Transformer encoder and a multi-layer stacked Transformer decoder; the Transformer encoder uses a self-attention mechanism, and the Transformer decoder uses a cross-stream visible multi-stream self-attention mechanism; The cross-stream multi-stream self-attention mechanism is visible. In the first prediction stream, each input Mongolian phoneme information is dependent, and each phoneme is predicted by autoregression; in the remaining prediction streams, the first phoneme is predicted by non-autoregression, and the other phonemes are predicted in a way between autoregression and non-autoregression; the target sequence length |Y|=n, then the Bang pre-training model sets n prediction streams, at this time, the first i information of each phoneme is replaced by a mask, and can be predicted in parallel in the same time step; Step 2: Use the Bang pre-trained model to train Mongolian phoneme information and align the phoneme information; Step 3: Generate acoustic features through phoneme information based on the deepvoice3 acoustic model with a lightweight ghost module; Step 4: Using the iLPCnet model that incorporates Mongolian speech features as a vocoder, the acoustic features are converted into speech waveforms, thereby achieving Mongolian speech synthesis.

2. The Mongolian speech synthesis method based on ghost and iLPCnet according to claim 1, characterized in that: In the step 1, the encoder is a ghost module encoder.

3. The Mongolian speech synthesis method based on ghost and iLPCnet according to claim 1, characterized in that: The deepvoice3 acoustic model uses a fully convolutional encoder and decoder and is composed in a serial manner, that is, acoustic features are generated in an autoregressive manner.

4. The Mongolian speech synthesis method based on ghost and iLPCnet according to claim 3, characterized in that: In step 3, in both the encoder and decoder of the deepvoice3 acoustic model, a multi-layer stacked ghost module is used to replace the full convolution module. The ghost module includes a convolution part and a linear transformation part. The convolution part uses one-dimensional convolution to reduce the number of output channels, thereby reducing the number of parameters and the amount of calculation. The linear transformation part uses multiple linear mappings to convert the few-channel features into multi-channel features, that is, by dynamically adjusting the number of channels, the output is guaranteed to be consistent with the acoustic feature dimension.

5. The Mongolian speech synthesis method based on ghost and iLPCnet according to claim 1, characterized in that: The vocoder consists of an upsampling network and a speech generation network. The upsampling network matches the time resolution of the input Mongolian acoustic features with the sampling rate of the speech signal in the original Mongolian speech library. The speech generation network is used to automatically regress and generate Mongolian speech according to the acoustic features.

6. The Mongolian speech synthesis method based on ghost and iLPCnet according to claim 5, characterized in that: The upsampling network uses two 1*3 CNN convolutional layers to extract the local context of acoustic features, construct a context vector, and connect the input acoustic features to make the context vector more dominant over the current frame information; the speech generation network uses a fully connected layer to map the dimension of the context vector constructed by the upsampling network to the input dimension of the context, and then filters the noise through a filter. Finally, the output dimension of the context is determined by the result in the upsampling network, and Mongolian speech is output.

7. The Mongolian speech synthesis method based on ghost and iLPCnet according to any one of claims 1 to 6, characterized in that: The acoustic feature is a mel spectrum.

Images