A cross-language speech synthesis method based on end-to-end timbre and emotion transfer

Abstract

The present invention discloses a cross-language speech synthesis method based on end-to-end timbre and emotion transfer, the steps of which are as follows: S1, collecting and processing Chinese and English speech training data, extracting the required speech features; S2, training the learning network architecture for speech synthesis of Chinese and English, including three parts: speaker encoder, synthesizer and vocoder; S3, using the trained learning network architecture, performing cross-language speech synthesis on the real-time speech input by the speaker, and the synthesized speech can effectively retain the speaker's timbre and emotion. The present method provides a cross-language speech synthesis method, which can synthesize cross-language speech under the condition of a small amount of speech of a given speaker, and can retain the speaker's timbre and emotion in the synthesized speech.

Description

Technical Field

[0001] The present invention relates to the field of computer speech synthesis, and in particular to a cross-language speech synthesis method based on end-to-end timbre and emotion transfer. Background Art

[0002] Machine translation refers to the process of translating a source language sentence into a semantically equivalent target language sentence through a computer. It is an important research direction in the field of natural language processing. With the continuous maturity of artificial intelligence technology, machine translation has made great progress, and its practicality and extensiveness have gradually emerged, providing great convenience for different language users to learn other languages ​​and communicate with each other.

[0003] Machine translation includes three steps: speech recognition technology, machine translation, and speech synthesis. Currently, there are several relatively mature machine translation service providers (such as Google Translate, DeepL, Baidu Translate, Youdao Cloud Translation, etc.), which can achieve translation and conversion between multiple languages, and provide services including text-to-text, text-to-speech, and speech-to-text. However, text cannot cover the timbre, pitch, speaking speed, speaker emotions and other information reflected in the acoustic characteristics of speech, and the translation content obtained by machine translation may have some errors. In this way, there will be information missing in the information conveyed by machine translation, which will bring certain obstacles to communication.

[0004] Speech synthesis refers to the generation of human voices through text, or the process of generating the corresponding human pronunciation given a piece of text. The sound here is a continuous analog signal, and the synthesis process that the computer needs to do is to simulate it through digital signals. In the process of speech synthesis, pronunciation, alignment, rhythm, intonation and other issues are all key to synthesizing speech, and the problems of tone change, polyphones, and complex rhythm in Chinese speech are difficult to deal with. Therefore, there are many difficulties in Chinese speech synthesis at present. Embedding the speaker's characteristics, namely the timbre and the emotional characteristics of the speaker's voice, in the synthesized speech can make the synthesized speech more natural and closer to the speaker's voice characteristics. Summary of the invention

[0005] The purpose of the present invention is to integrate and develop existing speech synthesis technologies, to build a learning network architecture for cross-language speech synthesis based on end-to-end timbre and emotion transfer, which can synthesize cross-language speech with the speaker's timbre and emotion under the condition that the speaker is given a small amount of language, and to provide a speech synthesis method for cross-language timbre and emotion transfer.

[0006] The purpose of the present invention can be achieved by adopting the following technical solutions:

[0007] A cross-language speech synthesis method based on end-to-end timbre and emotion transfer, the speech synthesis method comprising the following steps:

[0008] S1. Collect training data: Collect short multi-sentence recording files of multiple speakers, and create one-to-one corresponding text tags for each short multi-sentence recording file. The short multi-sentence recording file should not exceed 15 seconds, and the total duration should not be less than 30 hours. The recording environment should be a quiet indoor environment.

[0009] S2, preprocessing: perform speech processing on the short recording files of multiple sentences, convert them into the required audio files, and generate one or more files in JSON format;

[0010] S3. Construct a learning network architecture for speech synthesis in Chinese and English, the learning network architecture including a speaker encoder, a synthesizer and a vocoder;

[0011] S4, training speaker encoder: the speaker encoder takes the Mel spectrum of the audio file in step S2 as input and outputs an embedding vector of fixed dimension, which can preserve the speaker's timbre and emotion;

[0012] S5. Training synthesizer: The synthesizer consists of an encoder and a decoder connected in sequence, wherein the encoder consists of a preprocessing network, three one-dimensional convolutional layers and a bidirectional LSTM layer, the input is the JSON file generated in step S2, and the output is the encoder hidden state. The decoder consists of a preprocessing network, a two-layer LSTM network connected in sequence, a projection layer and a post-processing network. The hidden state output by the encoder is decoded to generate a Mel spectrum of the synthesized speech. First, the encoder parses the JSON file generated by the preprocessing to generate the encoder hidden state, and then the encoder hidden state is input into the decoder, and the decoder outputs the Mel spectrum of the synthesized speech.

[0013] S6. Train the vocoder: The vocoder consists of a parallel WaveRNN vocoder and a HiFi-GAN vocoder. It takes the Mel spectrum generated by the decoder as input and outputs a waveform for predicting speech synthesis.

[0014] S7. After preprocessing the real-time speech, input it into the trained learning network architecture to obtain cross-language synthesized speech.

[0015] Furthermore, the process of step S2 is as follows:

[0016] S2.1. Perform speech processing on short multi-sentence recording files. Short multi-sentence recording files will be converted into audio files with an audio sampling rate of 16000Hz, audio format wav format, bit depth of 16bits, and monophonic. The converted audio files can help the learning network architecture to better extract the speaker's features and some related information in the audio, and achieve better speech synthesis effects;

[0017] S2.2. Concatenate the text tag, speaker, speaker ID, and audio file tag obtained by speech processing into one or more JSON format files, where the text tag refers to the speaker and the speech content of the audio file corresponding to the speech content, the speaker ID refers to the number tag for the speaker, and the audio file tag refers to the name of the audio file corresponding to the speaker and the speech content. The generated JSON file provides the learning network architecture with the data required for training, namely, the speech and the text content of the speech, and at the same time, allows the learning network architecture to correspond speech information to speaker information one by one.

[0018] Furthermore, the process of training the speaker encoder in step S4 is as follows:

[0019] S4.1. For a given short recording of multiple sentences, use the following formula to calculate the Mel spectrum:

[0020]

[0021] Among them, f is the frequency of multiple short speech sentences, and the output is the Mel spectrum of multiple short speech sentences;

[0022] S4.2, the speaker encoder takes the Mel spectrum of multiple short speech as input and outputs an embedding vector of fixed dimension. The training process is as follows:

[0023] S4.2.1. Input the long Mel spectrum of indefinite length into three sequentially connected LSTM networks, each layer consists of 768 cells. The result of each frame output by the last layer of LSTM network is mapped to a 256-dimensional fixed-length vector, where a frame refers to a fixed time unit. In the LSTM network, the speaker encoder can effectively extract the speaker's timbre and emotional characteristics from the speech content. Moreover, by mapping the output of the last layer of LSTM network into a vector, the speaker's timbre and emotional characteristics can be passed to the synthesizer.

[0024] S4.2.2. All outputs obtained in the above steps are averaged and normalized to obtain the final fixed-dimensional embedding vector, where the output embedding vector distinguishes the speaker corresponding to the speech from other speakers and retains the speaker's timbre and emotion. The average and normalization processing can speed up the training of the speaker encoder and achieve better training results.

[0025] Furthermore, the process of training the encoder in the synthesizer in step S5 is as follows:

[0026] S5.1.1. Get the input sequence required to generate word vectors:

[0027] S5.1.1.1. Parse the JSON file generated in step S2, and translate the text tags in the JSON file into phoneme sequences, where English text corresponds to English phonetic symbols and Chinese text corresponds to Chinese pinyin. This operation helps the encoder to better convert and retain the key information in the speech into word vectors, thereby improving the encoder's ability to extract information from the speech content;

[0028] S5.1.1.2. Combine the phoneme sequences corresponding to the text tags into a JSON file and use the resulting JSON file as the input sequence. Using the entire JSON file as input helps simplify operations and unify setting management;

[0029] S5.1.2, Generate word vector: Input the input sequence obtained in step S5.1.1 into the preprocessing network for analysis and transformation to obtain the preprocessed input sequence. The preprocessing operation can further convert the input sequence into a format that allows the encoder to better extract speech information, which is conducive to the preprocessing network to adjust parameters by itself and extract information more effectively; Perform word embedding operation on the preprocessed input sequence, calculate the weight of each phoneme in the phoneme sequence in the JSON file corresponding to the remaining phonemes, and output a 512-dimensional word vector, which contains the position information and content information of the speech;

[0030] S5.1.3, obtain the intermediate state: input the word vector obtained in step S5.1.2 into three one-dimensional convolutional layers connected in sequence, each convolutional layer includes 512 convolution kernels of size 5*1, and perform batchNorm and dropout operations on the output after each convolutional layer. The batchNorm operation is to keep the input of each layer of the neural network in the same distribution during the deep neural network training process, and the dropout operation is to temporarily mask the neural network unit with random probability. The intermediate state is obtained at the last convolutional layer. These two operations help the word vector to better retain the relevant information of the speech, so that the subsequent structure can better extract the required speech information;

[0031] S5.1.4, Obtain encoder hidden state: Input the intermediate state obtained in step S5.1.3 into a bidirectional long short-term memory network LSTM layer to generate the encoder hidden state. The bidirectional LSTM layer can better extract speech information;

[0032] S5.1.5. The embedding vectors extracted by the speaker encoder from multiple short speech sentences are stitched into the corresponding encoder hidden states. The encoder hidden states now contain the speech information and the speaker's timbre and emotional information.

[0033] Furthermore, the process of training the decoder in the synthesizer in step S5 is as follows:

[0034] S5.2.1. The decoder runs in a loop, each loop is called a time step, and the attention mechanism is operated in each decoder time step. The attention mechanism is a matrix composed of context weight vectors, which can score each dimension of the input, and then weight the features according to the scores to highlight the impact of important features on downstream models or modules. The attention mechanism processes the encoder hidden state, measures the similarity of the units in the hidden state, and then normalizes and obtains the context vector, which contains the information in the encoder hidden state;

[0035] S5.2.2, the preprocessing network in the decoder receives the context vector output in the previous time step, analyzes and processes the context vector, and then sends the context vector to the two layers of long short-term memory network LSTM layers connected in sequence in the decoder for processing and transformation to obtain a new context vector, and finally sends the new context vector to the projection layer to output the spectrogram frame and the end probability, where the spectrogram frame contains the predicted Mel spectrum. The transmission processing of the context vector enables the decoder to better extract speech information;

[0036] S5.2.3. Send the spectrogram frame obtained in step S5.2.2 to the post-processing network of the decoder. The post-processing network consists of three convolutional layers connected in sequence. The post-processing network is used to improve the quality of the generated Mel spectrum and finally output the Mel spectrum of the synthesized speech. This step can provide the input Mel spectrum for the subsequent vocoder to generate speech.

[0037] Furthermore, the WaveRNN vocoder is composed of a single-layer RNN network and a double softmax layer connected in sequence. The WaveRNN vocoder is based on WaveNet. The Mel spectrum of the synthesized speech in step S5 is input into the WaveRNN vocoder. The single-layer RNN network processes and analyzes the Mel spectrum of the synthesized speech, and the output is divided into two parts. Each part is input into the corresponding softmax layer respectively, and the output is a predicted 16-bit audio sample. The synthesis process can better balance the speed and quality of the synthesized speech.

[0038] Furthermore, the HiFi-GAN vocoder is composed of a generator and a discriminator connected in sequence, wherein the number of generators is 1 and the number of discriminators is 2, and the two discriminators receive the output of the generator in parallel. The input of the HiFi-GAN vocoder is the GAN predicted speech waveform, and the generator is a convolutional neural network. The input is the Mel spectrum, and the sampling rate is increased by convolution until the target sampling rate is reached to generate a predicted 16-bit audio sample; the two discriminators are a scale discriminator and a multi-cycle discriminator, which evaluate whether the waveform generated by the generator meets the standard, and optimize the training alternately to improve the inference speed while ensuring the synthesized sound quality.

[0039] Furthermore, the process of step S7 is as follows:

[0040] S7.1. Collect the speech that needs to be synthesized across languages, perform preprocessing, generate JSON format files and audio files, that is, convert them into an input format that can be received by the learning network architecture;

[0041] S7.2. Input the JSON format file into the encoder of the synthesizer to generate the encoder hidden state. Meanwhile, input the Mel spectrum of the audio file into the speaker encoder to generate an embedding vector, and concatenate the embedding vector into the encoder hidden state. The encoder hidden state can well contain the speech information and speaker information.

[0042] S7.3, input the encoder hidden state generated in step S7.2 into the decoder of the synthesizer to generate the Mel spectrum of the synthesized speech, which provides input for the subsequent vocoder;

[0043] S7.4. Input the Mel spectrum of the synthesized speech into a vocoder to obtain synthesized cross-language speech, and the cross-language speech has the timbre and emotion of the speaker.

[0044] Compared with the prior art, the present invention has the following advantages and effects:

[0045] The present invention collects a small amount of user's speech, obtains speech data after preprocessing the speech, and puts the speech data into a trained learning network architecture, extracts an embedding vector carrying the speaker's timbre and emotional characteristics through a speaker encoder in the learning network architecture, and then synthesizes the Mel spectrum of the target speech through a synthesizer in the learning network architecture, and finally inputs the Mel spectrum of the target speech into a vocoder to synthesize the target speech. The synthesis method proposed by the present invention can quickly synthesize cross-language speech, and the synthesized speech has the speaker's timbre and emotional characteristics, which can make the synthesized speech appear more real and close to the speaker. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0047] Figure 1 is a processing flow chart of the learning network architecture for speech synthesis disclosed in the present invention;

[0048] Figure 2 is a network structure diagram of the encoder in the present invention;

[0049] Figure 3 is a network structure diagram of the decoder in the present invention;

[0050] Figure 4 It is a processing flow chart of the learning network architecture of the present invention that removes the speaker encoder structure. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the method in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Example 1

[0053] Figure 1 is the processing flow of the learning network architecture of cross-language speech synthesis based on end-to-end timbre and emotion transfer disclosed in this embodiment, and Figure 2 is the encoder structure in this embodiment, Figure 3 This is the decoder structure in this embodiment. The user can input speech and synthesize the target speech according to his needs. In this embodiment, taking the computer end as an example, the processing flow of the speech synthesis method is specifically introduced, including the following steps:

[0054] Step 101, collect a small amount of user's voice, the recording time shall not exceed 15s, and the recording environment shall be relatively quiet.

[0055] Step 102, pre-processing the speech collected in step 101.

[0056] The preprocessing in step 102 specifically includes the following steps:

[0057] 1) Perform speech processing on short multi-sentence recording files. Short multi-sentence recording files will be converted into audio files with an audio sampling rate of 16000Hz, audio format WAV format, bit depth of 16 bits, and monophonic;

[0058] 2) The text tag, speaker, speaker ID, and audio file tag obtained by speech processing are spliced ​​into one or more JSON format files, where the text tag refers to the speaker and the speech content of the audio file corresponding to the speech content, the speaker ID refers to the number tag for the speaker, and the audio file tag refers to the name of the audio file corresponding to the speaker and the speech content.

[0059] Step 103: input the user's speech into a speaker encoder to extract a speaker embedding vector.

[0060] Extracting the speaker in step 103 specifically includes the following steps:

[0061] 1) For a given short recording of multiple sentences, use the following formula to calculate the Mel spectrum:

[0062]

[0063] Among them, f is the frequency of multiple short speech sentences, and the output is the Mel spectrum of multiple short speech sentences;

[0064] 2) The speaker encoder takes the Mel spectrum of multiple short speech as input and outputs an embedding vector of fixed dimension. The training process is as follows:

[0065] 2.1) Input the long Mel spectrum of indefinite length into three sequentially connected LSTM networks, each layer consists of 768 cells. The output of each frame of the last LSTM network is mapped to a 256-dimensional fixed-length vector, where a frame refers to a fixed time unit.

[0066] 2.2) All outputs obtained in the above steps are averaged and normalized to obtain the final fixed-dimensional embedding vector, where the output embedding vector distinguishes the speaker corresponding to the speech from other speakers and retains the speaker's timbre and emotion.

[0067] Step 104: input the JSON file obtained by preprocessing in step 102 into a synthesizer to obtain a Mel spectrum of speech synthesis.

[0068] The steps of step 104 to obtain the synthesized Mel spectrum are as follows:

[0069] 1) Process the JSON file to obtain the input sequence:

[0070] 1.1) Parse the JSON file generated in step S2, and translate the text tags in the JSON file into phoneme sequences, where English text corresponds to English phonetic symbols and Chinese text corresponds to Chinese pinyin;

[0071] 1.2) Combine the phoneme sequences corresponding to the text tags into a JSON file and use the resulting JSON file as the input sequence;

[0072] 2) Get word vector:

[0073] 2.1) Input the input sequence obtained in step 1) into the preprocessing network for analysis and transformation to obtain a preprocessed input sequence; perform a word embedding operation on the preprocessed input sequence, calculate the weight of each phoneme in the phoneme sequence in the JSON file corresponding to the remaining phonemes, and output a 512-dimensional word vector;

[0074] 3) Obtaining the intermediate state: Input the word vector obtained in step 2) into three sequentially connected one-dimensional convolutional layers. Each convolutional layer includes 512 5*1 convolution kernels. After each convolutional layer, the output is subjected to batchNorm and dropout operations. The batchNorm operation is to keep the input of each layer of the neural network in the same distribution during the deep neural network training process. The dropout operation is to temporarily mask the neural network unit with random probability. The intermediate state is obtained at the last convolutional layer.

[0075] 4) Obtaining the encoder hidden state: Input the intermediate state obtained in step 3) into a bidirectional long short-term memory network LSTM layer to generate the encoder hidden state;

[0076] 5) The embedding vectors extracted by the speaker encoder in multiple short sentences are stitched into the corresponding encoder hidden states.

[0077] 6) Input the encoder hidden state into the decoder, and the decoder runs in a loop, where each loop is called a time step. The attention mechanism is operated in each decoder time step. The attention mechanism is a matrix composed of context weight vectors. The attention mechanism processes the encoder hidden state, measures the similarity of the units in the hidden state, and then normalizes and obtains the context vector;

[0078] 7) The preprocessing network in the decoder receives the context vector output in the previous time step, analyzes and processes the context vector, and then sends the context vector to the two layers of long short-term memory network LSTM layers connected in sequence in the decoder for processing and transformation to obtain a new context vector, and finally sends the new context vector to the projection layer to output the spectrogram frame and the end probability, wherein the spectrogram frame contains the predicted Mel spectrum;

[0079] 8) The spectrogram frame obtained in step 7) is sent to the post-processing network of the decoder. The post-processing network consists of three convolutional layers connected in sequence. The post-processing network is used to improve the quality of the generated Mel spectrum and finally output the Mel spectrum of the synthesized speech.

[0080] Step 105: input the Mel spectrum generated in step 104 into a vocoder to obtain the required speech.

[0081] Step 105: The steps for obtaining the required voice are as follows

[0082] In the network architecture of speech synthesis, WaveRNN vocoder and HiFi-GAN vocoder are used, both of which receive Mel spectrum as input to predict the waveform of speech synthesis. The Mel spectrum generated in step 104 is input into the two vocoders to output speech with faster synthesis speed, and the synthesized speech has the speaker's timbre and emotional characteristics.

[0083] In summary, the present invention collects user voice, and after the user voice is preprocessed, a JSON file and Mel spectrum related to the user voice can be obtained. The Mel spectrum of the collected voice is input into the speaker encoder to obtain the embedding vector of the speaker's timbre and emotional characteristics. Then the JSON file is parsed, converted into a phoneme sequence, input into the encoder to extract the encoder hidden state, and the embedding vector generated by the speaker encoder is spliced ​​into the encoder hidden state, and the encoder hidden state is input into the decoder to generate the Mel spectrum of the required voice. Finally, the generated Mel spectrum is input into the vocoder to synthesize the voice required by the user. Finally, the synthesized voice of the present invention has the speaker's timbre and emotional characteristics, and can perform cross-language voice synthesis, which is not provided by existing text-to-voice services.

[0084] Example 2

[0085] This embodiment removes the speaker encoder part in the learning network architecture, such as Figure 4 As shown, the speaker encoder part of the present invention is effective in retaining the speaker's timbre and emotional information. In this embodiment, taking the computer as an example, the processing flow of the speech synthesis method is specifically introduced, including the following steps:

[0086] Step 101 and step 102 are consistent with those in Example 1, and reference is made to step 101 and step 102 in Example 1.

[0087] Step 103: input the JSON file obtained by preprocessing in step 102 into a synthesizer to obtain a Mel spectrum of speech synthesis.

[0088] Step 103 of obtaining the synthesized Mel spectrum is as follows:

[0089] The step of obtaining the encoder hidden state refers to 1) to 4) of step 104 in Example 1. Since there is no embedding vector generated by the speaker encoder, the generated encoder hidden state has no splicing operation and will be directly input into the decoder. The step of generating the synthetic Mel spectrum refers to 6) to 8) of step 104 in Example 1.

[0090] Step 104 synthesizes the required speech with reference to step 105 in Example 1. Since the speaker encoder structure is removed, the learning network architecture loses the ability to extract the speaker's timbre and emotional characteristics in the speech. The synthesizer and vocoder structure can only complete the task of cross-language speech synthesis. The speech synthesized in this embodiment obviously does not match the speaker's timbre, lacks emotional characteristics, and is relatively stiff.

[0091] For further verification, this embodiment uses the original learning network architecture and the learning network architecture without the speaker encoder structure to synthesize two groups of speech with the same content, respectively, and puts them on an online webpage together with the real speech for a questionnaire survey. The survey is in the form of a blind test, that is, the evaluators score the given speech without knowing the source of the speech. For the scoring, this embodiment uses the MOS score, which is the most authoritative international standard for judging speech quality, with a full score of 5 points. The final score also shows that the score synthesized by the learning network architecture without the speaker encoder structure is lower, as shown in Table 1 below. In addition, some users reported that some speech in the questionnaire survey was very stiff, and this part of the speech was synthesized by the learning network architecture without the speaker encoder structure.

[0092] Table 1. MOS scores in various scenarios

[0093] Voice Source MOS score Ground Truth 4.71±0.06 Original learning network architecture 4.59±0.05 Learning network architecture with speaker-removing encoder 4.46±0.05

[0094] It can be concluded from this embodiment that the learning network architecture of the present invention has the ability to transfer the speaker's timbre and emotional characteristics, and can better improve the quality of synthesized speech.

[0095] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A cross-language speech synthesis method based on end-to-end timbre and emotion transfer, characterized in that: The speech synthesis method comprises the following steps: S1. Collect training data: Collect short multi-sentence recording files of multiple speakers, and create one-to-one corresponding text tags for each multi-sentence recording file. The multi-sentence recording file should not exceed 15 seconds, and the total duration should not be less than 30 hours. The recording environment should be a quiet indoor environment. S2. Preprocessing: Perform speech processing on the short recording files of multiple sentences, convert them into the required audio files, and generate one or more files in JSON format; the process is as follows: S2.

1. Perform voice processing on the short multi-sentence recording files. The short multi-sentence recording files will be converted into audio files with an audio sampling rate of 16000Hz, an audio format of WAV, a bit depth of 16 bits, and a mono channel. S2.2, concatenating the text tag, speaker, speaker ID, and audio file tag obtained by speech processing into one or more JSON format files, wherein the text tag refers to the speaker and the speech content of the audio file corresponding to the speech content, the speaker ID refers to the number tag for the speaker, and the audio file tag refers to the name of the audio file corresponding to the speaker and the speech content; S3. Construct a learning network architecture for speech synthesis in Chinese and English, the learning network architecture including a speaker encoder, a synthesizer and a vocoder; S4, training speaker encoder: the speaker encoder takes the Mel spectrum of the audio file in step S2 as input and outputs an embedding vector of fixed dimension, which can preserve the speaker's timbre and emotion; S5. Training synthesizer: The synthesizer consists of an encoder and a decoder connected in sequence, wherein the encoder consists of a preprocessing network, three one-dimensional convolutional layers and a bidirectional LSTM layer, the input is the JSON file generated in step S2, and the output is the encoder hidden state. The decoder consists of a preprocessing network, a two-layer LSTM network connected in sequence, a projection layer and a post-processing network. The hidden state output by the encoder is decoded to generate a Mel spectrum of the synthesized speech. First, the encoder parses the JSON file generated by the preprocessing to generate the encoder hidden state, and then the encoder hidden state is input into the decoder, and the decoder outputs the Mel spectrum of the synthesized speech. S6. Train the vocoder: The vocoder consists of a parallel WaveRNN vocoder and a HiFi-GAN vocoder. It takes the Mel spectrum generated by the decoder as input and outputs a waveform for predicting speech synthesis. S7. After preprocessing the real-time speech, input it into the trained learning network architecture to obtain cross-language synthesized speech.

2. According to claim 1, a cross-language speech synthesis method based on end-to-end timbre and emotion transfer is characterized in that: The process of training the speaker encoder in step S4 is as follows: S4.

1. For a given short recording of multiple sentences, use the following formula to calculate the Mel spectrum: Among them, f is the frequency of multiple short speech sentences, and the output is the Mel spectrum of multiple short speech sentences; S4.2, the speaker encoder takes the Mel spectrum of multiple short speech as input and outputs an embedding vector of fixed dimension. The training process is as follows: S4.2.

1. Input the long Mel spectrum of indefinite length into three sequentially connected LSTM networks, each layer consisting of 768 cells. The output of each frame of the last LSTM network is mapped to a 256-dimensional fixed-length vector, where a frame refers to a fixed time unit. S4.2.

2. All outputs obtained in the above steps are averaged and normalized to obtain a final fixed-dimensional embedding vector, where the output embedding vector distinguishes the speaker corresponding to the speech from other speakers and retains the speaker's timbre and emotion.

3. The cross-language speech synthesis method based on end-to-end timbre and emotion transfer according to claim 1, characterized in that: The process of training the encoder in the synthesizer in step S5 is as follows: S5.1.

1. Get the input sequence required to generate word vectors: S5.1.1.

1. Parse the JSON file generated in step S2, and translate the text tags in the JSON file into phoneme sequences, where English text corresponds to English phonetic symbols and Chinese text corresponds to Chinese pinyin; S5.1.1.

2. Concatenate the phoneme sequences corresponding to the text tags into a JSON file, and use the resulting JSON file as the input sequence; S5.1.2, generate word vector: input the input sequence obtained in step S5.1.1 into the preprocessing network for analysis and transformation to obtain the preprocessed input sequence; perform word embedding operation on the preprocessed input sequence, calculate the weight of each phoneme in the phoneme sequence in the JSON file corresponding to the remaining phonemes, and output a 512-dimensional word vector; S5.1.3, obtain the intermediate state: input the word vector obtained in step S5.1.2 into three one-dimensional convolutional layers connected in sequence, each convolutional layer includes 512 convolution kernels of size 5*1, and perform batchNorm and dropout operations on the output after each convolutional layer. The batchNorm operation is to keep the input of each layer of the neural network in the same distribution during the deep neural network training process, and the dropout operation is to temporarily mask the neural network unit with random probability, and obtain the intermediate state at the last convolutional layer; S5.1.4, obtain the encoder hidden state: input the intermediate state obtained in step S5.1.3 into a bidirectional long short-term memory network LSTM layer to generate the encoder hidden state; S5.1.

5. Combine the embedding vectors extracted by the speaker encoder in multiple short sentences into the corresponding encoder hidden states.

4. The cross-language speech synthesis method based on end-to-end timbre and emotion transfer according to claim 3, characterized in that: The process of training the decoder in the synthesizer in step S5 is as follows: S5.2.

1. The decoder runs in a loop, where each loop is called a time step. In each decoder time step, an attention mechanism is performed. The attention mechanism is a matrix composed of context weight vectors. The attention mechanism processes the encoder hidden state, measures the similarity of the units in the hidden state, and then normalizes and obtains the context vector. S5.2.2, the preprocessing network in the decoder receives the context vector output in the previous time step, analyzes and processes the context vector, and then sends the context vector to the two layers of long short-term memory network LSTM layers connected in sequence in the decoder for processing and transformation to obtain a new context vector, and finally sends the new context vector to the projection layer to output the spectrogram frame and the end probability, wherein the spectrogram frame contains the predicted Mel spectrum; S5.2.

3. Send the spectrogram frame obtained in step S5.2.2 to the post-processing network of the decoder. The post-processing network consists of three convolutional layers connected in sequence. The post-processing network is used to improve the quality of the generated Mel spectrum and finally output the Mel spectrum of the synthesized speech.

5. The cross-language speech synthesis method based on end-to-end timbre and emotion transfer according to claim 1, characterized in that: The WaveRNN vocoder is composed of a single-layer RNN network and a double softmax layer connected in sequence. The WaveRNN vocoder is based on WaveNet. The Mel spectrum of the synthesized speech in step S5 is input into the WaveRNN vocoder. The single-layer RNN network processes and analyzes the Mel spectrum of the synthesized speech, and the output is divided into two parts. Each part is input into the corresponding softmax layer respectively, and the output is a predicted 16-bit audio sample.

6. The cross-language speech synthesis method based on end-to-end timbre and emotion transfer according to claim 1, characterized in that: The HiFi-GAN vocoder is composed of a generator and a discriminator connected in sequence, wherein the number of the generator is 1, the number of the discriminators is 2, the two discriminators receive the output of the generator in parallel, and the input of the HiFi-GAN vocoder is a GAN predicted speech waveform; The generator is a convolutional neural network that takes a Mel spectrum as input and increases the sampling rate through convolution until the target sampling rate is reached, generating predicted 16-bit audio samples. The two discriminators are the scale discriminator and the multi-cycle discriminator, which evaluate whether the waveform generated by the generator meets the standards and optimize the training alternately.

7. The cross-language speech synthesis method based on end-to-end timbre and emotion transfer according to claim 1, characterized in that: The process of step S7 is as follows: S7.

1. Collect the speech that needs to be synthesized across languages, perform preprocessing, and generate JSON format files and audio files; S7.

2. Input the JSON format file into the encoder of the synthesizer to generate the encoder hidden state; at the same time, input the Mel spectrum of the audio file into the speaker encoder to generate an embedding vector, and concatenate the embedding vector into the encoder hidden state; S7.3, inputting the encoder hidden state generated in step S7.2 into the decoder of the synthesizer to generate a Mel spectrum of the synthesized speech; S7.

4. Input the Mel spectrum of the synthesized speech into a vocoder to obtain synthesized cross-language speech, where the cross-language speech carries the timbre and emotion of the speaker.

Images