Cross-language speech synthesis method and device, electronic equipment and storage medium

Abstract

The application discloses a kind of Chinese-English cross-language speech synthesis method, device, electronic equipment and storage medium, the method includes using the sequence-to-sequence task in depth learning to construct first cross-language acoustic model;Text dataset is processed into basic sentence including phoneme sequence, tone sequence and language sequence;Utilize model encoder to encode basic sentence into high-level context semantic representation, while introducing language embedding and speaker embedding in multiple positions of model encoder;Utilize attention mechanism to learn the mapping relationship of high-level context semantic representation and acoustic feature mel-spectrogram, obtain linearly weighted high-level context semantic representation;Linearly weighted high-level context semantic representation is generated original spectrogram using model decoder.The method of the application is based on a variety of strategy fusion, constructs two cross-language acoustic models, so that the Chinese-English cross-language speech synthesis method of the application solves the deficiency of existing speech synthesis method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, and particularly relates to a method and device for cross-language speech synthesis of Chinese and English, an electronic device and a storage medium. BACKGROUND

[0002] With the start of the speech function of mobile phones, tablets, smart homes and wearable devices, the human-computer interaction mode gradually enters the speech era. Unlike traditional human-computer interaction, speech interaction has the advantages of convenience and intelligence, and can make the machine have the comprehensive ability of listening, speaking, reading and writing like a person. Speech synthesis is the last link of an intelligent speech interaction system, responsible for making the machine speak the speech audio of specific text and specific speakers, which is divided into two parts of text analysis and acoustic modeling. Text analysis is mainly to extract features from text to provide pronunciation, prosody and other text-related information for the back end. The work of the back end is to perform acoustic modeling based on the linguistic features extracted by the front end, so as to obtain natural and understandable speech output. From the mechanical simulation of simple words and phrases in the early 18th century, to the splicing synthesis method relying on a large database of recorded fragments, to the parameter synthesis system of modeling speech parameters and reconstructing speech, to the neural speech synthesis system based on neural network end-to-end modeling in the past ten years, speech synthesis has undergone great development. The model is easier to build, requires less professional knowledge, and has stronger performance. Especially based on the powerful learning ability of neural network, speech synthesis is more diversified, the synthesized audio is more expressive, and the synthesis is more controllable.

[0003] The development of speech synthesis not only meets the intelligibility and naturalness, but also gives birth to many research directions, such as personalized speech synthesis and low-resource speech synthesis. How to explore the realization of cross-language speech synthesis is a very important research content. In real-life scenarios, it is not uncommon to see English words mixed with Chinese characters and speaking expressions. Especially in educational scenarios such as mathematics, the reading of English letters, Greek letters, and triangular symbols in mathematical formulas requires the speech synthesis system to have the ability of cross-language synthesis. However, the current mainstream speech synthesis model only supports single-language synthesis by default. Since professional voice actors who are proficient in multiple languages are very scarce, high-quality English-Chinese mixed reading recordings are rare, and the recording and labeling of mixed reading audio are costly and expensive, which increases the difficulty of cross-language synthesis tasks. Fortunately, the open source of a large amount of high-quality single-language speech data makes it possible to realize a cross-language speech synthesis system. Therefore, it is of great significance to explore a natural and fluent cross-language speech synthesis system under the condition of single-language recording. In addition, while making the machine speak, it is also necessary to ensure that the machine speaks naturally, and the speech synthesis system should have high expressiveness, so prosody modeling is an essential step. There are mature methods for Chinese prosody prediction and prosody modeling, and it is also very important to explore how to better model the prosodic features of cross-language text and improve the naturalness and prosody of the cross-language speech synthesis system. SUMMARY

[0004] The present application is directed to the above problems, and provides a cross-language speech synthesis method, device, electronic equipment and storage medium, which is based on multiple strategy fusion and constructs two cross-language acoustic models, so that the cross-language speech synthesis method solves the shortcomings of the existing speech synthesis method.

[0005] In a first aspect of the present application, a cross-language speech synthesis method for English and Chinese includes the following steps:

[0006] A first cross-language acoustic model is constructed using a sequence-to-sequence task in deep learning;

[0007] A text data set is processed into a basic sentence including a phoneme sequence, a tone sequence and a language sequence;

[0008] The basic sentence is encoded into a high-level context semantic representation by using a first cross-language acoustic model encoder, and language embedding and speaker embedding are introduced at multiple positions of the first cross-language acoustic model encoder;

[0009] The mapping relationship between the high-level context semantic representation and the acoustic feature mel-spectrogram is learned by using an attention mechanism, and a linearly weighted high-level context semantic representation is obtained;

[0010] The linearly weighted high-level context semantic representation is decoded by a first cross-language acoustic model decoder to generate an original spectrogram.

[0011] Further, the first cross-language acoustic model is based on a Tacotron model, comprising a CBHG-based encoder, a GMMv2b attention mechanism module based on a Gaussian mixture distribution and a decoder.

[0012] Further, the method further comprises fine-tuning the first cross-language acoustic model by using an experience replay-based continual learning method, and fixing the parameters of the first cross-language acoustic model during fine-tuning in a minimum error range of the parameters of the first cross-language acoustic model before fine-tuning by using a regularization-based plastic weight stabilization method.

[0013] Further, the method further comprises extending the prosodic structure of Chinese to the English-Chinese cross-language text, and the specific method comprises: taking an English word or a single letter as a prosodic word in a Chinese four-level prosodic structure, marking prosodic phrase boundaries and intonation phrase boundaries according to different lengths of pauses in the text corresponding audio, and mixing the prosodic phrase boundaries and the intonation phrase boundaries into the phoneme sequence.

[0014] Further, the method further comprises constructing a hierarchical prosody graph according to the hierarchical prosody of the basic sentence, and integrating a graph neural network into the first cross-language acoustic model encoder to form a second cross-language acoustic model, wherein the graph neural network is used for multiple iterations in a spatial domain according to the hierarchical prosody graph to propagate prosodic information between phoneme node representations.

[0015] Further, the second cross-language acoustic model encoder comprises a convolutional layer, a highway network, a gated graph neural network and a bidirectional GRU, wherein the convolutional layer is used for extracting local context representations of the input basic sentence, the highway network is used for extracting high-dimensional features to construct robust input node representations, and the bidirectional GRU is used for learning forward and backward text representations of the input basic sentence.

[0016] In a second aspect, the present application provides an English-Chinese cross-language speech synthesis device, which comprises:

[0017] A first cross-language acoustic model construction unit is configured to construct a first cross-language acoustic model by using a sequence-to-sequence task in deep learning.

[0018] A text dataset processing unit is configured to process a text dataset into a basic sentence comprising a phoneme sequence, a tone sequence and a language sequence.

[0019] A first cross-language acoustic model encoder unit is configured to encode the basic sentence into a high-level context semantic representation by using a first cross-language acoustic model encoder, and introduce language embeddings and speaker embeddings at multiple positions of the first cross-language acoustic model encoder.

[0020] The attention mechanism learning unit is configured to learn a mapping relationship between the high-level context semantic representation and the acoustic feature mel-spectrogram by using an attention mechanism, and obtain a linearly weighted high-level context semantic representation.

[0021] The first cross-lingual acoustic model decoder unit is configured to generate the original spectrogram from the linearly weighted high-level context semantic representation by using a first cross-lingual acoustic model decoder.

[0022] Further, the first cross-lingual acoustic model encoder unit is also configured to construct a hierarchical prosody graph according to the hierarchical prosody of the base sentence, and integrate a graph neural network into the first cross-lingual acoustic model encoder to form a second cross-lingual acoustic model, wherein the graph neural network is configured to perform multiple iterations in a spatial domain to propagate prosody information between phoneme node representations.

[0023] In a third aspect, the present application provides an electronic device, comprising: a processor; and a memory, wherein the memory stores a computer executable program, and when the computer executable program is executed by the processor, the above-mentioned cross-lingual speech synthesis method for Chinese and English is performed.

[0024] In a fourth aspect, the present application provides a computer readable storage medium, which stores instructions, and when the instructions are executed by a processor, the processor performs the above-mentioned cross-lingual speech synthesis method for Chinese and English.

[0025] The cross-lingual speech synthesis method for Chinese and English, the device, the electronic device and the storage medium provided by the present application are based on in-depth research on the existing speech synthesis method, and a multi-strategy fusion cross-lingual acoustic model and a graph-based cross-lingual acoustic model are designed. The present application mainly includes a multi-strategy fusion cross-lingual acoustic model and a graph-based expressiveness speech synthesis method. In order to improve the robustness of the cross-lingual acoustic model and the naturalness of the synthesized speech, a plurality of optimization strategies are introduced, and a first cross-lingual acoustic model CS-Tacotron is proposed based on the Tacotron model: by optimizing the model input into three parts of phoneme sequence, tone sequence and language sequence to realize information decoupling and reduce the number of phonemes; by introducing language embedding at multiple positions of the model to better model the naturalness when switching between Chinese and English languages; by introducing speaker embedding at multiple positions of the model to improve the timbre similarity of multi-speaker modeling; and by introducing a continuous learning method to improve the synthesis effect when fine-tuning the cross-lingual model using Chinese monolingual recording data.

[0026] In order to improve the expressiveness of the synthesized speech of the Chinese cross-language speech synthesis model, the prosodic pause modeling of the speech synthesis is researched. The hierarchical prosodic structure of Chinese is generalized to the Chinese-English cross-language text to improve the effect of cross-language speech synthesis. The hierarchical prosodic structure usually mixes prosodic boundaries of different levels as phonemes into the input sequence, and the model learns the corresponding pause duration autonomously according to the training data. The hierarchical prosodic graph is constructed from the hierarchical prosodic structure of the input cross-language text, and a graph neural network is introduced for modeling, realizing an alternative way of prosodic information modeling, and exploring and proposing a second cross-language acoustic model GCS-Tacotron based on the graph. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a Chinese-English cross-language speech synthesis method flowchart of an embodiment of the present application;

[0028] Figure 2 is a structural schematic diagram of a first cross-language acoustic model CS-Tacotron model of an embodiment of the present application;

[0029] Figure 3 is an input data schematic diagram of the CS-Tacotron model of an embodiment of the present application;

[0030] Figure 4 is a CS-Tacotron model fine-tuning schematic diagram based on a continuous learning method of an embodiment of the present application;

[0031] Figure 5 is an extended Chinese-English hierarchical prosodic structure schematic diagram of an embodiment of the present application;

[0032] Figure 6 is a prosodic structure representation schematic diagram of the hierarchical prosodic structure in the input text of an embodiment of the present application;

[0033] Figure 7 is a hierarchical prosodic graph extracted by an embodiment of the present application;

[0034] Figure 8 is a structural schematic diagram of a second cross-language acoustic model GCS-Tacotron model of an embodiment of the present application;

[0035] Figure 9 is a Chinese-English cross-language speech synthesis device structural schematic diagram of an embodiment of the present application;

[0036] Figure 10 is an architecture diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0037] To make the technical solutions of the present application more comprehensible, the embodiments are implemented on the premise of the technical solutions of the present application, and detailed implementation manners and specific steps are given.

[0038] The embodiments of the present application are directed to a cross-language speech synthesis method and device, an electronic device and a storage medium, and provide the following embodiments.

[0039] Based on the embodiment 1 of the present application

[0040] The present embodiment is used to illustrate the principle and steps of solving the technical problems of the present application, as shown in Figure 1 The flow chart of the cross-language speech synthesis method of the embodiment 1 of the present application is shown in

[0041] S1, a first cross-language acoustic model is constructed by using a sequence-to-sequence task in deep learning.

[0042] Further, the first cross-language acoustic model is based on a Tacotron model, which includes a CBHG-based encoder, a GMMv2b attention mechanism module based on a Gaussian mixture distribution, and a decoder.

[0043] In the specific implementation process, as shown in Figure 2 The first cross-language acoustic model CS-Tacotron includes an encoder, which is a CBHG-based encoder, language embedding is added to a convolutional network after being activated by different linear layers and nonlinear layers, and is used as a high-speed network gate; a GMMv2b attention mechanism module based on a Gaussian mixture distribution; and a decoder that outputs a predicted spectrogram and a predicted stop symbol.

[0044] S2, processing the text data set into a basic sentence including a phoneme sequence, a tone sequence and a language sequence.

[0045] In the specific implementation process, before processing the text data set, the text data set needs to be preprocessed, that is, the data set needs to be cleaned and prosodic annotated. In order to design the text input format of the model, for cross-language speech synthesis of Chinese and English, Chinese generally uses pinyin characters as input, while English directly uses characters. The present application optimizes the preprocessed text data set into three parts of phoneme sequence, tone sequence and language sequence to realize the decoupling of language information, tone information and phoneme information and reduce the number of phonemes, thereby reducing the coding space size. As shown in Figure 3 The input is composed of a phoneme sequence, a tone sequence and a language sequence, which are respectively subjected to corresponding encoding layers to obtain an encoding representation, aiming to reduce the number of phonemes, optimize the input of the model and thus improve the naturalness of the synthesized speech.

[0046] S3, encoding the base sentence into a high-level context semantic representation using the first cross-language acoustic model encoder, while introducing language embeddings and speaker embeddings at multiple positions of the first cross-language acoustic model encoder;

[0047] Reference Figure 1 , the CS-Tacotron model encoder encodes the input base sentence into a high-level context semantic representation, while introducing language embeddings at multiple positions of the CS-Tacotron model to better model the local context information when switching between Chinese and English languages, making the modeling more natural, and introducing speaker embeddings at multiple positions of the CS-Tacotron model to improve the timbre similarity of multi-speaker modeling.

[0048] In the implementation process, the CS-Tacotron model input is the concatenation of phoneme embeddings, intonation embeddings, and language embeddings. The explicit language embeddings reduce the difficulty of model modeling; the language embeddings are processed by a fully connected layer and a Softsign function, added to the output of the encoder convolutional network, and used as the gating of the high-speed network, which strengthens the linguistic information and better models the context information when switching between languages; the speaker embeddings are processed by a fully connected layer and a Softsign function, used as the initial state of the bidirectional GRU, added to the encoder output, and concatenated with the input of the decoder LSTM, which strengthens the speaker information and improves the timbre similarity of multi-speaker modeling.

[0049] S4, learning the mapping relationship between the high-level context semantic representation and the acoustic feature mel-spectrogram using the attention mechanism, to obtain a linearly weighted high-level context semantic representation;

[0050] In the implementation process, the GMMv2b attention mechanism based on Gaussian mixture distribution is used to learn the mapping relationship between the input high-level context semantic representation sequence and the output sequence, i.e., the mapping relationship between the text and the acoustic feature mel-spectrogram. This part completes the semantic representation conversion from the input text to the mel-spectrogram, so that the subsequent decoding can be generated by the decoding end model. The GMMv2b attention mechanism is used to learn the mapping relationship between the high-level context semantic representation and the acoustic feature mel-spectrogram, and the input high-level context semantic representation is linearly weighted to obtain a linearly weighted high-level context semantic representation.

[0051] S5, generating the original spectrogram from the linearly weighted high-level context semantic representation using the first cross-language acoustic model decoder;

[0052] In the implementation process, reference is made to Figure 2As shown, the decoding end of the CS-Tacotron model based on the Tacotron model generates the original spectrogram in the first cross-language acoustic model decoder RNN network according to the mapping relationship learned by the attention mechanism and the high-level context semantic input encoded by the CS-Tacotron model encoder, and synthesizes the original spectrogram into speech, and by introducing speaker embedding at multiple positions of the decoder, the timbre similarity of the synthesized speech is improved, and the modeling difficulty is reduced.

[0053] Finally, the original spectrogram is smoothed by the post-processing network to generate a spectrogram that can be synthesized into speech, and then the spectrogram is reconstructed into an audio waveform by the vocoder.

[0054] Based on the embodiment 2 of the present application

[0055] The present embodiment is used to fine-tune the constructed first cross-language acoustic model CS-Tacotron model, and when the cross-language acoustic model is fine-tuned using Chinese monolingual data, the synthesized cross-language speech effect will be poor, and there is a catastrophic forgetting problem in cross-language speech synthesis. The present application introduces a continuous learning method to improve the synthesis effect when the cross-language CS-Tacotron model is fine-tuned using Chinese monolingual data.

[0056] Further, the first cross-language acoustic model is fine-tuned using the experience replay-based continuous learning method, and the first cross-language acoustic model parameters during fine-tuning are fixed within the minimum error range of the first cross-language acoustic model parameters before fine-tuning by using the regular-based plastic weight consolidation method. By introducing two continuous learning fine-tuning methods based on plastic weight consolidation and experience replay, the effect of synthesizing cross-language speech by the monolingual acoustic model during fine-tuning is improved. In the specific implementation process, it is time-consuming and laborious to train a new CS-Tacotron model from zero, so usually the CS-Tacotron model is trained using mixed data first, and then the CS-Tacotron model is fine-tuned by data to obtain a fine-tuned CS-Tacotron model with corresponding timbre customization. The fine-tuning method based on continuous learning is introduced to improve the problem of poor synthesized speech effect when the cross-language acoustic model is fine-tuned using monolingual data, which specifically includes: introducing the regular-based plastic weight consolidation (Elastic Weight Consolidation, EWC) method to fix the fine-tuning model parameters within the minimum error range of the base cross-language model parameters, so as to ensure the effect of synthesizing cross-language speech by the monolingual acoustic model; and introducing the experience replay (Experience Replay, ER) based continuous learning method to ensure the effect of synthesizing cross-language speech by the monolingual acoustic model. For example Figure 4As shown is a fine-tuning schematic diagram based on a continuous learning method, showing the training trajectory of the method based on plastic weight stabilization and experience replay at the time of fine-tuning, so that the model parameters are kept near the model parameters before fine-tuning to ensure the effect of synthesizing cross-language speech. A* in the figure is the low-error parameter of the model before fine-tuning, the dashed curve is the training trajectory without any constraint, which will lead to poor model performance, and the solid curve and the solid point curve are the training trajectories of the plastic weight stabilization and experience replay method, which can better find the optimal parameters before and after fine-tuning.

[0057] Based on the embodiment 3 of the present application

[0058] In order to improve the expressiveness of the CS-Tacotron model for synthesizing speech, the prosodic pause modeling for speech synthesis is studied in this embodiment, and the hierarchical prosodic structure of Chinese is generalized to English-Chinese cross-language text to improve the effect of cross-language speech synthesis. The hierarchical prosodic structure usually mixes different levels of prosodic boundaries into the input sequence as phonemes, and the model learns the corresponding pause duration autonomously according to the training data. A hierarchical prosodic graph is constructed from the hierarchical prosodic structure of the input cross-language text, and a graph neural network is introduced for modeling, realizing an alternative way of prosodic information modeling, and a second cross-language acoustic model GCS-Tacotron model based on graph is proposed.

[0059] Further, the prosodic structure of Chinese is extended to English-Chinese cross-language text, and the specific method includes: regarding English words or single letters as prosodic words in the four-level prosodic structure of Chinese, annotating prosodic phrase boundaries and intonation phrase boundaries according to different lengths of pauses in the text corresponding audio, and mixing prosodic phrase boundaries and intonation phrase boundaries into phoneme sequences as phonemes.

[0060] In the specific implementation process, considering that the prosodic structure of Chinese has a hierarchical nested property, the prosodic structure of Chinese is extended to English-Chinese text distribution scenarios to improve the expressiveness of English-Chinese cross-language speech synthesis. The hierarchical prosodic structure of Chinese is extended to English-Chinese distribution scenarios, and intonation phrase and prosodic phrase boundary graphs are constructed therefrom, specifically including:

[0061] Chinese contains four-level prosodic structures of prosodic words, prosodic phrases, intonation phrases, and sentences, and the present application regards English words or single letters as prosodic words, thereby extending the prosodic structure of Chinese to English-Chinese distribution scenarios, such as Figure 5 As shown is a schematic diagram of the English-Chinese hierarchical prosodic structure extended by the embodiment, the English-Chinese text prosody mainly includes four levels of prosodic words, prosodic phrases, intonation phrases, and sentences, and has a hierarchical nested property with the characteristic of gradually increasing pause duration;

[0062] The way of Chinese prosody annotation is extended to the prosody annotation of Chinese and English cross-language text, and the English word or single letter is regarded as the prosodic word of Chinese. The prosodic phrase PPH and intonation phrase IPH boundaries are annotated according to the different lengths of pauses in the text corresponding audio, as shown in Figure 6 Fig. 3 shows the prosodic structure representation of the hierarchical prosody in the input text in the embodiment, as shown in Figure 7 Fig. 4 shows the extracted hierarchical prosody graph in the embodiment, which includes two types of edges of prosodic phrase PPH and intonation phrase IPH, and the edges have uniqueness.

[0063] Further, the graph neural network is integrated into the first cross-language acoustic model encoder to form a second cross-language acoustic model, and the graph neural network is iterated in the spatial domain according to the hierarchical prosody graph to propagate prosodic information between the phoneme node representations.

[0064] In the implementation process, the graph neural network is integrated into the CS-Tacotron model to model the prosodic information of the input text according to the constructed hierarchical prosody graph.

[0065] Based on the embodiment 4 of the present application

[0066] The present application is based on the embodiment 3, and the second cross-language acoustic model GCS-Tacotron model is described, as shown in Figure 8 In order to better integrate the graph neural network into the acoustic model and reduce the calculation amount of the model, the second cross-language acoustic model GCS-Tacotron model based on the graph neural network is further proposed on the basis of the CS-Tacotron model. The GCS-Tacotron model encoder is composed of four parts: a convolutional layer with switchable normalization (SN) to extract the local context representation of the input text; a high-speed network to slow down the gradient vanishing and extract high-dimensional features to construct a robust input node representation; a gated graph neural network to propagate prosodic information between phoneme node representations through multiple iterations in the spatial domain according to the constructed hierarchical prosody graph; and a bidirectional GRU (bidirectional gated recurrent unit) to learn the forward and backward text representations of the input text. The GCS-Tacotron model encoder is used to model the local context information, prosodic information and forward and backward sequence features of the input text to obtain a robust text representation. The GCS-Tacotron model attention mechanism module is the same as the CS-Tacotron model, which is also the GMMv2b attention mechanism based on Gaussian mixture distribution, and the GCS-Tacotron model decoder is used to output the predicted spectrogram and stop symbol.

[0067] Based on the embodiment 5 of the present application

[0068] In the following, with reference toFigure 9 To describe the relationship with Embodiments 1 to 4 of this disclosure Figures 1 to 9 The apparatus corresponding to the method shown is a cross-lingual speech synthesis apparatus for Chinese and English. The apparatus 100 includes: a first cross-lingual acoustic model construction unit 101, used to construct a first cross-lingual acoustic model using a sequence-to-sequence task in deep learning; a text dataset processing unit 102, used to process the text dataset into basic sentences including phoneme sequences, tone sequences, and language sequences; a first cross-lingual acoustic model encoder unit 103, used to encode the basic sentences into high-level contextual semantic representations using the first cross-lingual acoustic model encoder, and simultaneously introduce language embeddings and speaker embeddings at multiple positions in the first cross-lingual acoustic model encoder; an attention mechanism learning unit 104, used to learn the mapping relationship between text mapping and acoustic feature Mel spectrograms using an attention mechanism; and a first cross-lingual acoustic model decoder unit 105, used to generate an original spectrogram from the high-level contextual semantic representations using the first cross-lingual acoustic model decoder according to the mapping relationship, and synthesize speech from the original spectrogram. In addition to the above five units, the apparatus 100 may also include other components; however, since these components are not related to the content of this disclosure embodiment, their illustrations and descriptions are omitted here.

[0069] Furthermore, the first cross-lingual acoustic model encoder unit 103 is also used to construct a hierarchical prosodic graph based on the hierarchical prosodic of the basic sentences, and to integrate the graph neural network into the first cross-lingual acoustic model encoder to form a second cross-lingual acoustic model. The graph neural network performs multiple iterations in the spatial domain based on the hierarchical prosodic graph to propagate prosodic information between phoneme node representations.

[0070] The specific working process of a Chinese-English cross-language speech synthesis device 100 is described in accordance with the above-described embodiments 1 to 4 of the Chinese-English cross-language speech synthesis method, and will not be repeated here.

[0071] Based on Embodiment 6 of the present invention

[0072] The apparatus according to embodiments of the present invention can also be used by means of Figure 10 The architecture of the electronic device shown is used to implement this. Figure 10 The architecture of the electronic device is shown. (For example...) Figure 10 As shown, the computer system 201 includes a system bus 203, one or more CPUs 204, input / output 202, and memory 205. Memory 205 can store various data or files used by the computer for processing and / or communication, as well as program instructions executed by the CPU, including the Chinese-English cross-language speech synthesis methods of Embodiments 1 to 4. Figure 10 The architecture shown is merely exemplary and should be adjusted according to actual needs when implementing different devices. Figure 10 One or more components in.

[0073] Based on the embodiment 7 of the present application

[0074] The embodiments of the present application can also be implemented as a computer readable storage medium. The computer readable storage medium according to the embodiment 7 has stored computer readable instructions. When the computer readable instructions are run by a processor, the method for Chinese-English cross-language speech synthesis according to the embodiments 1 to 4 of the present application described with reference to the above drawings can be executed.

[0075] The method, device, electronic equipment and storage medium provided by the above embodiments comprehensively provide a Chinese-English cross-language speech synthesis method, device, electronic equipment and storage medium. Based on in-depth research on the existing speech synthesis method, a multi-strategy fusion Chinese-English cross-language acoustic model and a graph-based cross-language acoustic model are designed. The present application mainly includes a multi-strategy fusion cross-language acoustic model and a graph-based expressiveness speech synthesis method. In order to improve the robustness of the cross-language acoustic model and the naturalness of the synthesized speech, multiple optimization strategies are introduced. A first cross-language acoustic model CS-Tacotron is proposed based on the Tacotron model: by optimizing the model input into three parts of phoneme sequence, tone sequence and language sequence to realize information decoupling and reduce the number of phonemes; by introducing language embedding at multiple positions of the model to better model the naturalness when switching between Chinese and English languages; by introducing speaker embedding at multiple positions of the model to improve the timbre similarity of multi-speaker modeling; by introducing a continuous learning method to improve the synthesis effect when fine-tuning the cross-language model using Chinese monolingual recording data. In order to improve the expressiveness of the synthesized speech of the Chinese cross-language speech synthesis model, the prosodic pause modeling of speech synthesis is studied. The hierarchical prosodic structure of Chinese is generalized to Chinese-English cross-language text in the present application to improve the effect of cross-language speech synthesis. The hierarchical prosodic structure usually mixes different levels of prosodic boundaries as phonemes into the input sequence, and the model learns the corresponding pause duration autonomously according to the training data. A hierarchical prosodic graph is constructed from the hierarchical prosodic structure of the input cross-language text, and a graph neural network is introduced for modeling, realizing an alternative way of prosodic information modeling, and exploring and proposing a second cross-language acoustic model GCS-Tacotron based on a graph.

[0076] In this document, the terms "comprises", "comprising", or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article of manufacture, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0077] The above is further detailed description of the present application in combination with specific preferred embodiments, and cannot be deemed as limitation of the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, and all should be deemed as falling within the protection scope of the present application.

Claims

1. A method for cross-lingual speech synthesis from Chinese to English, characterized in that, The method comprises the following steps: a first cross-language acoustic model is constructed using a sequence-to-sequence task in deep learning; a text data set is processed into a basic sentence comprising a phoneme sequence, a tone sequence and a language sequence; the basic sentence is encoded into a high-level context semantic representation using a first cross-language acoustic model encoder, while language embeddings are introduced at multiple positions of the first cross-language acoustic model encoder and speaker embeddings are introduced at multiple positions of the first cross-language acoustic model encoder; a mapping relationship between the high-level context semantic representation and an acoustic feature mel-spectrogram is learned using an attention mechanism to obtain a linearly weighted high-level context semantic representation; the linearly weighted high-level context semantic representation is generated into an original spectrogram using a first cross-language acoustic model decoder. The first cross-language acoustic model is based on a Tacotron model and comprises a CBHG-based encoder, a GMMv2b attention mechanism module based on a Gaussian mixture distribution and a decoder.

2. The method of claim 1, wherein, The method further comprises fine-tuning the first cross-language acoustic model using an experience replay-based continual learning method, and fixing the parameters of the first cross-language acoustic model during fine-tuning within a minimum error range of the parameters of the first cross-language acoustic model before fine-tuning using a regularized plastic weight stabilization method.

3. The method of claim 1, wherein, The method further comprises extending the prosodic structure of Chinese to cross-language text in Chinese and English, and the specific method comprises: regarding an English word or a single letter as a prosodic word in a Chinese four-level prosodic structure, labeling prosodic phrase boundaries and tone phrase boundaries according to different lengths of pauses in the text corresponding audio, and mixing the prosodic phrase boundaries and the tone phrase boundaries into a phoneme sequence.

4. The method of claim 1, wherein, The method further comprises constructing a hierarchical prosody graph according to the hierarchical prosody of the basic sentence, integrating a graph neural network into the first cross-language acoustic model encoder to form a second cross-language acoustic model, and the graph neural network is iterated in a spatial domain multiple times to propagate prosodic information between phoneme node representations.

5. The method of claim 4, wherein, The second cross-language acoustic model encoder comprises a convolutional layer, a highway network, a gated graph neural network and a bidirectional GRU, wherein the convolutional layer is used to extract local context representations of the input basic sentence; the highway network is used to extract high-dimensional features to construct robust input node representations; and the bidirectional GRU is used to learn forward and backward text representations of the input basic sentence.

6. A cross-lingual speech synthesis apparatus for Chinese and English, characterized by, The device comprises: a first cross-language acoustic model construction unit configured to construct a first cross-language acoustic model using a sequence-to-sequence task in deep learning; a text data set processing unit configured to process a text data set into a basic sentence comprising a phoneme sequence, a tone sequence and a language sequence; a first cross-language acoustic model encoder unit configured to encode the basic sentence into a high-level context semantic representation using a first cross-language acoustic model encoder, while introducing language embeddings at multiple positions of the first cross-language acoustic model encoder and introducing speaker embeddings at multiple positions of the first cross-language acoustic model encoder; an attention mechanism learning unit configured to learn a mapping relationship between the high-level context semantic representation and an acoustic feature mel-spectrogram using an attention mechanism to obtain a linearly weighted high-level context semantic representation; The first cross-language acoustic model decoder unit is configured to generate the original spectrogram from the linearly weighted high-level context semantic representation by using a first cross-language acoustic model decoder. The first cross-language acoustic model is based on a Tacotron model, and includes a CBHG-based encoder, a GMMv2b attention mechanism module based on a Gaussian mixture distribution, and a decoder.

7. The apparatus according to claim 6, wherein The first cross-language acoustic model encoder unit is further configured to construct a hierarchical prosody graph according to a hierarchical prosody of the base sentence, and to form a second cross-language acoustic model by integrating a graph neural network into the first cross-language acoustic model encoder, the graph neural network being configured to perform multiple iterations in a spatial domain to propagate prosody information between phoneme node representations.

8. An electronic device, comprising: The method comprises: a processor; and a memory, wherein the memory has stored therein computer executable programs which, when executed by the processor, perform the method of any one of claims 1-5.

9. A computer-readable storage medium having stored thereon instructions, The instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1-5.

Images