Cross-language speech synthesis method and system based on dual speaker embedding

Abstract

The embodiment of the application provides a cross-language speech synthesis method based on double speaker embedding. The method comprises the following steps: inputting text and native language speaker embedding into a txt2vec acoustic model, determining the phoneme sequence code of the text through a text encoder, and determining the vector quantization acoustic feature and the auxiliary feature from the phoneme sequence code and the native language speaker embedding through a decoder; inputting the target language speaker embedding, the vector quantization acoustic feature and the auxiliary feature into a vec2wav vocoder, extracting the X-vector feature of the target language speaker embedding, inputting the X-vector feature, the vector quantization acoustic feature and the auxiliary feature into a feature encoder, and obtaining a cross-language acoustic feature; and determining the cross-language synthesized speech of the cross-language acoustic feature by using a generator. The embodiment of the application constructs a cross-language TTS model based on VQTTS, models the language speaking style and the speaker timbre respectively, and thus realizes cross-language speech synthesis with high nativeness and similar timbre to the target speaker.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent speech, in particular to a cross-lingual speech synthesis method and system based on dual speaker embedding. BACKGROUND

[0002] With the development of technology, TTS (Text To Speech) models have made great progress in synthesizing high-fidelity and rich prosody speech. However, in the multi-lingual TTS scenario, the speech effect of cross-lingual synthesis is still not satisfactory, because it is difficult for the synthesized speech in this scenario to accurately preserve the timbre of the speaker and eliminate the stress in their first language. More specifically, cross-lingual synthesis (speech synthesis from a native language to a non-native language) is difficult to obtain non-native nativeness while maintaining speaker similarity, where nativeness refers to the closeness of the speech to the native language (i.e. the synthesized speech has a heavy accent and does not feel natural).

[0003] To solve the above problems, the prior art will use a domain adversarial training method to enable the speech synthesis model to transfer the voice features of different speakers of different languages; will use a method of minimizing mutual information to maintain speaker consistency during cross-lingual synthesis; and will use a loss function to encourage the speech synthesis model to learn language-independent speaker representations.

[0004] In the process of implementing the present application, the inventors have found that the related art has at least the following problems:

[0005] These methods in the prior art usually rely on fused spectrograms as acoustic features, which are highly correlated in the time and frequency axes and contain rich speaker-related information. However, the rich speaker-related information makes it difficult to completely decouple speaker information and language information during cross-lingual synthesis, which makes it difficult to obtain naturalness in non-native languages while maintaining speaker similarity. SUMMARY

[0006] To at least solve the problem in the prior art that it is difficult to obtain naturalness in non-native languages while maintaining speaker similarity during cross-lingual synthesis.

[0007] In a first aspect, an embodiment of the present application provides a cross-lingual speech synthesis method based on dual speaker embedding, comprising:

[0008] inputting text and a native language speaker embedding into a txt2vec acoustic model, in which the text encoder determines the phoneme sequence encoding of the text, and the decoder determines the vector quantized acoustic features and auxiliary features of the native language speaker pronunciation style from the phoneme sequence encoding and the native language speaker embedding.

[0009] inputting the vector-quantized acoustic features and the auxiliary features into a vec2wav vocoder, in which an X-vector feature of the target language speaker embedding is extracted, and inputting the X-vector feature, the vector-quantized acoustic features and the auxiliary features into a feature encoder to obtain cross-language acoustic features simulating a tone color of a target language speaker on the basis of a pronunciation style of a native language speaker;

[0010] determining cross-language synthesized speech of the cross-language acoustic features by using a generator.

[0011] In a second aspect, an embodiment of the present application provides a cross-language speech synthesis system based on double speaker embeddings, comprising:

[0012] a pronunciation style feature determination program module configured to input text and a native language speaker embedding into a txt2vec acoustic model, in which a phoneme sequence code of the text is determined by a text encoder, and vector-quantized acoustic features and auxiliary features of a pronunciation style of a native language speaker are determined from the phoneme sequence code and the native language speaker embedding by a decoder;

[0013] a tone color feature determination program module configured to input a target language speaker embedding of a non-native language, the vector-quantized acoustic features and the auxiliary features into a vec2wav vocoder, in which an X-vector feature of the target language speaker embedding is extracted, and inputting the X-vector feature, the vector-quantized acoustic features and the auxiliary features into a feature encoder to obtain cross-language acoustic features simulating a tone color of a target language speaker on the basis of a pronunciation style of a native language speaker;

[0014] a speech synthesis program module configured to determine cross-language synthesized speech of the cross-language acoustic features by using a generator.

[0015] In a third aspect, an electronic device is provided, comprising at least one processor, and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform steps of the cross-language speech synthesis method based on double speaker embeddings of any embodiment of the present application.

[0016] In a fourth aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method for cross-lingual speech synthesis based on dual speaker embedding according to any one of the embodiments of the present application.

[0017] The method according to the embodiment of the present application has the advantages that the method constructs a cross-lingual TTS model based on VQTTS, which is composed of dual speaker embedding and independently models language speaking style and speaker timbre. The VQ feature has less speaker-related characteristics. By using this finding, cross-lingual speech synthesis with high naturalness and similar timbre to the target speaker is achieved. Experiments show that the method is superior to the prior art in terms of intra-lingual and cross-lingual speech synthesis. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0019] Figure 1 is a flowchart of a method for cross-lingual speech synthesis based on dual speaker embedding provided by an embodiment of the present application;

[0020] Figure 2 is a DSE-TTS framework diagram of a method for cross-lingual speech synthesis based on dual speaker embedding provided by an embodiment of the present application;

[0021] Figure 3 is a training data set information diagram of a method for cross-lingual speech synthesis based on dual speaker embedding provided by an embodiment of the present application;

[0022] Figure 4 is a speaker classification accuracy diagram of different acoustic features of a method for cross-lingual speech synthesis based on dual speaker embedding provided by an embodiment of the present application;

[0023] Figure 5 is a native language MOS and ASR intra-lingual synthesis data diagram of a method for cross-lingual speech synthesis based on dual speaker embedding provided by an embodiment of the present application;

[0024] Figure 6 is a cross-lingual speech synthesis data diagram of a method for cross-lingual speech synthesis based on dual speaker embedding provided by an embodiment of the present application;

[0025] Figure 7is a structural schematic diagram of a cross-language speech synthesis system based on double speaker embedding provided by an embodiment of the present application.

[0026] Figure 8 is a structural schematic diagram of an embodiment of an electronic device for cross-language speech synthesis based on double speaker embedding provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0028] As Figure 1 is a flowchart of a cross-language speech synthesis method based on double speaker embedding provided by an embodiment of the present application, including the following steps:

[0029] S11: inputting text and native language speaker embedding into a txt2vec acoustic model, in which a text encoder is used to determine a phoneme sequence code of the text, and a decoder is used to determine a vector quantized acoustic feature of a native language speaker pronunciation style and an auxiliary feature from the phoneme sequence code and the native language speaker embedding;

[0030] S12: inputting a target language speaker embedding as a non-native language, the vector quantized acoustic feature and the auxiliary feature into a vec2wav vocoder, in which an X-vector feature of the target language speaker embedding is extracted, the X-vector feature, the vector quantized acoustic feature and the auxiliary feature are inputted into a feature encoder to obtain a cross-language acoustic feature simulating a target language speaker timbre based on a native language speaker pronunciation style;

[0031] S13: determining a cross-language synthesized speech of the cross-language acoustic feature by using a generator.

[0032] In the present embodiment, it is found by experiments that the quantized acoustic feature contains little speaker information. Therefore, the quantized feature is easier to separate timbre and language information than the traditional mel-frequency spectrum. Based on this finding, the present method uses double speaker embedding (native language speaker embedding, target language speaker embedding as a non-native language) to control the timbre and speaking style of the synthesized speech respectively, so as to solve the problem of cross-language speech synthesis.

[0033] For step S11, the present method is improved on the basis of a VQTTS (vector-quantized TTS (Text To Speech)) speech synthesis model. The improved acoustic model architecture and the framework of double speaker embedding are as shown in Figure 2 In a specific speech synthesis step, the text to be synthesized by speech is input into the model architecture of the present method. For double speaker embedding, the native language speaker embedding is input into the acoustic model txt2vec, and the target language speaker embedding of the non-native language is input into the vocoder vec2wav.

[0034] For the acoustic model txt2vec, the present method takes speaker and language ID as input. The speaker ID is embedded into a 256-dimensional vector, and then the vectors are projected and added to the encoder output. The language ID is processed in a similar way to support various languages embedded in a 128-dimensional vector. These two embeddings are used to learn the language features of different languages.

[0035] In the acoustic model txt2vec processing part, the input text to be synthesized by speech is standardized, and the text is converted into an IPA (International Phonetic Alphabet) phoneme sequence using a phoneme toolkit.

[0036] In order to facilitate the alignment between text and speech, the tones and accents of different languages are retained in the input sequence. Shared punctuation symbols across languages are also used, which are divided into four groups according to the length of the pause, and are denoted as “sp1”, “sp2”, “sp3” and “sp4” respectively. In addition, “sil” is used as the start and end markers of each sentence. Before feeding the input sequence into the text encoder in txt2vec, an embedding table is used to assign a 384-dimensional vector to each phoneme (token).

[0037] The international phonetic alphabet phoneme sequence and the native language speaker embedding are processed through an auxiliary controller to obtain PL (Phone-level) auxiliary labels, which can be used to assist in more accurate prediction of the index of each codebook (these steps are preliminary preparations for subsequent extraction of VQ (vector-quantized) acoustic features, and accurate prediction of these indexes can construct high-fidelity speech).

[0038] As an implementation, after determining the phoneme sequence encoding of the text through the text encoder, the method further comprises:

[0039] aligning the phoneme sequence and the native language speaker embedding using a length regulator to preserve the intonation and stress of the native language speaker when speaking the text.

[0040] In this embodiment, since the shared punctuation is used in the pre-processing step and the data is divided into four groups according to the pause length, the phoneme sequence and the native language speaker embedding can be aligned using a length regulator to preserve the intonation and stress of the native language speaker when speaking the text.

[0041] After the length regulator, decoding is performed to extract the VQ acoustic features from the phoneme sequence and the native language speaker embedding.

[0042] As an embodiment, the vector quantized acoustic features and auxiliary features of the native language speaker pronunciation style determined by the decoder from the phoneme sequence encoding and the native language speaker embedding include:

[0043] quantizing the phoneme sequence encoding into a plurality of speech frames, and predicting the codebook index of each of the plurality of speech frames in the native language speaker embedding separately using an auxiliary controller to construct a native language high-fidelity speech;

[0044] determining the vector quantized acoustic features and auxiliary features of the native language speaker pronunciation style from the native language high-fidelity speech by the decoder, wherein the auxiliary features include the codebook index probability of the predicted speech frame.

[0045] In this embodiment, the method uses the wav2vec 2.0 model to extract VQ acoustic features, which has two quantization codebooks, each containing 320 code words. The wav2vec 2.0 model is pre-trained on 10,000 hours of Mandarin data. It quantizes each input speech into multiple frames with a step size of 20ms, and each frame can be represented by concatenating two 256-dimensional code words in each codebook. In the mixed language dataset of the method, there are about 28.8k possible index combinations. The purpose is to accurately predict these index pairs to construct high-fidelity speech. In addition, the method predicts the index of each codebook separately, rather than their combination, resulting in a two-class classification problem of 320. The method chooses wav2vec 2.0 as the VQ feature extractor because it provides a more robust speech representation with less speaker information compared to other VQ features. The rationale for this choice will be further explained in the experimental section.

[0046] Similar to VQTTS (vector-quantized TTS), the method utilizes log pitch, energy, and POV (probability of voice) as auxiliary features. First, the phone-level representations of the mixed-language dataset are computed and normalized. Then, k-means clustering is applied to group these representations into 128 different classes, resulting in the cluster indices used as auxiliary labels for the PL (Phone-level) information.

[0047] Prior art cross-lingual TTS struggles to accurately preserve the speaker's timbre and eliminate the accent of their first language, resulting in unnatural synthesized speech. The main reason often stems from the entanglement between the speaker and the language, which is often reflected in the nature of traditional acoustic features, such as the fused spectrogram. However, the method experimentally found that the self-supervised VQ features contain much less speaker information compared to traditional acoustic features. Therefore, in the VQ-based TTS method, there is no need to use additional techniques to disentangle the speaker and language within the acoustic model. This allows the model to focus solely on modeling the text and language features while delegating the task of controlling the speaker's timbre to the vocoder. As a result, the VQ-based TTS model naturally learns how to speak different languages in the native language with the personality of a non-native speaker.

[0048] For step S12, the method DSE-TTS (Dual Speaker Embedding-TTS) framework is used to improve the nativeness and speaker similarity in cross-lingual TTS scenarios. The vector-quantized acoustic features, auxiliary features, and non-native language target speaker embedding in the dual speaker embedding determined in step S11 are input into the vec2wav vocoder in the framework. During the speech synthesis stage, whether it is intra-lingual synthesis or cross-lingual synthesis, the native speaker embedding corresponding to the input language is selected as the speaker embedding in the vec2wav vocoder, because the input speaker embedding of the txt2vec uses the native speaker in the language of the input text. In contrast, the speaker embedded in the vec2wav is set as the target speaker. Therefore, in the case of cross-lingual, this means that the native speaker embedding in the txt2vec represents the language speaking style, while the target speaker embedding in the vec2wav controls the timbre. In this way, the speaking style of a specific language and the timbre of the speaker are naturally separated by the dual embedding.

[0049] For vec2wav, the X-vector is used as the speaker embedding to control the timbre, which is extracted from a pre-trained speaker recognition model. In addition, to make the timbre closer to the target speaker when performing cross-lingual synthesis, the distribution of the native language speaker's pitch predicted by txt2vec is changed to match the pitch of the target speaker. The formula is as follows:

[0050]

[0051] Where the subscripts "tgt" and "ntv" represent the target and native language users, respectively. μ and σ are the mean and standard deviation of the pitch values of the target or native language speakers in the training set. This method performs the pitch shift before sending the auxiliary features to vec2wav for synthesis.

[0052] In the above manner, the cross-lingual acoustic features that simulate the timbre of the target language speaker on the basis of the pronunciation style of the native language speaker can be obtained.

[0053] For step S13, the final cross-lingual synthesized speech is generated by the HIFIGAN (High Fidelity Generative Adversarial Networks) generator using the cross-lingual acoustic features, which can be spoken in a foreign language by a designated non-native speaker.

[0054] As an implementation manner, the txt2vec acoustic model and the vec2wav vocoder are pre-trained by a training data set including native language speaker embeddings and target language speaker embeddings of non-native languages.

[0055] In the present embodiment, specifically, in the model training phase, the training data set (both the native language speaker embedding and the target language speaker embedding of the non-native language correspond to the same speaker).

[0056] The speaker embedding of the native speaker corresponding to the target language is selected as the speaker embedding of the acoustic model txt2vec, and the speaker embedding of the native speaker and the corresponding language ID number are input to the auxiliary controller. The auxiliary controller outputs a predicted phoneme level auxiliary label, and the auxiliary controller is trained based on the error between the pre-prepared reference phoneme level auxiliary label and the obtained predicted phoneme level auxiliary label, so that the predicted phoneme level auxiliary label output by the auxiliary controller approaches the reference phoneme level auxiliary label. In the same way, in vec2wav, the extraction of X-vector is trained using the target language speaker embedding of the non-native language to more accurately extract the timbre of the target language speaker of the non-native language.

[0057] It can be seen from the embodiment that the method constructs a cross-lingual TTS model based on VQTTS, which is composed of dual-speaker embeddings and independently models language speaking style and speaker voice respectively. The VQ feature has less speaker-related characteristics. Using this finding, cross-lingual speech synthesis with high naturalness and similar voice to the target speaker is realized. Experiments show that the speech effect of the method is better than the prior art in intra-lingual and cross-lingual synthesis.

[0058] The method is tested and explained, and the data set of the method includes four languages: Mandarin (ZH), English (EN), Spanish (ES) and German (DE). The data of German and Spanish is obtained from M_AILASS (data set), and the data of English and Mandarin is from LibriTTS and Aishell3 respectively. In fact, it may be difficult to collect enough data for some languages. In order to simulate this situation and test the language adaptation ability of the method, several hours of data are randomly selected from German and Spanish as low-resource languages. As Figure 3 The total duration and number of speakers of the speeches are listed. During the training process, all the speech is resampled to 24 kHz, and 5% of the speech is used for verification and test set. In order to extract the basic fact phoneme duration, forced alignment is performed using Kaldi.

[0059] Regarding the experimental setup, the method uses batch sizes of 16 and 8 respectively, and trains 200 time periods (epochs) on txt2vec and 100 epochs on vec2wav. The training process is performed separately on an NVIDIA 2080Ti GPU. A publicly available pre-trained wav2vec 2.0 model is used for VQ acoustic feature extraction. In order to evaluate the performance of the method model, SANE-TTS in the prior art is used as a baseline. The method trains a SANE-TS model for 200 epochs using a batch size of 16, while keeping all other parameters consistent.

[0060] In order to study the relationship between different acoustic features and speakers, a speaker classification model is first constructed to evaluate the classification accuracy of various features. The method compares mel-spectrogram (an acoustic feature widely used in TTS models) with four different VQ features extracted from open-source pre-trained models, including VQ-wav2vec, wav2vec 2.0, XLSR-53 and Encodec. The classification model of the method uses an X-vector architecture and adds two linear layers to predict speaker features. The model is trained on the LibriTTS training set, which includes more than 2000 speakers. After training the model for 80 epochs, the classification accuracy of speaker features on the test set is analyzed. As Figure 4As shown, the mel spectrograms contain enough information about the speaker characteristics, resulting in a high accuracy of speaker classification. In contrast, the VQ features have significantly less speaker information, resulting in a lower accuracy than the mel spectrograms. Based on the experimental results of the present method, wav2vec 2.0 is selected as the acoustic feature of the present method because it has a relatively low speaker recognition performance, which indicates that it contains less speaker-related information.

[0061] The present method uses both subjective and objective measures to evaluate the quality of intra- and cross-lingual synthesis. The subjective measures include NMOS (Nativeness MOS) and SMOS (Similarity MOS), where MOS stands for Mean Option Score. NMOS is used to evaluate the fluency of the synthesized speech, while SMOS is used to evaluate the degree of speaker similarity. The higher the NMOS score, the closer the synthesized speech is to the native language. The MOS rating is based on a 1-5 point scale with increments of 0.5 points and a 95% confidence interval. 30 speech samples were synthesized for each language using random texts in the test set, and multiple raters were recruited for evaluation. The raters include 15 bilingual speakers of Mandarin and English to evaluate the speech quality of English and Mandarin, and 15 trilingual speakers of Mandarin, English, German, and English, Mandarin, and Spanish to evaluate the synthesized speech of German and Spanish, respectively. For objective indicators, the WER (word error rate), CER (character error rate), and SECS (speaker embedding cosine similarity) between the synthesized speech and the real speech were calculated. WER is used for Spanish, German, and English, and CER is used for Mandarin. Pre-trained ASR models were used, with Whisper for Spanish, German, and English, and a transformer ASR model for Mandarin. For speaker similarity, an independently trained ResNet-based r-vector speaker verification model was used, and the cosine similarity score between 0 and 1 was calculated. The larger the score, the better the speaker similarity. To compare the present method model and the baseline model, 100 speech samples were synthesized for each language by randomly selecting sentences from the test set.

[0062] As Figure 5The average NMOS and WER(CER) in intra-lingual evaluation are shown. It is clear that DSE-TS has achieved near-perfect NMOS scores and outperforms the baseline model in all metrics and all languages. Specifically, DSE-TTS scores more than 4.3 in NMOS for each language and has lower WER(CER) than the baseline model.

[0063] As Figure 6 The results of the cross-lingual evaluation of the present method are given. It can be observed that the results are consistent with those obtained in intra-lingual synthesis, as DSE-TTS greatly outperforms SANE-TTS in both NMOS and WER scores. Specifically, in NMOS scores, raters preferred DSE-TTS over the baseline by more than 0.3 in all speaker language combinations. Moreover, SMOS and SECS scores show that DSE-TTS preserves similar speaker characteristics as SANE-TTS. These findings suggest that DSE-TS can synthesize high-quality German and Spanish speech in the voice of non-native language speakers, but with greater similarity to native language speakers compared to the baseline model.

[0064] The present method also conducted an ablation study to investigate the impact of dual speaker embedding (DSE) on the performance of the present method model. As Figure 6 The results shown in Table 6 indicate that after integrating DSE, the naturalness of the synthesized speech is significantly enhanced and the WER is significantly reduced. The observations also show that using DSE results in a slight decrease in speaker similarity scores compared to not using DSE. This can be because different languages have unique speaking styles, and non-native language users can sound slightly different when speaking a foreign language fluently. This is also evidence that DSE-TS produces speech in the native language, although it is not trained on bilingual users.

[0065] In summary, the present method proposes DSE-TTS, a VQ-TTS based cross-lingual TTS model that consists of dual speaker embedding, which models language speaking style and speaker timbre separately. VQ features have less speaker-dependent characteristics. Taking advantage of this finding, the present method improves the present method model by dual speaker embedding, thereby enabling cross-lingual speech synthesis with high naturalness and similar timbre to the target speaker. Experiments show that DSE-TTS outperforms SANE-TTS in both intra-lingual and cross-lingual synthesis, especially in terms of naturalness. The present method also verifies the effectiveness of dual speaker embedding through an ablation study.

[0066] As Figure 7Fig. 1 shows a structural schematic diagram of a cross-language speech synthesis system based on dual speaker embedding according to an embodiment of the present application. The system can execute the cross-language speech synthesis method based on dual speaker embedding according to any of the above embodiments, and is configured in a terminal.

[0067] The cross-language speech synthesis system 10 based on dual speaker embedding according to the embodiment comprises a pronunciation style feature determination program module 11, a timbre feature determination program module 12, and a speech synthesis program module 13.

[0068] The pronunciation style feature determination program module 11 is configured to input text and native language speaker embedding into a txt2vec acoustic model, in which a text encoder is used to determine a phoneme sequence encoding of the text, and a decoder is used to determine vector quantized acoustic features and auxiliary features of a native language speaker pronunciation style from the phoneme sequence encoding and the native language speaker embedding. The timbre feature determination program module 12 is configured to input a target language speaker embedding as a non-native language, the vector quantized acoustic features and auxiliary features into a vec2wav vocoder, in which an X-vector feature of the target language speaker embedding is extracted, the X-vector feature, the vector quantized acoustic features and auxiliary features are input into a feature encoder to obtain cross-language acoustic features simulating a target language speaker timbre based on a native language speaker pronunciation style. The speech synthesis program module 13 is configured to determine a cross-language synthesized speech of the cross-language acoustic features by using a generator.

[0069] The embodiment of the present application further provides a non-volatile computer storage medium, which stores computer executable instructions. The computer executable instructions can execute the cross-language speech synthesis method based on dual speaker embedding in any of the above method embodiments.

[0070] As an implementation form, the non-volatile computer storage medium of the present application stores computer executable instructions, which are configured to:

[0071] input text and native language speaker embedding into a txt2vec acoustic model, in which a text encoder is used to determine a phoneme sequence encoding of the text, and a decoder is used to determine vector quantized acoustic features and auxiliary features of a native language speaker pronunciation style from the phoneme sequence encoding and the native language speaker embedding;

[0072] inputting the target language speaker embedding as a non-native language, the vector quantized acoustic feature and the auxiliary feature into a vec2wav vocoder, in the vec2wav vocoder, extracting an X-vector feature of the target language speaker embedding, inputting the X-vector feature, the vector quantized acoustic feature and the auxiliary feature into a feature encoder, obtaining a cross-language acoustic feature simulating a voice color of the target language speaker on the basis of a pronunciation style of a native language speaker;

[0073] determining a cross-language synthesized speech of the cross-language acoustic feature by using a generator.

[0074] As a non-volatile computer readable storage medium, it can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the application. One or more program instructions are stored in the non-volatile computer readable storage medium, and when executed by the processor, the cross-language speech synthesis method based on double speaker embedding in any method embodiment described above is executed.

[0075] Figure 8 is a hardware structure schematic diagram of an electronic device for the cross-language speech synthesis method based on double speaker embedding provided by another embodiment of the application, as shown in Figure 8 The device comprises:

[0076] one or more processors 810 and memories 820, Figure 8 In the embodiment, the processor 810 is taken as an example. The device for the cross-language speech synthesis method based on double speaker embedding can further comprise an input device 830 and an output device 840.

[0077] The processor 810, the memory 820, the input device 830 and the output device 840 can be connected through a bus or other means, Figure 8 In the embodiment, the connection through the bus is taken as an example.

[0078] The memory 820 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the cross-language speech synthesis method based on double speaker embedding in the embodiment of the application. The processor 810 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 820, that is, the cross-language speech synthesis method based on double speaker embedding in the method embodiment is realized.

[0079] The memory 820 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs, and / or data needed by at least one function. The data storage area can store data, etc. In addition, the memory 820 can include a high-speed random access memory, and can also include a nonvolatile memory such as at least one disk memory device, a flash memory device, or other nonvolatile solid-state memory device. In some embodiments, the memory 820 can optionally include a memory disposed remotely from the processor 810, which can be connected to the mobile device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0080] The input device 830 can receive input digital or character information. The output device 840 can include a display screen and the like display device.

[0081] The one or more modules are stored in the memory 820 and, when executed by the one or more processors 810, perform the steps of the cross-lingual speech synthesis method based on dual speaker embedding in any of the method embodiments described above.

[0082] The above product can perform the method provided by the embodiments of the present application, and has the corresponding function modules and beneficial effects of performing the method. Technical details not described in detail in the embodiments can be referred to the method provided by the embodiments of the present application.

[0083] The non-volatile computer readable storage medium can include a program storage area and a data storage area. The program storage area can store an operating system, application programs, and / or data needed by at least one function. The data storage area can store data created according to the use of the device, etc. In addition, the non-volatile computer readable storage medium can include a high-speed random access memory, and can also include a non-volatile memory such as at least one disk memory device, a flash memory device, or other non-volatile solid-state memory device. In some embodiments, the non-volatile computer readable storage medium can optionally include a memory disposed remotely from the processor, which can be connected to the device through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0084] The embodiments of the present application also provide an electronic device, which includes at least one processor, and a memory connected in communication with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the cross-lingual speech synthesis method based on dual speaker embedding of any of the embodiments of the present application.

[0085] The electronic device of the embodiments of the present application exists in various forms, including but not limited to:

[0086] (1) Mobile communication device: This kind of device is characterized by having mobile communication function, and the main target is to provide voice and data communication. This kind of terminal includes smart phone, multimedia phone, functional phone, and low-end phone.

[0087] (2) Ultra-mobile personal computer device: This kind of device belongs to the category of personal computer, has computing and processing function, and generally has mobile internet characteristic. This kind of terminal includes PDA, MID and UMPC device, such as tablet computer.

[0088] (3) Portable entertainment device: This kind of device can display and play multimedia content. This kind of device includes audio and video player, palm game machine, electronic book, and smart toy and portable car navigation device.

[0089] (4) Other electronic devices with data processing function.

[0090] In this document, the terms "first", "second", etc. are used merely to distinguish one entity or action from another, and do not necessarily require or imply any actual such relationship or order between or among the entities or actions. Moreover, the terms "comprises", "comprising", "includes", "including", "has", "having" and the like are inclusive and are used as equivalents for "consisting of" and "consisting essentially of" and specify the presence of stated features, integers, steps or components but do not preclude the presence or addition of one or more other features, integers, steps, components or groups thereof. In the absence of more restrictive language, the word "comprising" does not exclude the presence of other elements or additional steps.

[0091] The device embodiments described above are only illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e. can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0092] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0093] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of cross-lingual speech synthesis based on dual speaker embeddings, comprising: inputting a text and a native language speaker embedding into a txt2vec acoustic model, in which a text encoder determines a phoneme sequence encoding of the text, and a decoder determines a vector-quantized acoustic feature and an auxiliary feature of a native language speaker pronunciation style from the phoneme sequence encoding and the native language speaker embedding, including: quantizing the phoneme sequence encoding into a plurality of speech frames, individually predicting a codebook index of each of the plurality of speech frames in the native language speaker embedding using an auxiliary controller to construct a native language high-fidelity speech, and determining the vector-quantized acoustic feature and the auxiliary feature of the native language speaker pronunciation style from the native language high-fidelity speech using the decoder, wherein the auxiliary feature includes a codebook index probability of a speech frame; inputting a target language speaker embedding of a non-native language, the vector-quantized acoustic feature and the auxiliary feature into a vec2wav vocoder, in which an X-vector feature of the target language speaker embedding is extracted, the X-vector feature, the vector-quantized acoustic feature and the auxiliary feature are input into a feature encoder to obtain a cross-lingual acoustic feature simulating a target language speaker timbre based on a native language speaker pronunciation style; determining a cross-lingual synthesized speech of the cross-lingual acoustic feature using a generator.

2. The method of claim 1, wherein, After the text encoder determines the phoneme sequence encoding of the text, the method further comprises: aligning the phoneme sequence and the native language speaker embedding using a length adjuster to retain a tone and stress of the native language speaker when speaking the text.

3. The method of claim 1, wherein, The txt2vec acoustic model and the vec2wav vocoder are obtained by pre-training using a training data set including a native language speaker embedding and a target language speaker embedding of a non-native language. 4.A system of cross-lingual speech synthesis based on dual speaker embeddings, comprising: a pronunciation style feature determination program module for inputting a text and a native language speaker embedding into a txt2vec acoustic model, in which a text encoder determines a phoneme sequence encoding of the text, and a decoder determines a vector-quantized acoustic feature and an auxiliary feature of a native language speaker pronunciation style from the phoneme sequence encoding and the native language speaker embedding, including: quantizing the phoneme sequence encoding into a plurality of speech frames, individually predicting a codebook index of each of the plurality of speech frames in the native language speaker embedding using an auxiliary controller to construct a native language high-fidelity speech, and determining the vector-quantized acoustic feature and the auxiliary feature of the native language speaker pronunciation style from the native language high-fidelity speech using the decoder, wherein the auxiliary feature includes a codebook index probability of a speech frame; The timbre feature determination program module is configured to input a target language speaker embedding as a non-native language, the vector-quantized acoustic feature, and the auxiliary feature into a vec2wav vocoder, extract an X-vector feature of the target language speaker embedding in the vec2wav vocoder, input the X-vector feature, the vector-quantized acoustic feature, and the auxiliary feature into a feature encoder, and obtain a cross-language acoustic feature that simulates a timbre of a target language speaker on the basis of a native language speaker pronunciation style. The speech synthesis program module is configured to determine a cross-language synthesized speech of the cross-language acoustic feature by using a generator.

5. The system of claim 4, wherein, The system further comprises a length adjustment module configured to: align the phoneme sequence and the native language speaker embedding by using a length adjuster to retain a tone and stress of the native language speaker when speaking the text.

6. The system of claim 4, wherein, The txt2vec acoustic model and the vec2wav vocoder are obtained by pre-training using a training data set comprising a native language speaker embedding and a target language speaker embedding as a non-native language.

7. An electronic device comprising: At least one processor and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of any one of claims 1-3.

8. A storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the steps of any one of claims 1-3.

Images